AWS Batch - User Guide

1 AWS BatchUser GuideAWS Batch user GuideAWS Batch : user GuideCopyright 2018 Amazon Web Services, Inc. and/or its affiliates. All rights 's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any mannerthat is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks notowned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored Batch user GuideTable of ContentsWhat Is AWS Batch ? .. 1 Components of AWS Batch .. 1 Job 1 Job 1 Compute Environment .. 1 Getting Started .. 2 Setting Up.

2 3 Sign Up for AWS .. 3 Create an IAM user .. 4 Create IAM Roles for your Compute Environments and Container Instances .. 5 Create a Key Pair .. 5 Create a Virtual Private Cloud .. 6 Create a Security Group .. 7 Install the AWS CLI .. 8 Getting Started .. 9 Step 1: Define a 9 Step 2: Configure the Compute Environment and Job Queue .. 13 Submitting a Job .. 13 Job 15 Job Environment Variables .. 16 Automated Job Retries .. 16 Job 17 Job 17 Array Jobs .. 18 Example Array Job Workflow .. 20 Tutorial: Using Array Job Index .. 22 Job 27 Creating a Job Definition .. 27 Job Definition Template .. 29 Job Definition Parameters .. 30 Job Definition 31 Type.

3 31 Parameters .. 31 Retry Strategy .. 32 Container Properties .. 32 Example Job 37 Use Environment Variables .. 37 Using Parameter Substitution .. 38 Test GPU Functionality .. 38 Job 39 Creating a Job Queue .. 39 Job Queue Template .. 39 Job Queue Parameters .. 40 Job Queue 40 Priority .. 40 Compute Environment Order .. 41 Job 42 Compute Environments .. 43 Managed Compute Environments .. 43 Unmanaged Compute Environments .. 44iiiAWS Batch user GuideCompute Resource AMIs .. 44 Compute Resource AMI Specification .. 44 Creating a Compute Resource AMI .. 45 Creating a GPU Workload AMI .. 47 Launch Template Support .. 49 Amazon EC2 user Data in Launch Templates.

4 50 Creating a Compute Environment .. 51 Compute Environment Template .. 54 Compute Environment Parameters .. 55 Compute Environment Name .. 55 Type .. 55 Compute Resources .. 56 Service Role .. 58 Memory Management .. 59 Reserving System Memory .. 59 Viewing Compute Resource Memory .. 59 IAM Policies, Roles, and Permissions .. 61 Policy Structure .. 61 Policy Syntax .. 62 Actions for AWS Batch .. 62 Amazon Resource Names for AWS Batch .. 63 Testing Permissions .. 63 Supported Resource-Level Permissions .. 64 Example Policies .. 64 Read-Only Access .. 65 Restricting user , Image, Privilege, Role .. 65 Restrict Job Definition .. 66 Restrict Job Queue .. 67 AWS Batch Managed Policy.

5 67 AWSB atchFullAccess .. 67 Creating IAM Policies .. 68 AWS Batch Service IAM Role .. 68 Amazon ECS Instance Role .. 70 Amazon EC2 Spot Fleet Role .. 71 Create Amazon EC2 Spot Fleet Roles in the AWS Management Console .. 71 Create Amazon EC2 Spot Fleet Roles with the AWS CLI .. 72 CloudWatch Events IAM Role .. 73 CloudWatch Events .. 74 AWS Batch Events .. 74 Job State Change Events .. 74 AWS Batch Jobs as CloudWatch Events Targets .. 75 Creating a Scheduled Job .. 76 Event Input Transformer .. 77 Tutorial: Listening for AWS Batch CloudWatch Events .. 78 Prerequisites .. 78 Step 1: Create the Lambda Function .. 78 Step 2: Register Event Rule .. 79 Step 3: Test Your Configuration.

6 79 Tutorial: Sending Amazon Simple Notification Service Alerts for Failed Job Events .. 80 Prerequisites .. 80 Step 1: Create and Subscribe to an Amazon SNS Topic .. 80 Step 2: Register Event Rule .. 80 Step 3: Test Your Rule .. 81 CloudTrail .. 82 AWS Batch Information in CloudTrail .. 82 Understanding AWS Batch Log File Entries .. 83ivAWS Batch user GuideTutorial: Creating a VPC .. 85 Step 1: Create an Elastic IP Address for Your NAT Gateway .. 85 Step 2: Run the VPC Wizard .. 85 Step 3: Create Additional Subnets .. 86 Next Steps .. 86 Service Limits .. 88 Troubleshooting .. 89 INVALID Compute Environment .. 89 Incorrect Role Name or ARN .. 89 Repairing an INVALID Compute Environment.

7 90 Jobs Stuck in RUNNABLE 90 Spot Instances Not Tagged on Creation .. 91 Document History .. 92 AWS Glossary .. 93vAWS Batch user GuideComponents of AWS BatchWhat Is AWS Batch ?AWS Batch enables you to run Batch computing workloads on the AWS Cloud. Batch computing is acommon way for developers, scientists, and engineers to access large amounts of compute resources,and AWS Batch removes the undifferentiated heavy lifting of configuring and managing the requiredinfrastructure, similar to traditional Batch computing software. This service can efficiently provisionresources in response to jobs submitted in order to eliminate capacity constraints, reduce compute costs,and deliver results a fully managed service, AWS Batch enables you to run Batch computing workloads of any scale.

8 AWSB atch automatically provisions compute resources and optimizes the workload distribution based onthe quantity and scale of the workloads. With AWS Batch , there is no need to install or manage batchcomputing software, which allows you to focus on analyzing results and solving of AWS BatchAWS Batch is a regional service that simplifies running Batch jobs across multiple Availability Zoneswithin a region. You can create AWS Batch compute environments within a new or existing VPC. Aftera compute environment is up and associated with a job queue, you can define job definitions thatspecify which Docker container images to run your jobs. Container images are stored in and pulled fromcontainer registries, which may exist within or outside of your AWS unit of work (such as a shell script, a Linux executable, or a Docker container image) that you submitto AWS Batch .

9 It has a name, and runs as a containerized application on an Amazon EC2 instance inyour compute environment, using parameters that you specify in a job definition. Jobs can referenceother jobs by name or by ID, and can be dependent on the successful completion of other jobs. For moreinformation, see Jobs (p. 13).Job DefinitionsA job definition specifies how jobs are to be run; you can think of it as a blueprint for the resourcesin your job. You can supply your job with an IAM role to provide programmatic access to other AWSresources, and you specify both memory and CPU requirements. The job definition can also controlcontainer properties, environment variables, and mount points for persistent storage.

10 Many of thespecifications in a job definition can be overridden by specifying new values when submitting individualJobs. For more information, see Job Definitions (p. 27)Job QueuesWhen you submit an AWS Batch job, you submit it to a particular job queue, where it resides until it isscheduled onto a compute environment. You associate one or more compute environments with a jobqueue, and you can assign priority values for these compute environments and even across job queuesthemselves. For example, you could have a high priority queue that you submit time-sensitive jobs to,and a low priority queue for jobs that can run anytime when compute resources are EnvironmentA compute environment is a set of managed or unmanaged compute resources that are used to runjobs.