AWS Glue - Developer Guide

1 AWS GlueDeveloper GuideAWS glue Developer GuideAWS glue : Developer GuideCopyright 2019 Amazon Web Services, Inc. and/or its affiliates. All rights 's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any mannerthat is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks notowned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored glue Developer GuideTable of ContentsWhat Is AWS glue ?

2 1 When Should I Use AWS glue ? .. 1 How It Works .. 3 Serverless ETL Jobs Run in Isolation .. 3 Concepts .. 4 AWS glue Terminology .. 6 AWS glue Console .. 6 AWS glue Data Catalog .. 7 AWS glue Crawlers and Classifiers .. 7 AWS glue ETL Operations .. 7 The AWS glue Jobs 7 Converting Semi-Structured Schemas to Relational Schemas .. 8 Getting Started .. 10 Setting up IAM Permissions for AWS glue .. 10 Step 1: Create an IAM Policy for the AWS glue Service .. 10 Step 2: Create an IAM Role for AWS glue .

3 14 Step 3: Attach a Policy to IAM Users That Access AWS glue .. 15 Step 4: Create an IAM Policy for Notebook Servers .. 22 Step 5: Create an IAM Role for Notebook Servers .. 24 Step 6: Create an IAM Policy for Amazon SageMaker Notebooks .. 25 Step 7: Create an IAM Role for Amazon SageMaker Notebooks .. 27 Setting Up DNS in Your VPC .. 28 Setting Up Your Environment to Access Data Stores .. 28 Amazon VPC Endpoints for Amazon S3 .. 29 Setting Up a VPC to Connect to JDBC Data Stores .. 30 Setting Up Your Environment for Development Endpoints.

4 33 Setting Up Your Network for a Development Endpoint .. 33 Setting Up Amazon EC2 for a Notebook Server .. 34 Setting Up Encryption .. 35 Console Workflow Overview .. 37 Security .. 39 Authentication and Access Control .. 40 Access-Control Overview .. 41 Cross-Account Access .. 50 Resource ARNs .. 54 Policy Examples .. 57 API Permissions Reference .. 66 Encryption and Secure Access .. 80 Encrypting Your Data Catalog .. 81 Encrypting Connection Passwords .. 82 Encrypting Data Written by AWS glue .

5 82 Populating the AWS glue Data Catalog .. 86 Defining a Database in Your Data Catalog .. 88 Working with Databases on the Console .. 88 Defining Tables in the AWS glue Data Catalog .. 88 Table Partitions .. 89 Working with Tables on the Console .. 89 Adding a Connection to Your Data Store .. 92 When Is a Connection Used? .. 92 Defining a Connection in the AWS glue Data Catalog .. 92 Connecting to a JDBC Data Store in a VPC .. 93 Working with Connections on the Console .. 94iiiAWS glue Developer GuideCataloging Tables with a Crawler.

6 97 Defining a Crawler in the AWS glue Data Catalog .. 98 Which Data Stores Can I Crawl? .. 98 Using Include and Exclude Patterns .. 98 What Happens When a Crawler Runs? .. 101 Are Amazon S3 Folders Created as Tables or Partitions? .. 102 Configuring a Crawler .. 103 Scheduling a Crawler .. 106 Working with Crawlers on the Console .. 107 Adding Classifiers to a Crawler .. 109 When Do I Use a Classifier?.. 109 Custom 109 Built-In Classifiers in AWS glue .. 110 Writing Custom Classifiers .. 112 Working with Classifiers on the Console.

7 122 Working with Data Catalog Settings on the AWS glue Console .. 123 Populating the Data Catalog Using AWS CloudFormation Templates .. 124 Sample 125 Sample Database, Table, Partitions .. 126 Sample Grok Classifier .. 129 Sample JSON Classifier .. 130 Sample XML 130 Sample Amazon S3 Crawler .. 131 Sample Connection .. 132 Sample JDBC Crawler .. 133 Sample Job for Amazon S3 to Amazon S3 .. 135 Sample Job for JDBC to Amazon S3 .. 136 Sample On-Demand Trigger .. 137 Sample Scheduled Trigger .. 138 Sample Conditional Trigger.

8 139 Sample Development Endpoint .. 140 Authoring 141 Workflow Overview .. 142 Adding Jobs .. 142 Defining Job Properties .. 142 Built-In Transforms .. 144 Jobs on the 146 Editing Scripts .. 151 Defining a 151 Scripts on the 152 Providing Your Own Custom Scripts .. 153 Triggering Jobs .. 154 Triggering Jobs Based on Schedules or Events .. 154 Defining Trigger Types .. 154 Working with Triggers on the Console .. 154 Using Development Endpoints .. 156 Managing the Environment .. 156 Using a Dev 156 Accessing Your Dev Endpoint.

9 156 Development Endpoints on the Console .. 157 Tutorial Prerequisites .. 161 Tutorial: Local Zeppelin Notebook .. 164 Tutorial: Amazon EC2 Zeppelin Notebook Server .. 167 Tutorial: Use a REPL Shell .. 170 Tutorial: Use PyCharm Professional .. 171 Managing 177 Notebook Server Considerations .. 179ivAWS glue Developer GuideNotebooks on the 185 Running and 188 Automated Tools .. 189 Time-Based Schedules for Jobs and Crawlers .. 189 Cron Expressions .. 189 Job 191 Using Job 192 Using an AWS glue Script.

10 193 Using Modification 194 Automating with CloudWatch Events .. 196 Monitoring with Amazon CloudWatch .. 196 Using CloudWatch Metrics .. 197 Setting Up Amazon CloudWatch Alarms on AWS glue Job Profiles .. 210 Job Monitoring and 210 Debugging OOM Exceptions and Job Abnormalities .. 211 Debugging Demanding Stages and Straggler Tasks .. 218 Monitoring the Progress of Multiple Jobs .. 222 Monitoring for DPU Capacity Planning .. 226 Logging Using CloudTrail .. 230 AWS glue Information in CloudTrail .. 231 Understanding AWS glue Log File Entries.