Amazon Kinesis Data Analytics - AWS Documentation

1 Amazon Kinesis data AnalyticsDeveloper GuideAmazon Kinesis data Analytics Developer GuideAmazon Kinesis data Analytics : Developer GuideCopyright 2018 Amazon Web Services, Inc. and/or its affiliates. All rights 's trademarks and trade dress may not be used in connection with any product or service that is not Amazon 's, in any mannerthat is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon . All other trademarks notowned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored Kinesis data Analytics Developer GuideTable of ContentsWhat Is Amazon Kinesis data Analytics ?

2 1 When Should I Use Amazon Kinesis data Analytics ? .. 1 Are You a First-Time User of Amazon Kinesis data Analytics ? .. 1 How It Works .. 5 Configuring a Streaming Source .. 5 Configuring a Reference Source .. 7 Working with JSONPath .. 9 Mapping Streaming Source Elements to SQL Input Columns .. 12 Using the Schema Discovery Feature on Streaming data .. 16 Using the Schema Discovery Feature on Static data .. 18 Preprocessing data Using a Lambda Function .. 21 Parallelizing Input Streams for Increased Throughput.

3 27 Application 32 Creating an Output Using the AWS CLI .. 32 Using a Lambda Function as Output .. 33 Application Output Delivery Model .. 39 Error Handling .. 40 Reporting Errors Using an In-Application Error 40 Granting Permissions .. 41 Trust Policy .. 41 Permissions Policy .. 41 Auto Scaling 43 Getting Started .. 45 Step 1: Set Up an Account .. 45 Sign Up for AWS .. 45 Create an IAM User .. 46 Next Step .. 46 Step 2: Set Up the AWS CLI .. 46 Next Step.

4 47 Step 3: Create Your Starter Analytics Application .. 47 Step : Create an Application .. 49 Step : Configure Input .. 50 Step : Add Real-Time Analytics (Add Application Code) .. 52 Step : (Optional) Update the Application Code .. 54 Step 4 (Optional) Edit the Schema and SQL Code Using the Console .. 56 Working with the Schema Editor .. 56 Working with the SQL Editor .. 63 Streaming SQL Concepts .. 66In-Application Streams and Pumps .. 66 Timestamps and the ROWTIME Column .. 67 Understanding Various Times in Streaming Analytics .

5 67 Continuous 69 Windowed Queries .. 70 Stagger Windows .. 70 Tumbling Windows .. 75 Sliding Windows .. 76 Stream Joins .. 80 Example 1: Report Orders Where There Are Trades Within One Minute of the Order BeingPlaced .. 82 Transforming data .. 82 Preprocessing Streams with Lambda .. 82iiiAmazon Kinesis data Analytics Developer GuideTransforming String Values .. 82 Transforming DateTime Values .. 96 Transforming Multiple data Types .. 99 Windows and Aggregation .. 104 Stagger Window.

6 104 Tumbling Window Using ROWTIME .. 107 Tumbling Window Using an Event Time Stamp .. 109 Most Frequently Occurring Values (TOP_K_ITEMS_TUMBLING) .. 112 Aggregating Partial Results .. 116 Example: Add Reference data Source .. 116 Machine 119 Detecting Anomalies .. 119 Example: Detect Anomalies and Get an Explanation .. 125 Example: Detect Hotspots .. 129 Alerts and Errors .. 139 Simple Alerts .. 139 Throttled Alerts .. 140In-Application Error 141 Solution Accelerators .. 142 Real-Time Insights on AWS Account Activity.

7 142 Real-Time IoT Device Monitoring with Kinesis data Analytics .. 143 Real-Time Web Analytics with Kinesis data Analytics .. 143 AWS Connected Vehicle Solution .. 144 Monitoring Tools .. 144 Automated Tools .. 144 Manual Tools .. 145 Monitoring with Amazon CloudWatch .. 145 Metrics and 146 Viewing Metrics and 154 Best Practices .. 155 Managing 155 Defining Input 156 Connecting to Outputs .. 157 Authoring Application 157 Testing Applications .. 157 Setting up a Test Application.

8 157 Testing Schema Changes .. 158 Testing Code Changes .. 158 Troubleshooting .. 159 Unable to Run SQL Code .. 159 Unable to Detect or Discover My Schema .. 159 Reference data is Out of Date .. 160 Application Not Writing to Destination .. 160 Important Application Health Parameters to Monitor .. 160 Invalid Code Errors When Running an Application .. 161 Application is Writing Errors to the Error 161 Insufficient Throughput or High MillisBehindLatest .. 161 Authentication and Access Control.

9 163 Access Control .. 164 Overview of Managing Access .. 164 Amazon Kinesis data Analytics Resources and Operations .. 165ivAmazon Kinesis data Analytics Developer GuideUnderstanding Resource Ownership .. 165 Managing Access to Resources .. 165 Specifying Policy Elements: Actions, Effects, and Principals .. 167 Specifying Conditions in a Policy .. 167 Using Identity-Based Policies (IAM Policies) .. 168 Permissions Required to Use the Amazon Kinesis data Analytics Console .. 168 AWS Managed (Predefined) Policies for Amazon Kinesis data Analytics .

10 169 Customer Managed Policy Examples .. 170 Amazon Kinesis data Analytics API Permissions Reference .. 173 SQL Reference .. 175 API Reference .. 176 Actions .. 176 AddApplicationCloudWatchLoggingOption .. 177 AddApplicationInput .. 179 AddApplicationInputProcessingConfigurati on .. 182 AddApplicationOutput .. 184 AddApplicationReferenceDataSource .. 187 CreateApplication .. 195 DeleteApplicationCloudWatchLoggingOption .. 197 DeleteApplicationInputProcessingConfigur ation .. 201 DeleteApplicationReferenceDataSource.