Amazon Kinesis Data Streams

1 Amazon Kinesis data StreamsDeveloper GuideAmazon Kinesis data Streams Developer GuideAmazon Kinesis data Streams : Developer GuideCopyright 2018 Amazon Web Services, Inc. and/or its affiliates. All rights 's trademarks and trade dress may not be used in connection with any product or service that is not Amazon 's, in any mannerthat is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon . All other trademarks notowned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored Kinesis data Streams Developer GuideTable of ContentsWhat Is Amazon Kinesis data Streams ? .. 1 What Can I Do with Kinesis data Streams ?

2 1 Benefits of Using Kinesis data Streams .. 2 Related Services .. 2 Key Concepts .. 2 High-level Architecture .. 2 Terminology .. 3 data Streams .. 5 Determining the Initial Size of a Kinesis data Stream.. 5 Creating a Stream.. 6 Updating a Stream.. 6 Producers .. 7 Consumers.. 8.. 8 Limits.. 8 API limits.. 8 Increasing Limits .. 9 Getting Started .. 10 Setting Up .. 10 Sign Up for AWS .. 10 Download Libraries and Tools .. 10 Configure Your Development Environment .. 11 Tutorial: Visualizing Web Traffic .. 11 Kinesis data Streams data Visualization Sample Application .. 11 Prerequisites .. 12 Step 1: Start the Sample Application .. 12 Step 2: View the Components of the Sample Application.

3 13 Step 3: Delete Sample Application.. 16 Step 4: Next Steps .. 16 Tutorial: Getting Started Using the CLI .. 17 Install and Configure the AWS CLI .. 17 Perform Basic Stream Operations .. 19 Tutorial: Analyzing Real-Time Stock data .. 23 Prerequisites .. 24 Step 1: Create a data Stream.. 25 Step 2: Create an IAM Policy and User .. 26 Step 3: Download and Build the Implementation Code .. 29 Step 4: Implement the Producer .. 30 Step 5: Implement the Consumer.. 32 Step 6: (Optional) Extending the Consumer .. 35 Step 7: Finishing Up.. 36 Creating and Managing Streams .. 38 Creating a Stream.. 38 Build the Kinesis data Streams Client.

4 38 Create the Stream.. 39 Listing Streams .. 40 Listing Shards .. 41 Retrieving Shards from a Stream.. 42 Deleting a Stream.. 42 Resharding a Stream.. 42 Strategies for Resharding .. 43 Splitting a Shard .. 44 Merging Two Shards .. 45 After Resharding .. 46iiiAmazon Kinesis data Streams Developer GuideChanging the data Retention Period .. 47 Tagging Your Streams .. 48 Tag Basics .. 48 Tracking Costs Using Tagging .. 49 Tag Restrictions .. 49 Tagging Streams Using the Kinesis data Streams Console .. 49 Tagging Streams Using the AWS CLI .. 50 Tagging Streams Using the Kinesis data Streams API .. 50 Monitoring Streams .. 51 Monitoring the Service with CloudWatch.

5 51 Monitoring the Agent with CloudWatch .. 60 Logging Amazon Kinesis data Streams API Calls with AWS CloudTrail .. 61 Monitoring the KCL with CloudWatch .. 65 Monitoring the KPL with CloudWatch .. 73 Controlling Access .. 77 Policy Syntax .. 78 Actions for Kinesis data Streams .. 78 Amazon Resource Names (ARNs) for Kinesis data Streams .. 79 Example Policies for Kinesis data Streams .. 79 Using Server-Side Encryption .. 80 What Is Server-Side Encryption for Kinesis data Streams ? .. 81 Costs, Regions, and Performance Considerations .. 81 How Do I Get Started with Server-Side Encryption? .. 82 Creating and Using User-Generated KMS master Keys .. 83 Permissions to Use User-Generated KMS master Keys.

6 84 Verifying and Troubleshooting KMS Key Permissions .. 85 Using Interface VPC Endpoints .. 85 Interface VPC endpoints for Kinesis data Streams .. 85 Using interface VPC endpoints for Kinesis data Streams .. 85 Availability .. 86 Managing Streams Using the Console .. 86 Writing data to Streams .. 88 Using the KPL.. 88 Role of the KPL .. 89 Advantages of Using the KPL .. 89 When Not to Use the KPL.. 90 Installing the KPL.. 90 Transitioning to Amazon Trust Services (ATS) Certificates for the Kinesis Producer Library .. 90 KPL Supported Platforms .. 90 KPL Key Concepts .. 91 Integrating the KPL with Producer Code .. 92 Writing to your Kinesis data stream.

7 94 Configuring the KPL.. 95 Consumer De-aggregation .. 96 Using the KPL with Kinesis data Firehose .. 98 Using the API.. 98 Adding data to a Stream.. 98 Using the Agent .. 102 Prerequisites .. 103 Download and Install the Agent .. 103 Configure and Start the Agent .. 104 Agent Configuration Settings .. 104 Monitor Multiple File Directories and Write to Multiple Streams .. 107 Use the Agent to Pre-process data .. 107 Agent CLI Commands .. 110 Troubleshooting .. 111 Producer Application is Writing at a Slower Rate Than Expected .. 111ivAmazon Kinesis data Streams Developer GuideUnauthorized KMS master key permission error .. 112 Advanced Topics.

8 112 Retries and Rate Limiting .. 112 Considerations When Using KPL Aggregation .. 113 Reading data from Streams .. 115 Using Consumers.. 116 Using the Kinesis Client Library .. 116 Using the Kinesis Client Library .. 130 Using the API.. 134 Using Consumers with Enhanced Fan-Out .. 138 Using the Kinesis Client Library .. 139 Using the API.. 143 Using the AWS Management Console .. 144 Migrating from Kinesis Client Library to .. 145 Migrating the Record Processor .. 145 Migrating the Record Processor Factory .. 149 Migrating the Worker .. 149 Configuring the Amazon Kinesis Client .. 150 Idle Time Removal .. 153 Client Configuration Removals.

9 153 Troubleshooting .. 154 Some Kinesis data Streams Records are Skipped When Using the Kinesis Client Library .. 154 Records Belonging to the Same Shard are Processed by Different Record Processors at theSame Time.. 154 Consumer Application is Reading at a Slower Rate Than Expected .. 155 GetRecords Returns Empty Records Array Even When There is data in the Stream.. 155 Shard Iterator Expires Unexpectedly .. 156 Consumer Record Processing Falling Behind .. 156 Unauthorized KMS master key permission error .. 157 Advanced Topics .. 157 Tracking State .. 157 Low-Latency Processing .. 158 Using AWS Lambda with the Kinesis Producer Library .. 159 Resharding, Scaling, and Parallel Processing.

10 159 Handling Duplicate Records .. 160 Recovering from Failures .. 161 Handling Startup, Shutdown, and Throttling .. 162 Document History .. 164 AWS Glossary .. 166vAmazon Kinesis data Streams Developer GuideWhat Can I Do with Kinesis data Streams ?What Is Amazon Kinesis DataStreams?You can use Amazon Kinesis data Streams to collect and process large Streams of data records in realtime. You can create data -processing applications, known as Kinesis data Streams applications. A typicalKinesis data Streams application reads data from a data stream as data records. These applications canuse the Kinesis Client Library, and they can run on Amazon EC2 instances. You can send the processedrecords to dashboards, use them to generate alerts, dynamically change pricing and advertisingstrategies, or send data to a variety of other AWS services.