Amazon Kinesis Data Streams

1 Amazon Kinesis data StreamsDeveloper GuideAmazon Kinesis data Streams Developer GuideAmazon Kinesis data Streams : Developer GuideCopyright 2018 Amazon Web Services, Inc. and/or its affiliates. All rights 's trademarks and trade dress may not be used in connection with any product or service that is not Amazon 's, in any mannerthat is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon . All other trademarks notowned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored Kinesis data Streams Developer GuideTable of ContentsWhat Is Amazon Kinesis data Streams ? .. 1 What Can I Do with Kinesis data Streams ? .. 1 Benefits of Using Kinesis data Streams .. 2 Related Services .. 2 Key Concepts .. 2 High-level Architecture .. 2 Terminology .. 3 data Streams .. 5 Determining the Initial Size of a Kinesis data Stream.. 5 Creating a Stream.

2 6 Updating a Stream.. 6 Producers .. 7 Consumers.. 8.. 8 Limits.. 8 API limits.. 8 Increasing Limits .. 9 Getting Started .. 10 Setting Up .. 10 Sign Up for AWS .. 10 Download Libraries and Tools .. 10 Configure Your Development Environment .. 11 Tutorial: Visualizing Web Traffic .. 11 Kinesis data Streams data Visualization Sample Application .. 11 Prerequisites .. 12 Step 1: Start the Sample Application .. 12 Step 2: View the Components of the Sample Application.. 13 Step 3: Delete Sample Application.. 16 Step 4: Next Steps .. 16 Tutorial: Getting Started Using the CLI .. 17 Install and Configure the AWS CLI .. 17 Perform Basic Stream Operations .. 19 Tutorial: Analyzing Real-Time Stock data .. 23 Prerequisites .. 24 Step 1: Create a data Stream.. 25 Step 2: Create an IAM Policy and User .. 26 Step 3: Download and Build the Implementation Code .. 29 Step 4: Implement the Producer.

3 30 Step 5: Implement the Consumer.. 32 Step 6: (Optional) Extending the Consumer .. 35 Step 7: Finishing Up.. 36 Creating and Managing Streams .. 38 Creating a Stream.. 38 Build the Kinesis data Streams Client .. 38 Create the Stream.. 39 Listing Streams .. 40 Listing Shards .. 41 Retrieving Shards from a Stream.. 42 Deleting a Stream.. 42 Resharding a Stream.. 42 Strategies for Resharding .. 43 Splitting a Shard .. 44 Merging Two Shards .. 45 After Resharding .. 46iiiAmazon Kinesis data Streams Developer GuideChanging the data Retention Period .. 47 Tagging Your Streams .. 48 Tag Basics .. 48 Tracking Costs Using Tagging .. 49 Tag Restrictions .. 49 Tagging Streams Using the Kinesis data Streams Console .. 49 Tagging Streams Using the AWS CLI .. 50 Tagging Streams Using the Kinesis data Streams API .. 50 Monitoring Streams .. 51 Monitoring the Service with CloudWatch.

4 51 Monitoring the Agent with CloudWatch .. 60 Logging Amazon Kinesis data Streams API Calls with AWS CloudTrail .. 61 Monitoring the KCL with CloudWatch .. 65 Monitoring the KPL with CloudWatch .. 73 Controlling Access .. 77 Policy Syntax .. 78 Actions for Kinesis data Streams .. 78 Amazon Resource Names (ARNs) for Kinesis data Streams .. 79 Example Policies for Kinesis data Streams .. 79 Using Server-Side Encryption .. 80 What Is Server-Side Encryption for Kinesis data Streams ? .. 81 Costs, Regions, and Performance Considerations .. 81 How Do I Get Started with Server-Side Encryption? .. 82 Creating and Using User-Generated KMS master Keys .. 83 Permissions to Use User-Generated KMS master Keys .. 84 Verifying and Troubleshooting KMS Key Permissions .. 85 Using Interface VPC Endpoints .. 85 Interface VPC endpoints for Kinesis data Streams .. 85 Using interface VPC endpoints for Kinesis data Streams .. 85 Availability.

5 86 Managing Streams Using the Console .. 86 Writing data to Streams .. 88 Using the KPL.. 88 Role of the KPL .. 89 Advantages of Using the KPL .. 89 When Not to Use the KPL.. 90 Installing the KPL.. 90 Transitioning to Amazon Trust Services (ATS) Certificates for the Kinesis Producer Library .. 90 KPL Supported Platforms .. 90 KPL Key Concepts .. 91 Integrating the KPL with Producer Code .. 92 Writing to your Kinesis data stream.. 94 Configuring the KPL.. 95 Consumer De-aggregation .. 96 Using the KPL with Kinesis data Firehose .. 98 Using the API.. 98 Adding data to a Stream.. 98 Using the Agent .. 102 Prerequisites .. 103 Download and Install the Agent .. 103 Configure and Start the Agent .. 104 Agent Configuration Settings .. 104 Monitor Multiple File Directories and Write to Multiple Streams .. 107 Use the Agent to Pre-process data .. 107 Agent CLI Commands.

6 110 Troubleshooting .. 111 Producer Application is Writing at a Slower Rate Than Expected .. 111ivAmazon Kinesis data Streams Developer GuideUnauthorized KMS master key permission error .. 112 Advanced Topics .. 112 Retries and Rate Limiting .. 112 Considerations When Using KPL Aggregation .. 113 Reading data from Streams .. 115 Using Consumers.. 116 Using the Kinesis Client Library .. 116 Using the Kinesis Client Library .. 130 Using the API.. 134 Using Consumers with Enhanced Fan-Out .. 138 Using the Kinesis Client Library .. 139 Using the API.. 143 Using the AWS Management Console .. 144 Migrating from Kinesis Client Library to .. 145 Migrating the Record Processor .. 145 Migrating the Record Processor Factory .. 149 Migrating the Worker .. 149 Configuring the Amazon Kinesis Client .. 150 Idle Time Removal .. 153 Client Configuration Removals .. 153 Troubleshooting .. 154 Some Kinesis data Streams Records are Skipped When Using the Kinesis Client Library.

7 154 Records Belonging to the Same Shard are Processed by Different Record Processors at theSame Time.. 154 Consumer Application is Reading at a Slower Rate Than Expected .. 155 GetRecords Returns Empty Records Array Even When There is data in the Stream.. 155 Shard Iterator Expires Unexpectedly .. 156 Consumer Record Processing Falling Behind .. 156 Unauthorized KMS master key permission error .. 157 Advanced Topics .. 157 Tracking State .. 157 Low-Latency Processing .. 158 Using AWS Lambda with the Kinesis Producer Library .. 159 Resharding, Scaling, and Parallel Processing .. 159 Handling Duplicate Records .. 160 Recovering from Failures .. 161 Handling Startup, Shutdown, and Throttling .. 162 Document History .. 164 AWS Glossary .. 166vAmazon Kinesis data Streams Developer GuideWhat Can I Do with Kinesis data Streams ?What Is Amazon Kinesis DataStreams?You can use Amazon Kinesis data Streams to collect and process large Streams of data records in realtime.

8 You can create data -processing applications, known as Kinesis data Streams applications. A typicalKinesis data Streams application reads data from a data stream as data records. These applications canuse the Kinesis Client Library, and they can run on Amazon EC2 instances. You can send the processedrecords to dashboards, use them to generate alerts, dynamically change pricing and advertisingstrategies, or send data to a variety of other AWS services. For information about Kinesis data Streamsfeatures and pricing, see Amazon Kinesis data data Streams is part of the Kinesis streaming data platform, along with Kinesis data Firehose, Kinesis Video Streams , and Kinesis data more information about AWS big data solutions, see Big data on AWS. For more information aboutAWS streaming data solutions, see What is Streaming data ?.Topics What Can I Do with Kinesis data Streams ? (p. 1) Benefits of Using Kinesis data Streams (p. 2) Related Services (p. 2) Kinesis data Streams Key Concepts (p.)

9 2) Creating and Updating data Streams (p. 5) Kinesis data Streams Producers (p. 7) Kinesis data Streams Consumers (p. 8) Kinesis data Streams Limits (p. 8)What Can I Do with Kinesis data Streams ?You can use Kinesis data Streams for rapid and continuous data intake and aggregation. The type ofdata used can include IT infrastructure log data , application logs, social media, market data feeds, andweb clickstream data . Because the response time for the data intake and processing is in real time, theprocessing is typically following are typical scenarios for using Kinesis data Streams :Accelerated log and data feed intake and processingYou can have producers push data directly into a stream. For example, push system and applicationlogs and they are available for processing in seconds. This prevents the log data from being lost ifthe front end or application server fails. Kinesis data Streams provides accelerated data feed intakebecause you don't batch the data on the servers before you submit it for metrics and reportingYou can use data collected into Kinesis data Streams for simple data analysis and reporting in realtime.

10 For example, your data -processing application can work on metrics and reporting for systemand application logs as the data is streaming in, rather than wait to receive batches of Kinesis data Streams Developer GuideBenefits of Using Kinesis data StreamsReal-time data analyticsThis combines the power of parallel processing with the value of real-time data . For example,process website clickstreams in real time, and then analyze site usability engagement using multipledifferent Kinesis data Streams applications running in stream processingYou can create Directed Acyclic Graphs (DAGs) of Kinesis data Streams applications and datastreams. This typically involves putting data from multiple Kinesis data Streams applications intoanother stream for downstream processing by a different Kinesis data Streams of Using Kinesis data StreamsAlthough you can use Kinesis data Streams to solve a variety of streaming data problems, a common useis the real-time aggregation of data followed by loading the aggregate data into a data warehouse ormap-reduce is put into Kinesis data Streams , which ensures durability and elasticity.