Amazon Kinesis Data Streams

1 Amazon Kinesis data StreamsDeveloper GuideAmazon Kinesis data Streams Developer GuideAmazon Kinesis data Streams : Developer GuideCopyright 2018 Amazon Web Services, Inc. and/or its affiliates. All rights 's trademarks and trade dress may not be used in connection with any product or service that is not Amazon 's, in any mannerthat is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon . All other trademarks notowned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored Kinesis data Streams Developer GuideTable of ContentsWhat Is Amazon Kinesis data Streams ?

2 1 What Can I Do with Kinesis data Streams ? .. 1 Benefits of Using Kinesis data Streams .. 2 Related Services .. 2 Key Concepts .. 2 High-level Architecture .. 2 Terminology .. 3 data Streams .. 5 Determining the Initial Size of a Kinesis data Stream.. 5 Creating a Stream.. 6 Updating a Stream.. 6 Producers .. 7 Consumers.. 8.. 8 Limits.. 8 API limits.. 8 Increasing Limits .. 9 Getting Started .. 10 Setting Up .. 10 Sign Up for AWS .. 10 Download Libraries and Tools.

3 10 Configure Your Development Environment .. 11 Tutorial: Visualizing Web Traffic .. 11 Kinesis data Streams data Visualization Sample Application .. 11 Prerequisites .. 12 Step 1: Start the Sample Application .. 12 Step 2: View the Components of the Sample Application.. 13 Step 3: Delete Sample Application.. 16 Step 4: Next Steps .. 16 Tutorial: Getting Started Using the CLI .. 17 Install and Configure the AWS CLI .. 17 Perform Basic Stream Operations .. 19 Tutorial: Analyzing Real-Time Stock data .

4 23 Prerequisites .. 24 Step 1: Create a data Stream.. 25 Step 2: Create an IAM Policy and User .. 26 Step 3: Download and Build the Implementation Code .. 29 Step 4: Implement the Producer .. 30 Step 5: Implement the Consumer.. 32 Step 6: (Optional) Extending the Consumer .. 35 Step 7: Finishing Up.. 36 Creating and Managing Streams .. 38 Creating a Stream.. 38 Build the Kinesis data Streams Client .. 38 Create the Stream.. 39 Listing Streams .. 40 Listing Shards.

5 41 Retrieving Shards from a Stream.. 42 Deleting a Stream.. 42 Resharding a Stream.. 42 Strategies for Resharding .. 43 Splitting a Shard .. 44 Merging Two Shards .. 45 After Resharding .. 46iiiAmazon Kinesis data Streams Developer GuideChanging the data Retention Period .. 47 Tagging Your Streams .. 48 Tag Basics .. 48 Tracking Costs Using Tagging .. 49 Tag Restrictions .. 49 Tagging Streams Using the Kinesis data Streams Console .. 49 Tagging Streams Using the AWS CLI.

6 50 Tagging Streams Using the Kinesis data Streams API .. 50 Monitoring Streams .. 51 Monitoring the Service with CloudWatch .. 51 Monitoring the Agent with CloudWatch .. 60 Logging Amazon Kinesis data Streams API Calls with AWS CloudTrail .. 61 Monitoring the KCL with CloudWatch .. 65 Monitoring the KPL with CloudWatch .. 73 Controlling Access .. 77 Policy Syntax .. 78 Actions for Kinesis data Streams .. 78 Amazon Resource Names (ARNs) for Kinesis data Streams .. 79 Example Policies for Kinesis data Streams .

7 79 Using Server-Side Encryption .. 80 What Is Server-Side Encryption for Kinesis data Streams ? .. 81 Costs, Regions, and Performance Considerations .. 81 How Do I Get Started with Server-Side Encryption? .. 82 Creating and Using User-Generated KMS master Keys .. 83 Permissions to Use User-Generated KMS master Keys .. 84 Verifying and Troubleshooting KMS Key Permissions .. 85 Using Interface VPC Endpoints .. 85 Interface VPC endpoints for Kinesis data Streams .. 85 Using interface VPC endpoints for Kinesis data Streams .

8 85 Availability .. 86 Managing Streams Using the Console .. 86 Writing data to Streams .. 88 Using the KPL.. 88 Role of the KPL .. 89 Advantages of Using the KPL .. 89 When Not to Use the KPL.. 90 Installing the KPL.. 90 Transitioning to Amazon Trust Services (ATS) Certificates for the Kinesis Producer Library .. 90 KPL Supported Platforms .. 90 KPL Key Concepts .. 91 Integrating the KPL with Producer Code .. 92 Writing to your Kinesis data stream.. 94 Configuring the KPL.

9 95 Consumer De-aggregation .. 96 Using the KPL with Kinesis data Firehose .. 98 Using the API.. 98 Adding data to a Stream.. 98 Using the Agent .. 102 Prerequisites .. 103 Download and Install the Agent .. 103 Configure and Start the Agent .. 104 Agent Configuration Settings .. 104 Monitor Multiple File Directories and Write to Multiple Streams .. 107 Use the Agent to Pre-process data .. 107 Agent CLI Commands .. 110 Troubleshooting .. 111 Producer Application is Writing at a Slower Rate Than Expected.

10 111ivAmazon Kinesis data Streams Developer GuideUnauthorized KMS master key permission error .. 112 Advanced Topics .. 112 Retries and Rate Limiting .. 112 Considerations When Using KPL Aggregation .. 113 Reading data from Streams .. 115 Using Consumers.. 116 Using the Kinesis Client Library .. 116 Using the Kinesis Client Library .. 130 Using the API.. 134 Using Consumers with Enhanced Fan-Out .. 138 Using the Kinesis Client Library .. 139 Using the API.. 143 Using the AWS Management Console.