Developer Guide API Version 2013-12-02

1 Amazon KinesisDeveloper GuideAPI Version 2013-12-02 Amazon kinesis : Developer GuideCopyright 2014 Amazon Web Services, Inc. and/or its affiliates. All rights following are trademarks of Amazon Web Services, Inc.: Amazon, Amazon Web Services Design, AWS, Amazon CloudFront,Cloudfront, Amazon DevPay, DynamoDB, ElastiCache, Amazon EC2, Amazon Elastic Compute Cloud, Amazon Glacier, Kindle, KindleFire, AWS Marketplace Design, Mechanical Turk, Amazon Redshift, Amazon Route 53, Amazon S3, Amazon VPC. In addition, graphics, logos, page headers, button icons, scripts, and service names are trademarks, or trade dress of Amazon inthe and/or other countries. Amazon's trademarks and trade dress may not be used in connection with any product or service thatis not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connectedto, or sponsored by kinesis Developer GuideTable of ContentsWhat Is Amazon kinesis ?

2 1 What Can I Do with Amazon kinesis ?.. 1 Benefits of Using Amazon 2 Key 2 High-level 3 Sizes and 5 Getting 6 Before You 6 Libraries and 7 Overview of Using Amazon 7 How Do I Size an Amazon kinesis Stream?.. 8 Step 1: Create an Amazon kinesis 8 Step 2: Configure the Producers that Submit Data to the Amazon kinesis 10 Step 3: Build a Amazon kinesis 10 Step 4: Operate the Amazon kinesis 11 Where Do I Go From Here?.. 11 Visualizing Web Traffic with the Amazon kinesis Sample 12 Amazon kinesis Data Visualization Sample 12 Before You 13 Running the Sample 13 Viewing the Results of the Sample 14 Deleting Sample Application 16 Using the Amazon kinesis Service 17 Create the Amazon kinesis 17 Create the 18 List All Streams in an AWS 19 Add Data to a 20 Sequence 20 Partition Key .. 20 Retrieve the Shards from a 21 Get Data from the Shards in a 22 Using Shard 22 Using GetRecords().

3 23 Adapting to a 25 Resharding a 26 Split a 26 Merge Two 27 Wait after Resharding for Stream to Become Active .. 28 Data Routing, Data Persistence, and Shard State after a 29 Strategies for 30 Delete the 30 Related 31 Developing Record Consumer Applications with the Amazon kinesis Client Library .. 32 Overview .. 32 Amazon kinesis Client Library (KCL).. 32 kinesis 33 Building a kinesis 33 Role of the Amazon kinesis Client Library .. 33 Amazon kinesis Application 33 Advanced Topics and 37 kinesis Application State and Amazon 38 Resharding, Scaling, and Parallel 38 Failover and Recovery .. 39 API Version 2013-12-02iiiAmazon kinesis Developer GuideStartup, Shutdown, and 40 Access Control with 42 Amazon kinesis 42 Amazon Resource Names (ARNs) for Amazon 43 Example Policies for Amazon 43 Logging Amazon kinesis API Calls Using AWS 47 Amazon kinesis Information in 47 Understanding Amazon kinesis Log File 48 History.

4 51 API Version 2013-12-02ivAmazon kinesis Developer GuideWhat Is Amazon kinesis ?Amazon kinesis is a managed service that scales elastically for real-time processing of streaming section introduces the Amazon kinesis What Can I Do with Amazon kinesis ? (p. 1) Benefits of Using Amazon kinesis (p. 2) Key Concepts (p. 2)Amazon kinesis takes in large streams of data records that can then be consumed in real time by multipledata-processing applications that can be run on Amazon Elastic Compute Cloud (Amazon EC2) data-processing applications use the Amazon kinesis Client Library and are called "Amazon Kinesisapplications." For more information about Amazon EC2, see the Amazon Elastic Compute Cloud GettingStarted kinesis applications read data from the Amazon kinesis stream and process the read data inreal time. The processed records can be emitted to dashboards, used to generate alerts, dynamicallychange pricing and advertising strategies, or emit data to a variety of other Amazon big data servicessuch as Amazon Simple Storage Service (Amazon S3), Amazon Elastic MapReduce (Amazon EMR), orAmazon Redshift.

5 Amazon kinesis applications can also emit data into another Amazon kinesis stream,enabling more complex data processing. For information about Amazon kinesis service highlights andpricing, go to Amazon Can I Do with Amazon kinesis ?Amazon kinesis takes in large streams of data for processing in real time. The most common AmazonKinesis use case scenario is rapid and continuous data intake and type of data usedin an Amazon kinesis use case includes IT infrastructure log data, application logs, social media, marketdata feeds, web clickstream data, and more. Because the response time for the data intake and processingis in real time, the processing is typically kinesis enables sophisticated streaming data processing, because one Amazon kinesis applicationmay emit Amazon kinesis stream data into another Amazon kinesis stream. Near-real-time aggregationof data enables processing logic that can extract complex key performance indicators and metrics fromthat data.

6 For example, complex data-processing graphs can be generated by emitting data from multipleAmazon kinesis applications to another Amazon kinesis stream for downstream processing by a differentAmazon kinesis Version 2013-12-021 Amazon kinesis Developer GuideWhat Can I Do with Amazon kinesis ?The following are typical scenarios for using Amazon kinesis . Accelerated log and data feed intake and processing: Using Amazon kinesis you can have producerspush data directly into an Amazon kinesis stream. For example, system and application logs can besubmitted to Amazon kinesis and be available for processing in seconds. This prevents the log datafrom being lost if the front end or application server fails. Amazon kinesis provides accelerated datafeed intake because you are not batching up the data on the servers before you submit them for intake. Real-time metrics and reporting:You can use data ingested into Amazon kinesis for simple data analysisand reporting in real time.

7 For example, metrics and reporting for system and application logs ingestedinto the Amazon kinesis stream are available in real time. This enables data-processing applicationlogic to work on data as it is streaming in, rather than wait for data batches to be sent to thedata-processing applications. Real-time data analytics: Amazon kinesis enables real-time analytics of streaming big data, combiningthe power of parallel processing with the value of real-time data. For example, website clickstreamscan be ingested in real time, and then site usability engagement can be analyzed by many differentAmazon kinesis client applications running in parallel. Complex stream processing: Lastly, Amazon kinesis enables you to create Directed Acyclic Graphs(DAGs) of Amazon kinesis applications and data streams. This scenario typically involves emittingdata from multiple Amazon kinesis applications to another Amazon kinesis stream for downstreamprocessing by a different Amazon kinesis of Using Amazon KinesisWhile Amazon kinesis can be used to solve a variety of streaming data problems, a common use is thereal-time aggregation of data followed by loading the aggregate data into a data warehouse or can be taken into Amazon kinesis streams, which will ensure durability and elasticity.

8 The delaybetween the time a record is added to the stream and the time it can be retrieved (put-to-get delay) isless than 10 seconds in other words, Amazon kinesis applications can start consuming the data fromthe stream less than 10 seconds after the data is managed service aspect of Amazon Kinesisrelieves customers of the operational burden of creating and running a data intake pipeline. Customerscan create streaming map-reduce type applications, and the elasticity of the Amazon kinesis serviceenables customers to scale the stream up or down, ensuring they never lose data records prior to Amazon kinesis applications can consume data from an Amazon kinesis stream, so that multipleactions, like archiving and processing, can take place concurrently and independently. For example, twoAmazon kinesis applications can read data from the same Amazon kinesis first AmazonKinesis application calculates running aggregates and updates an Amazon DynamoDB table, and thesecond Amazon kinesis application compresses and archives data to a data store like Amazon DynamoDB table with running aggregates is then read by a dashboard for Amazon kinesis Client Library enables fault-tolerant consumption of data from the Amazon Kinesisstream and provides scaling support for Amazon kinesis ConceptsThis section describes the concepts and terminology you need to understand to use Amazon Kinesiseffectively in the following Version 2013-12-022 Amazon kinesis Developer GuideBenefits of Using Amazon kinesis High-level Architecture (p.)

9 3) Terminology (p. 3) Sizes and Limits (p. 5)Amazon kinesis High-level ArchitectureThe following diagram illustrates the high-level architecture of Amazon kinesis TerminologyThe following terms describe concepts you need to understand in order to use Amazon kinesis RecordData records are the units of data that are stored in an Amazon kinesis stream. Data records are composedof a sequence number, a partition key, and a data blob, which is an un-interpreted, immutable sequenceof bytes. The Amazon kinesis service does not inspect, interpret, or change the data in the blob in anyway. The maximum size of a data blob (the data payload after Base64-decoding) is 50 kilobytes (KB).StreamA stream is an ordered sequence of data records. Each record in the stream has a sequencenumber (p. 4) that is assigned by the service. The data records in the stream are distributed intoshards (p. 3).ShardA shard is a uniquely identified group of data records in an Amazon kinesis stream.

10 A stream is composedof multiple shards, each of which provides a fixed unit of capacity. Each open shard can support up to 5read transactions per second, up to a maximum total of 2 MB of data read per second. Each shard cansupport up to 1000 write transactions per second, up to a maximum total of 1 MB data written per data capacity of your stream is a function of the number of shards that you specify for the total capacity of the stream is the sum of the capacities of its your data rate increases, then you just add more shards to increase the size of your stream. Similarly,you can remove shards if the data rate Version 2013-12-023 Amazon kinesis Developer GuideHigh-level ArchitecturePartition KeyThe partition key is used to group data by shard within the stream. Amazon kinesis segregates the datarecords belonging to a data stream into multiple shards, using the partition key associated with each datarecord to determine which shard a given data record belongs to.