Deep Dive Amazon Kinesis - d0.awsstatic.com

1 Deep Dive Amazon KinesisIan Meyers, Principal Solution Architect- Amazon Web ServicesAnalyticsAmazon KinesisManaged Service for Real Time Big Data ProcessingCreate Streams to Produce & Consume DataElastically Add and Remove Shards for PerformanceUse Kinesis Worker Library, AWS Lambda, Apache Spark and Apache Storm to Process DataIntegration with S3, Redshift and Dynamo DBComputeStorageAWS Global InfrastructureDatabaseApp ServicesDeployment & AdministrationNetworkingAnalyticsData [Machine Learning]AWS [Aggregate & De-Duplicate]Data SourcesData SourcesData [Metric Extraction] [Sliding Window Analysis]Data SourcesShard 1 Shard 2 Shard NAvailability ZoneAvailability ZoneAmazon Kinesis DataflowAvailability ZoneBilling AuditorsIncremental Bill ComputationMetering ArchiveBilling ManagementServiceExample Architecture -MeteringStreamsNamed Event Streams of DataShardsYou scale Kinesis streams by adding or removing ShardsEach Shard ingests up to 1MB/sec of data and up to 1000 TPSAll data is stored for 24 hoursPartition KeyIdentifier used for Ordered Delivery & Partitioning of Data across ShardsSequenceNumber of an event as assigned by KinesisAmazon Kinesis ComponentsGetting Data InProducers use a PUT call to store data in a StreamA Partition Key is used to distribute the PUTs across ShardsA unique Sequence #is returned to the Producer for each EventData can be ingested at 1MB/second or 1000

2 Transactions/second per Shard1MB / EventKinesis -Ingesting Fast Moving DataNative Code Module to perform efficient writes to Multiple Kinesis StreamsC++/BoostAsynchronous ExecutionConfigurable Aggregation of EventsIntroducing the Kinesis Producer LibraryMy ApplicationKPL DaemonPutRecord(s) Kinesis StreamKinesis StreamKinesis StreamKinesis StreamAsyncKPL AggregationMy ApplicationKPL DaemonPutRecord(s) Kinesis StreamKinesis StreamKinesis StreamKinesis StreamAsync1MB Max Event SizeAggregate100k20k500k200k40k20k40k500 k100k200k20k40k40k20kProtobufHeaderProto bufFooterApache FlumeSource & Partitioning & Log4 NetIncluded in Kinesis SamplesKinesis Ecosystem -IngestKinesisGetting Data OutKCL Libraries available for Java, Ruby, Node, Go, and a Multi-Lang Implementation with Native Python supportAll State Management in Dynamo DBKinesis Client LibraryDynamoDBClient library for fault-tolerant, at least-once, real-time processing Kinesis Client Library (KCL)

3 Simplifies reading from the stream by abstracting your code from individual shardsAutomatically starts a Worker Thread for each ShardIncreases and decreases Thread count as number of Shards changesUses checkpoints to keep track of a Thread s location in the stream Restarts Threads & Workers if they failConsuming Data - Kinesis Enabled ApplicationsAnalytics Tooling Integration ( )S3 Batch Write Files for Archive into S3 Sequence Based File NamingRedshiftOnce Written to S3, Load to RedshiftManifest SupportUser Defined TransformersDynamoDBBatchPutAppend to TableUser Defined TransformersElasticSearchAutomatically index Stream ContentsKinesis ConnectorsS3 Dynamo DBRedshiftKinesisElasticSearchConnectors ArchitectureApache StormKinesis SpoutAutomatic Checkpointingwith Ecosystem -StormStormKinesisApache SparkDStreamReceiver runs KCLOne DStreamper ShardCheckpointedvia KCLS park Natively Available on EMREMRFS overlay on HDFSAMI Ecosystem -SparkDistributed Event Processing PlatformStateless JavaScript & Java functions run against an Event StreamAWS SDK Built InConfigure RAM and Execution TimeoutFunctions automatically invoked against a ShardCommunity libraries for Python & GoAccess to underlying filesystemfor read/writeCall other Lambda FunctionsConsuming Data -AWS

4 LambdaKinesisShard 1 Shard 2 Shard 3 Shard 4 Shard nWhy Kinesis ? DurabilityRegional ServiceSynchronous Writes to Multiple AZ sExtremely High Durability?May be in-memory for PerformanceRequirement to understand Disk Sync SemanticsUser Managed ReplicationReplication Lag -> RPOWhy Kinesis ? PerformancePerform continual processing on streaming big data. Processing latencies fall to a <1 second, compared with the minutes or hours associated with batch processing?Processing latencies < 1 secondBased on CPU & Disk PerformanceCluster Interruption -> Processing OutageWhy Kinesis ? AvailabilityRegional ServiceSynchronous Writes to Multiple AZ sExtremely High DurabilityAZ, Networking, & Chain Server Issues Transparent to Producers & Consumers?Many Depend on a CP DatabaseLost Quorum can result in failure/inconsistency of the clusterHighest Availability is determined by Availability of Cross-AZ Links or Availability of an AZWhy Kinesis ?

5 OperationsManaged service for real-time streaming data collection, processing and analysis. Simply create a new stream, set the desired level of capacity, and let the service handle the rest?Build InstancesInstall SoftwareOperate ClusterManage Disk SpaceManage ReplicationMigrate to new Stream on Scale UpWhy Kinesis ? ElasticitySeamlessly scale to match your data throughput rate and volume. You can easily scale up to gigabytes per second. The service will scale up or down based on your operational or business needs?Fixed Partition Count up FrontMaximum Performance ~ 1 Partition/Core | MachineConvert from 1 Stream to Another to ScaleApplication ReconfigurationScaling Kinesis ? CostCost-efficient for workloads of any scale. You can get started by provisioning a small stream, and pay low hourly rates only for what you Up/Down Dynamically$.

6 015/Hour/1MB?Run your Own EC2 InstancesMulti-AZ Configuration for increased DurabilityUtiliseInstance AutoScalingon Worker Lag from HEAD with Custom MetricsWhy Kinesis ? CostPrice Dropped on 2ndJune 2015, Restructured to support KPLOld Pricing: $.028 / 1M Records PUTNew Pricing: $.014/1M 25KB Payload Units UnitsCostUnitsCostShards50$55825$279 PutRecords4,320M Records$ ,648M Payload Units$ $ $ : 50,000 Events / Second, 512B / Event = MB/SecondOld PricingNew Pricing + KPLK inesis Consumer Application Best PracticesTolerate Failure of: Threads Consider Event Serialisationissues and Lease Stealing; Hardware AutoScalingmay add nodes as neededScale Consumers up and down as the number of Shards increase or decreaseDon t store data in memory in the workers. Use an elastic data store such as Dynamo DB for StateElastic Beanstalk provides all Best Practices in a simple to deploy, multi-version Application ContainerKCL will automatically redistribute Workers to use new InstancesLogic implemented in Lambda doesn t require any Servers at all!

7 Managing Application StateConsumer Local State Anti-PatternConsumer binds to a configured number of PartitionsConsumer stores the state of a data structure, as defined by the event stream, on local storageRead API can access that local storage as a shard of the overall database?ConsumerConsumerPartition 1 Partition ..Partition P/2 Partition P/2+1 Partition ..Partition PLocal DiskLocal DiskLocal StoragePartitions StoragePartitions P/2+ APIC onsumer Local State Anti-PatternBut what happens when an instance fails??ConsumerConsumerPartition 1 Partition ..Partition P/2 Partition P/2+1 Partition ..Partition PLocal DiskLocal DiskLocal StoragePartitions StoragePartitions P/2+ APIC onsumer Local State Anti-PatternA new consumer process starts up for the required PartitionsConsumer must read from the beginning of the Stream to rebuild local storageComplex, error prone, user constructed softwareLong StartupTime?

8 ConsumerConsumerPartition 1 Partition ..Partition P/2 Partition P/2+1 Partition ..Partition PLocal DiskLocal DiskLocal StoragePartitions StoragePartitions P/2+ APIT0 THeadExternal Highly Available State Best PracticeConsumerConsumerShard 1 Shard ..Shard S/2 Shard S/2+ SRead APIC onsumer binds to a even number of Shards based on number of ConsumersConsumer stores the state in Dynamo DBDynamo DB is Highly Available, Elastic & DurableRead API can access Dynamo DBDynamoDBExternal Highly Available State Best PracticeConsumer binds to a even number of Shards based on number of ConsumersConsumer stores the state in Dynamo DBDynamo DB is Highly Available, Elastic & DurableRead API can access Dynamo DBRead APID ynamoDBShard 1 Shard ..Shard S/2 Shard S/2+ SConsumerConsumerExternal Highly Available State Best PracticeRead APID ynamoDBAWS LambdaShard 1 Shard.

9 Shard S/2 Shard S/2+ SConsumer binds to a even number of Shards based on number of ConsumersConsumer stores the state in Dynamo DBDynamo DB is Highly Available, Elastic & DurableRead API can access Dynamo DBIdempotencyProperty of a system whereby the repeated application of a function on a single input results in the same end state of the Once ProcessingIdempotency Writing DataThe Kinesis SDK & KPL may retry PUT in certain circumstancesKinesis Record acknowledged with a Sequence Number isdurable to Multiple Availability there could be a duplicate entryMy ApplicationPutRecord(s)403 (Endpoint Redirect)StorageIdempotency Writing DataThe Kinesis SDK & KPL may retry PUT in certain circumstancesKinesis Record acknowledged with a Sequence Number isdurable to Multiple Availability there could be a duplicate entryMy ApplicationPutRecord(s)PutRecord(s)200 (OK)403 (Endpoint Redirect)StorageStorageWrite123 Write123 Coming Rolling IdempotencyCheckKinesis will manage a rolling time window of Record ID s in Dynamo DBRecord ID s are User BasedDuplicates in storage tier will be acknowledged as SuccessfulMy ApplicationPutRecord(s)StorageDynamoDBX Hour Rolling WindowIdempotency Rolling IdempotencyCheckKinesis will manage a rolling time window of Record ID s in Dynamo DBRecord ID s are User BasedDuplicates in storage tier will be acknowledged as SuccessfulMy ApplicationPutRecord(s)

10 StorageWrite Record ID OKDynamoDBX Hour Rolling WindowIdempotency Rolling IdempotencyCheckKinesis will manage a rolling time window of Record ID s in Dynamo DBRecord ID s are User BasedDuplicates in storage tier will be acknowledged as SuccessfulMy ApplicationPutRecord(s)403 (Endpoint Redirect)StorageWrite Record ID OKDynamoDBX Hour Rolling WindowIdempotency Rolling IdempotencyCheckKinesis will manage a rolling time window of Record ID s in Dynamo DBRecord ID s are User BasedDuplicates in storage tier will be acknowledged as SuccessfulMy ApplicationPutRecord(s)PutRecord(s)403 (Endpoint Redirect)StorageStorageWrite Record ID OKWrite Record IDDynamoDBX Hour Rolling WindowConditionCheckFailedExceptionIdemp otency Rolling IdempotencyCheckKinesis will manage a rolling time window of Record ID s in Dynamo DBRecord ID s are User BasedDuplicates in storage tier will be acknowledged as SuccessfulMy ApplicationPutRecord(s)PutRecord(s)200 (OK)403 (Endpoint Redirect)StorageStorageWrite Record ID OKWrite Record IDDynamoDBX Hour Rolling WindowConditionCheckFailedExceptionEasy AdministrationReal-time DurabilityHigh Throughput.