Kafka: The Definitive Guide - Huihoo

1 Neha Narkhede, Gwen Shapira & Todd Palino KafkaThe Defi nitive GuideREAL TIME data AND STREAM PROCESSING AT SCALEC ompliments ofGet Started With Apache kafka Today Thoroughly tested and quality assured Additional client support, including Python, C/C++ and .NET Easy upgrade path to Confluent EnterpriseCONFLUENT OPEN SOURCE CONNECTORSCLIENTSSCHEMA REGISTRYREST PROXYS tart today at A 100% open source Apache kafka distribution for building robust streaming Narkhede, Gwen Shapira, and Todd PalinoKafka: The Definitive GuideReal-Time data and Stream Processing at ScaleBostonFarnhamSebastopolTokyoBeijing BostonFarnhamSebastopolTokyoBeijing978-1 -491-99065-0[LSI] kafka : The Definitive Guideby Neha Narkhede, Gwen Shapira, and Todd PalinoCopyright 2017 Neha Narkhede, Gwen Shapira, Todd Palino. All rights in the United States of by O Reilly Media, Inc.

2 , 1005 Gravenstein Highway North, Sebastopol, CA Reilly books may be purchased for educational, business, or sales promotional use. Online editions arealso available for most titles ( ). For more information, contact our corporate/insti tutional sales department: 800-998-9938 or Shannon CuttProduction Editor: Shiny KalapurakkelCopyeditor: Christina EdwardsProofreader: Amanda KerseyIndexer: WordCo Indexing Services, Designer: David FutatoCover Designer: Karen MontgomeryIllustrator: Rebecca DemarestJuly 2017: First EditionRevision History for the First Edition2017-07-07: First ReleaseSee for release O Reilly logo is a registered trademark of O Reilly Media, Inc. kafka : The Definitive Guide , the coverimage, and related trade dress are trademarks of O Reilly Media, the publisher and the authors have used good faith efforts to ensure that the information andinstructions contained in this work are accurate, the publisher and the authors disclaim all responsibilityfor errors or omissions, including without limitation responsibility for damages resulting from the use ofor reliance on this work.

3 Use of the information and instructions contained in this work is at your ownrisk. If any code samples or other technology this work contains or describes is subject to open sourcelicenses or the intellectual property rights of others, it is your responsibility to ensure that your usethereof complies with such licenses and/or of ContentsForeword.. xiiiPreface.. kafka .. 1 Publish/Subscribe Messaging 1 How It Starts 2 Individual Queue Systems 3 Enter kafka 4 Messages and Batches 4 Schemas 5 Topics and Partitions

4 5 Producers and Consumers 6 Brokers and Clusters 7 Multiple Clusters 8 Why kafka ? 10 Multiple Producers 10 Multiple Consumers 10 Disk-Based Retention 10 Scalable 10 High Performance 11 The data Ecosystem

5 11 Use Cases 12 kafka s Origin 14 LinkedIn s Problem 14 The Birth of kafka 15 Open Source 15 The Name 16vGetting Started with kafka kafka .

6 17 First Things First 17 Choosing an Operating System 17 Installing Java 17 Installing Zookeeper 18 Installing a kafka Broker 20 Broker Configuration 21 General Broker

7 21 Topic Defaults 24 Hardware Selection 28 Disk Throughput 29 Disk Capacity 29 Memory 29 Networking 30 CPU 30 kafka in the Cloud 30 kafka Clusters 31 How Many Brokers?

8 32 Broker Configuration 32OS Tuning 32 Production Concerns 36 Garbage Collector Options 36 Datacenter Layout 37 Colocating Applications on Zookeeper 37 Summary Producers: Writing Messages to kafka .

9 41 Producer Overview 42 Constructing a kafka Producer 44 Sending a Message to kafka 46 Sending a Message Synchronously 46 Sending a Message Asynchronously 47 Configuring Producers 48 Serializers

10 52 Custom Serializers 52 Serializing Using Apache Avro 54 Using Avro Records with kafka 56 Partitions 59 Old Producer APIs 61 Summary 62vi | Table of Consumers: Reading data from kafka .. 63 kafka Consumer Concepts 63 Consumers and Consumer Groups 63 Consumer Groups and Partition Rebalance 66 Creating a kafka Consumer 68 Subscribing to Topics 69 The Poll Loop 70 Configuring Consumers