Transcription of The Data Engineering Cookbook - Darwin Pricing
1 The data Engineering CookbookMastering The Plumbing Of data ScienceAndreas KretzMay 18, Introduction91 How To Use This Cookbook102 data Engineer vs data data Scientist .. data Engineer .. Who Companies Need ..13II Basic data Engineering Skills143 Learn To Code154 Get Familiar With Github165 Agile Development Why is agile so important? .. Agile rules I learned over the years available .. the method making a difference? .. problem with outsourcing .. is king: A lesson from Elon Musk.
2 You really can be agile .. Agile Techniques ..206 Learn how a Computer CPU,RAM,GPU,HDD .. Differences between PCs and Servers ..217 Computer Networking - data ISO/OSI Model .. IP Subnetting .. Switch, Level 3 Switch .. Router .. Firewalls ..228 Security and SSL Public & Private Key Certificates .. What is a certificate authority .. JAva Web Tokens .. GDPR regulations .. Privacy by design ..239 OS Basics .. Shell scripting.
3 Cron jobs .. Packet management ..2410 The AWS,Azure, IBM, Google Cloud basics .. cloud vs on premise .. up & downsides .. Security ..2511 Security Zone How to secure a multi layered application .. Cluster security with Kerberos .. Kerberos Tickets ..2612 Stream Three methods of streaming available .. At Least Once .. At Most Once .. Exactly Once .. Check The Tools! ..2813 Big What is big data and where is the difference to data science and dataanalytics?
4 The 4Vs of Big data available .. Why Big data ? available .. Planning is Everything .. The Problem With ETL .. Scaling Up .. Scaling Out .. Please Don t go Big data ..3414 data Warehouse vs data Lake3515 Hadoop Platforms What is Hadoop .. What makes Hadoop so popular? available .. Hadoop Ecosystem Components .. Hadoop Is Everywhere? .. SHOULD YOU LEARN HADOOP? ..40 How does a Hadoop System architecture look like ..40 What tools are usually in a with Hadoop Cluster.
5 How to select Hadoop Cluster Hardware ..4016 Is ETL still relevant for Analytics?4117 What is docker and what do you use it for available .. Don t Mess Up Your System .. Preconfigured Images .. Take It With You .. Kubernetes Container Deployment .. How to create, start,stop a Container .. Docker micro services? .. Kubernetes .. Why and how to do Docker container orchestration ..4418 REST HTTP Post/Get .. API Design .. Implementation .. OAuth security ..4519 SQL Databases.
6 Database Design .. SQL Queries .. Stored Procedures .. ODBC/JDBC Server Connections .. NoSQL Stores .. KeyValue Stores (HBase) .. Document Store HDFS available .. Document Store MongoDB .. Hive Warehouse .. Impala .. Kudu .. Time Series Databases .. MPP Databases (Greenplum) ..4820 data Processing / Analytics - MapReduce .. How does MapReduce work available .. Example .. What is the limitation of MapReduce? available .. Apache Spark .. What is the difference to MapReduce?
7 Available .. How does Spark fit to Hadoop? available .. Where s the difference? .. Spark and Hadoop is a perfect fit .. Spark on YARN: .. My simple rule of thumb: .. Available Languages available .. How to do stream processing .. How to do batch processing .. How does Spark use data from Hadoop available .. What is a RDD and what is a DataFrame? .. Spark coding with Scala .. Spark coding with Python .. How and why to use SparkSQL? .. Machine Learning on Spark?
8 (Tensor Flow) .. MLlib: .. Spark Setup available .. Resource Management available ..59521 Apache Why a message queue tool? .. Kakfa architecture .. What are topics .. What does Zookeeper have to do with Kafka .. How to produce and consume messages ..6022 Machine Training and Applying models .. What is deep learning .. How to do Machine Learning in production available .. Why machine learning in production is harder then you think Models Do Not Work Forever.
9 Where The Platforms That Support This? .. Training Parameter Management .. What s Your Solution? .. How to convince people machine learning works available .. Rules, No Physical Models .. Have The data . USE IT! .. is Stronger Than Opinions ..6523 data Android & IOS .. How to design APIs for mobile apps .. How to use Webservers to display content .. Tomcat .. Jetty .. NodeRED .. React .. Business Intelligence Tools .. Tableau .. PowerBI .. Quliksense.
10 Identity & Device Management .. What is a digital twin? .. Active Directory ..676 III Building A data Platform Example6824 My Big data Platform Ingest .. Analyse / Process .. Store .. Display ..7225 Lambda Batch Processing .. Stream Processing .. Should you do stream or batch processing? .. Lambda Architecture Alternative .. Kappa Architecture .. Kappa Architecture with Kudu ..7526 Thoughts On Choosing The Target Cloud vs On-Premise .. Cloud Native or Independent Vendors.
