Transcription of Data Storage Infrastructure at Facebook - …
{{id}} {{{paragraph}}}
Example: air traffic controller
Data Storage Infrastructure at Facebook - …
data Storage Infrastructure at FacebookSpring 2018 Cleveland State UniversityCIS 601 PresentationYi DongInstructor: Dr. Chung Outline Strategy of data Storage , processing, and log collection data flow from the source to the data warehouse Storage systems and optimization data discovery and analysis Challenges in resource sharing Facebook s Architecture Facebook s ArchitecturePHPHipHop compilerScribeThriftHadoop HbaseHayStackHiveMySQLM emcached Part 1: Strategy for data Storage , Processing, Log collection Apache Hadoop Apache Hive Scribe Hadoop, Why? Scalability Able to process multi petabyte datasets Fault Tolerance Node failure is expected everyday Number of nodes is not constant High Availability User can access from nearest node Cost Efficiency Open source Use commodity hardware as a node in Hadoop clusters Eliminates particular technology dependency Hadoop Architecture HDFS (Hadoop Distributed File System) Map-Reduce Infrastructure Hive SQL-like analysis tool (HiveQL) on top of Hadoop Dramatically improve the productivity and usage for Hadoop With Hive, users without programming experience can use Hadoop for their work Without Hive, one basic Hadoop data manipulation, like GROUP BY will take >100 lines of Java/Python code Even worse, if the programmer does not have database knowledge, the code will likely use sub-optimal algorithm, often it is pretty sub-optimal Hiv
Outline Strategy of data storage, processing, and log collection Data flow from the source to the data warehouse Storage systems and optimization Data discovery and analysis Challenges in resource sharing
Loading..
Tags:
Information
Domain:
Source:
Link to this page:
Please notify us if you found a problem with this document:
Related search queries
Introduction to Data Warehousing, Data Warehouse, Application, Querying the Data Warehouse with, KNJohnson Data Testing on Business, Data Testing on Business Intelligence, Warehouse Management System, Warehouse, HighJump Warehouse Advantage, SCDigest Best Practices Warehouse Returns, Data, Iapmo research and testing, inc