Gorilla: A Fast, Scalable, In-Memory Time Series Database

1 gorilla : A Fast, Scalable, In-Memory Time Series DatabaseTuomas Pelkonen Scott Franklin Justin TellerPaul Cavallaro Qi Huang Justin Meza Kaushik VeeraraghavanFacebook, Park, CAABSTRACTL arge-scale internet services aim to remain highly availableand responsive in the presence of unexpected failures. Pro-viding this service often requires monitoring and analyzingtens of millions of measurements per second across a largenumber of systems, and one particularly effective solutionis to store and query such measurements in a time seriesdatabase (TSDB).A key challenge in the design of TSDBs is how to strikethe right balance between efficiency, scalability, and relia-bility.

2 In this paper we introduce gorilla , Facebook s In-Memory TSDB. Our insight is that users of monitoring sys-tems do not place much emphasis on individual data pointsbut rather on aggregate analysis, and recent data points areof much higher value than older points to quickly detect anddiagnose the root cause of an ongoing problem. gorilla op-timizes for remaining highly available for writes and reads,even in the face of failures, at the expense of possibly drop-ping small amounts of data on the write path. To improvequery efficiency, we aggressively leverage compression tech-niques such as delta-of-delta timestamps and XOR d floatingpoint values to reduce gorilla s storage footprint by allows us to store gorilla s data in memory, reduc-ing query latency by 73x and improving query throughputby 14x when compared to a traditional Database (HBase)-backed time Series data.

3 This performance improvement hasunlocked new monitoring and debugging tools, such as timeseries correlation search and more dense visualization also gracefully handles failures from a single-node toentire regions with little to no operational INTRODUCTIONL arge-scale internet services aim to remain highly-availableand responsive for their users even in the presence of unex-pected failures. As these services have grown to supporta global audience, they have scaled beyond a few systemsrunning on hundreds of machines to thousands of individ-This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs Unported License. To view a copy of this li-cense, visit Obtain per-mission prior to any use beyond those covered by the license.

4 Contactcopyright holder by emailing Articles from this volumewere invited to present their results at the 41st International Conference onVery Large Data Bases, August 31st - September 4th 2015, Kohala Coast, of the VLDB Endowment,Vol. 8, No. 12 Copyright 2015 VLDB Endowment 2150-8097/15 Services Web Tier FB Servers Long term storage (HBase)GorillaAd-hoc visualizations and dashboardsAlarms and automatic remediationTime Series CorrelationFigure 1: High level overview of the ODS monitor-ing and alerting system, showing gorilla as a write-through cache of the most recent 26 hours of timeseries systems running on many thousands of machines, oftenacross multiple geo-replicated important requirement to operating these large scaleservices is to accurately monitor the health and performanceof the underlying system and quickly identify and diagnoseproblems as they arise.

5 Facebook uses a time Series Database (TSDB) to store system measuring data points and providesquick query functionalities on top. We next specify some ofthe constraints that we need to satisy for monitoring andoperating Facebook and then describe gorilla , our new In-Memory TSDB that can store tens of millions of datapoints( , CPU load, error rate, latency etc.) every second andrespond queries over this data within primary requirement for a TSDBis that it should always be available to take writes. Aswe have hundreds of systems exposing multiple data items,the write rate might easily exceed tens of millions of datapoints each second. In constrast, the read rate is usuallya couple orders of magnitude lower as it is primarily fromautomated systems watching important time Series , data1816visualization systems presenting dashboards for human con-sumption, or from human operators wishing to diagnose anobserved wish to identify issues that emergefrom a new software release, an unexpected side effect of aconfiguration change, a network cut and other issues that re-sult in a significant state transition.

6 Thus, we wish for ourTSDB to support fine-grained aggregations over short-timewindows. The ability to display state transitions within tensof seconds is particularly prized as it allows automation toquickly remediate problems before they become wide if a network partition or otherfailure leads to disconnection between different datacenters,systems operating within any given datacenter ought to beable to write data to local TSDB machines and be able toretrieve this data on wish to replicate all writes to multi-ple regions so we can survive the loss of any given datacenteror geographic region due to a is Facebook s new TSDB that satisfies these con-straints. gorilla functions as a write-through cache of themost recent data entering the monitoring system.

7 We aimto ensure that most queries run within 10 s of insight in gorilla s design is that users of monitor-ing systems do not place much emphasis on individual datapoints but rather on aggregate analysis. Additionally, thesesystems do not store any user data so traditional ACID guar-antees are not a core requirement for TSDBs. However, ahigh percentage of writes must succeed at all times, evenin the face of disasters that might render entire datacentersunreachable. Additionally, recent data points are of highervalue than older points given the intuition that knowing ifa particular system or service is brokenright nowis morevaluable to an operations engineer than knowing if it wasbroken an hour ago.

8 gorilla optimizes for remaining highlyavailable for writes and reads, even in the face of failures, atthe expense of possibly dropping small amounts of data onthe write challenge then arises from high data insertion rate,total data quantity, real-time aggregation, and reliability re-quirements. We addressed each of these in turn. To addressthe first couple requirements, we analyzed the OperationalData Store (ODS) TSDB, an older monitoring system thatwas widely used at Facebook. We noticed that at least 85%of all queries to ODS was for data collected in the past 26hours. Further analysis allowed us to determine that wemight be able to serve our users best if we could replace adisk-based Database with an In-Memory Database .

9 Further,by treating this In-Memory Database as a cache of the persis-tent disk-based store, we could achieve the insertion speedof an In-Memory system with the persistence of a disk of Spring 2015, Facebook s monitoring systems gener-ate more than 2 billion unique time Series of counters, withabout 12 million data points added per second. This repre-sents over 1 trillion points per day. At 16 bytes per point,the resulting 16TB of RAM would be too resource intensivefor practical deployment. We addressed this by repurposingan existing XOR based floating point compression scheme towork in a streaming manner that allows us to compress timeseries to an average of bytes per point, a 12x reductionin addressed the reliability requirements by running mul-tiple instances of gorilla in different datacenter regions andstreaming data to each without attempting to guaranteeconsistency.

10 Read queries are directed at the closest avail-able gorilla instance. Note that this design leverages ourobservation that individual data points can be lost withoutcompromising data aggregation unless there s significant dis-crepancy between the gorilla is currently running in production at Facebookand is used daily by engineers for real-time firefighting anddebugging in conjunction with other monitoring and analy-sis systems like Hive [27] and Scuba [3] to detect and diag-nose BACKGROUND & Operational Data Store (ODS)Operating and managing Facebook s large infrastructurecomprised of hundreds of systems distributed across mul-tiple data centers would be very difficult without a moni-toring system that can track their health and Operational Data Store (ODS) is an important portionof the monitoring system at Facebook.