Incident Metrics in SRE

1 T p n Davidovi Incident Metrics in SREC ritically EvaluatingMTTR and FriendsBostonFarnhamSebastopolTokyoBeiji ngBostonFarnhamSebastopolTokyoBeijing978 -1-098-10313-2[LSI] Incident Metrics in SREby t p n Davidovi Copyright 2021 O Reilly Media, Inc. All rights in the United States of by O Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, Reilly books may be purchased for educational, business, or sales promotional editions are also available for most titles ( ). For more infor mation, contact our corporate/institutional sales department: 800-998-9938 Editor: John DevinsDevelopment Editor: Virginia WilsonProduction Editor: Kate GallowayCopyeditor: Shannon TurlingtonProofreader: Holly Bauer ForsythInterior Designer: David FutatoCover Designer: Kenn VondrakIllustrator: Kate DulleaMarch 2021: First EditionRevision History for the First Edition2021-03-19: First ReleaseThe O Reilly logo is a registered trademark of O Reilly Media, Inc.

2 Incident Metricsin SRE, the cover image, and related trade dress are trademarks of O Reilly Media, views expressed in this work are those of the author, and do not represent thepublisher s views. While the publisher and the author have used good faith efforts toensure that the information and instructions contained in this work are accurate, thepublisher and the author disclaim all responsibility for errors or omissions, includ ing without limitation responsibility for damages resulting from the use of or reli ance on this work.

3 Use of the information and instructions contained in this work isat your own risk. If any code samples or other technology this work contains ordescribes is subject to open source licenses or the intellectual property rights of oth ers, it is your responsibility to ensure that your use thereof complies with such licen ses and/or work is part of a collaboration between O Reilly and Google. See our statementof editorial of ContentsIncident Metrics in SRE.. 1 Abstract 1 Introduction 1 Incident Life Cycle and Timing 2 Analyzing Improvements 7 Analytical Approach 18 Large Company Incident Data Set

4 21Is It About Data Quality? 24 And That s Why MTTx Will Probably Mislead You 24 Better Analysis Options 26 Conclusion 28 Acknowledgments 29iiiIncident Metrics in SREA bstractMeasuring improvements as a result of a process change, productpurchase.

5 Or technological change is commonplace. In reliabilityengineering, statistics such as mean time to recovery (MTTR) ormean time to mitigation (MTTM) are often measured. These statis tics are sometimes used to evaluate improvements or track this report, I use a simple Monte Carlo simulation process (whichcan be applied in many other situations), as well as statistical analy sis, to demonstrate that these statistics are poorly suited for decisionmaking or trend analysis in the context of production incidents. Toreplace these, I propose better ways to achieve the same measure ments for some of the key responsibilities of a site reliability engineer (SRE) isto manage incidents of the production system(s) they are responsi ble for.

6 Within an Incident , SREs contribute to debugging the sys tem, choosing the right immediate mitigation, and organizing theincident response if it requires broader the responsibility of an SRE is not limited just to managing inci dents. Some of the work involves prevention, such as devisingrobust strategies for performing changes in production or automati cally responding to problems and reverting the system to a known-safe state. The work also includes mitigation, such as betterprocesses for communication, improvements in monitoring, ordevelopment of tooling that provides assistance during debugging of1the Incident .

7 As a matter of fact, there are products dedicated toimproving the process of Incident want your incidents (if you must have any at all!) to have as lit tle impact as possible. That often means short Incident durations,which I ll focus on here. Understanding how a process change or aproduct purchase shortens the durations of incidents is important,especially if there are real costs associated with the incidents. How ever, we can t jump to conclusions from a single Incident ; an analysisof a whole body of incidents is quick search with your favorite search engine might reveal manyarticles that state that MTTx Metrics (including mean time to recov ery and mean time to mitigation) should be considered the key per formance indicators of your service s reliability.

8 These articles aresometimes authored by high-profile companies with a track recordof delivering their services reliably or providing reliability-relatedtooling. But are these Metrics good indicators of reliability? In fact,are they indicators that can even be used at all? How can you tell?When applying MTTx Metrics , the goal is to understand the evolu tion of the reliability of your systems. But the reality is that applyingthese Metrics is trickier than it seems, and these popular Metrics aredangerously misleading in most practical report will show that MTTx is not useful in most typical SREsettings, for reasons that apply to many summary statistics and donot depend on company size or strictness of enforcement of produc tion practices.

9 Whatever metric you choose to use, it is important totest that it can give you robust insights regardless of the shape of theincident duration distribution. There may not be a silver bullet metric that could serve as a general-purpose replacement whereMTTx is currently considered, but you may have more success inmeasurement by tailoring the metric to the question at hand. I ll endthis report by exploring some alternative methods for achievingthese Life Cycle and TimingBefore analyzing incidents in aggregate, let me quickly introducesome language.

10 Language may vary from company to company, butthe underlying principles should be | Incident Metrics in SRE1 See, for example, A List of Post-mortems! and Postmortem Index .Figure 1 demonstrates a simple timeline model of an Incident thatI ll be using going forward. In this model, the Incident shows thesekey stages:First product impactThe first moment of severe impact to the productDetectionWhen the system s operator becomes aware of the ongoingproblemMitigationWhen there is no longer severe product impact but the systemmight still be degraded in some wayRecoveryWhen the system has been fully recovered into normal opera tion; recovery and mitigation are often the same stage, butsometimes they differFigure 1.