Cyber Resiliency Metrics, Measures of Effectiveness, and ...

1 Cyber Resiliency metrics , Measures of effectiveness , and Scoring Enabling Systems Engineers and Program Managers to Select the Most Useful Assessment Methods Deborah J. Bodeau Richard D. Graubart Rosalie M. McQuaid John Woodill September 2018 M T R 1 8 0 3 1 4 M I T R E T E C H N I C A L R E P O R T Dept. No.: T8A2 Project No.: 5118MC18-KA The views, opinions and/or findings contained in this report are those of The MITRE Corporation and should not be construed as an official government position, policy, or decision, unless designated by other documentation. Approved for Public Release; Distribution Unlimited.

2 Public Release Case Number 18-2579 NOTICE This technical data was produced for the U. S. Government under Contract No. FA8702-18-C-0001, and is subject to the Rights in Technical Data-Noncommercial Items Clause DFARS (JUN 2013) 2018 The MITRE Corporation. All rights reserved. Bedford, MA Approved for Public Release; Distribution Unlimited. Public Release Case Number 18-2579 iii Abstract This report is intended to serve as a general reference for systems engineers, program management staff, and others concerned with assessing or scoring Cyber Resiliency for systems and missions; selecting Cyber Resiliency metrics to support Cyber Resiliency assessment; and defining, evaluating, and using Cyber Resiliency Measures of effectiveness (MOEs) for alternative Cyber Resiliency solutions.

3 Background material is provided on how Cyber Resiliency scores, metrics , and MOEs can be characterized and derived; based on that material, a wide range of potential Cyber Resiliency metrics are identified. Topics to address when specifying a Cyber Resiliency metric are identified so that evaluation can be repeatable and reproducible, and so that the metric can be properly interpreted. A tailorable, extensible Cyber Resiliency scoring methodology is defined. A notional example is provided of how scoring, metrics , and MOEs can be used by systems engineers and program management to identify potential areas of Cyber Resiliency improvement and to evaluate the potential benefits of alternative solutions.

4 Iv This page intentionally left blank. v Executive Summary Introduction. This report is intended to serve as a general reference for systems engineers, program management staff, and others concerned with Cyber Resiliency metrics for systems and missions. Such stakeholders may be interested in Assessing or scoring Cyber Resiliency to compare a current or planned system with an ideal; Selecting Cyber Resiliency metrics which can be evaluated in a lab, test, or operational setting to support Cyber Resiliency assessment; and/or Defining, evaluating, and using Measures of effectiveness (MOEs) for alternative Cyber Resiliency solutions.

5 Cyber Resiliency metrics can inform investment and design decisions. They are closely related to, but not identical with, metrics for system resilience and security, and share challenges related to definition and evaluation with such metrics . A Cyber Resiliency metric is derived from or relatable to some element of the Cyber Resiliency Engineering Framework (CREF)1 a Cyber Resiliency goal, objective, design principle, technique, or implementation approach to a technique. As illustrated in Figure ES-1, the selection and prioritization of elements of the CREF for a given system or program is driven by the risk management strategy of the program or the system s owning organization.

6 Figure ES-1. Cyber Resiliency Engineering Framework: Mapping the Cyber Resiliency Domain2 By contrast, MOEs for alternative Cyber Resiliency solutions , combinations of architectural decisions, technologies, and operational processes intended to improve how well Cyber Resiliency goals and objectives are achieved by applying Cyber Resiliency design principles and techniques may not be Cyber Resiliency metrics per se. Cyber Resiliency MOEs can take the form of changes in mission MOEs or Measures of performance (MOPs), metrics related to adversary activities, or other risk factors. A scoring methodology for Cyber Resiliency can be used to assess how well a given system can meet its operational or mission objectives, and to compare alternative solutions.

7 Any scoring methodology is inherently situated in a programmatic, operational, and threat context; for Cyber Resiliency scoring, the 1 The CREF provides a structure for understanding different aspects of Cyber Resiliency and how those aspects interrelate. 2 Adapted from Figure 1 of the Initial Public Draft (IPD) of NIST SP 800-160 Vol. 2 [1]. vi threat model is particularly important. The situated Scoring Methodology for Cyber Resiliency (SSM-CR) provides a way to capture stakeholder priorities, restating what Cyber Resiliency objectives and more detailed CREF elements (sub-objectives and activities) mean for a given system or program, and to capture subject matter expert (SME) assessments of how well the relevant activities are or can be performed.

8 Supporting evidence for qualitative assessments can be developed by identifying and evaluating relevant Cyber Resiliency metrics and MOEs for alternative solutions; in addition, a set of Cyber Resiliency metrics can be selected and tailored for inclusion in a larger metrics program. Such metrics can be defined using a template to ensure repeatability and reproducibility. A catalog of representative Cyber Resiliency metrics has been developed and is described in a companion report. The remainder of this Executive Summary expands upon these points. The report itself provides considerable detail, and is designed to be a general reference on Cyber Resiliency metrics .

9 Why consider Cyber Resiliency metrics ? Cyber Resiliency the ability to anticipate, withstand, recover from, and adapt to adverse conditions, stresses, attacks, or compromises on Cyber resources is increasingly a concern for mission owners, program managers, and systems engineers. When these stakeholders consider a system (or a system-of-systems, as identified with a mission or a mission thread, or a family of systems, as identified with an acquisition program) from the standpoint of Cyber Resiliency , they tend to pose several questions: Which aspects of Cyber Resiliency matter to us? As illustrated in Figure ES-2, aspects of Cyber Resiliency which can be prioritized and assessed include properties, capabilities, and behaviors.

10 How well does the system provide these aspects? That is, o How completely or with how much confidence are properties and capabilities provided? o How quickly, completely, and confidently can behaviors occur? What risks to the missions the system supports, to the program, or to the information the system handles and to stakeholders in the security of that information are addressed by the way the system provides Cyber Resiliency ? What risks remain? Figure ES-2. Assessable or Measurable Aspects of Cyber Resiliency for a System If the system is not sufficiently Cyber resilient to address stakeholder concerns, a set of alternative Cyber Resiliency solutions can be defined, by applying Cyber Resiliency design principles to make architectural vii decisions, and by using Cyber Resiliency techniques, approaches to implementing those techniques, and specific technologies, products, and processes or procedures.