DeepXplore: Automated Whitebox Testing of Deep Learning ...

1 DeepXplore: Automated Whitebox Testingof Deep Learning SystemsKexin Pei , Yinzhi Cao , Junfeng Yang , Suman Jana Columbia University, Lehigh UniversityABSTRACTDeep Learning (DL) systems are increasingly deployed insafety- and security-critical domains including self-drivingcars and malware detection, where the correctness and pre-dictability of a system s behavior for corner case inputs areof great importance. Existing DL Testing depends heavilyon manually labeled data and therefore often fails to exposeerroneous behaviors for rare design, implement, and evaluate DeepXplore, the firstwhitebox framework for systematically Testing real-world DLsystems. First, we introduce neuron coverage for systemati-cally measuring the parts of a DL system exercised by testinputs.

2 Next, we leverage multiple DL systems with similarfunctionality as cross -referencing oracles to avoid manualchecking. Finally, we demonstrate how finding inputs forDL systems that both trigger many differential behaviors andachieve high neuron coverage can be represented as a jointoptimization problem and solved efficiently using gradient-based search efficiently finds thousands of incorrect cor-ner case behaviors ( , self-driving cars crashing into guardrails and malware masquerading as benign software) in state-of-the-art DL models with thousands of neurons trained onfive popular datasets including ImageNet and Udacity self-driving challenge data. For all tested DL models, on average,DeepXplore generated one test input demonstrating incorrectbehavior within one second while running only on a commod-ity laptop.

3 We further show that the test inputs generated byDeepXplore can also be used to retrain the corresponding DLmodel to improve the model s accuracy by up to3%.Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bearthis notice and the full citation on the first page. Copyrights for componentsof this work owned by others than ACM must be honored. Abstracting withcredit is permitted. To copy otherwise, or republish, to post on servers or toredistribute to lists, requires prior specific permission and/or a fee. Requestpermissions from 17, October 28, 2017, Shanghai, China 2017 Association for Computing ISBN 978-1-4503-5085-3/17/10.

4 $ CONCEPTS Computing methodologies Neural networks; Com-puter systems organization Neural networks;Reliabil-ity; Software and its engineering Software Testing anddebugging;KEYWORDSDeep Learning Testing , differential Testing , Whitebox testingACM reference Format:Kexin Pei, Yinzhi Cao, Junfeng Yang, Suman Jana. 2017. Deep-Xplore: Automated Whitebox Testing of Deep Learning of ACM Symposium on Operating Systems Prin-ciples (SOSP 17).ACM, New York, NY, USA, 18 pages. INTRODUCTIONOver the past few years, Deep Learning (DL) has madetremendous progress, achieving or surpassing human-levelperformance for a diverse set of tasks including image classi-fication [31,66], speech recognition [83], and playing gamessuch as Go [64].

5 These advances have led to widespread adop-tion and deployment of DL in security- and safety-criticalsystems such as self-driving cars [10], malware detection [88],and aircraft collision avoidance systems [35].This wide adoption of DL techniques presents new chal-lenges as the predictability and correctness of such systemsare of crucial importance. Unfortunately, DL systems, despitetheir impressive capabilities, often demonstrate unexpected orincorrect behaviors in corner cases for several reasons such asbiased training data, overfitting, and underfitting of the mod-els. In safety- and security-critical settings, such incorrectbehaviors can lead to disastrous consequences such as a fatalcollision of a self-driving car. For example, a Google self-driving car recently crashed into a bus because it expectedthe bus to yield under a set of rare conditions but the busdid not [27].

6 A Tesla car in autopilot crashed into a trailerbecause the autopilot system failed to recognize the trailer asan obstacle due to its white color against a brightly lit sky and the high ride height [73]. Such corner cases were notpart of Google s or Tesla s test set and thus never showed upduring 17, October 28, 2017, Shanghai, ChinaK. Pei, Y. Cao, J. Yang, S. JanaTherefore, safety- and security-critical DL systems, justlike traditional software, must be tested systematically for diff-erent corner cases to detect and fix ideally any potential flawsor undesired behaviors. This presents a new systems problemas Automated and systematic Testing of large-scale, real-worldDL systems with thousands of neurons and millions of param-eters for all corner cases is extremely standard approach for Testing DL systems is to gatherand manually label as much real-world test data as possi-ble [1,3].

7 Some DL systems such as Google self-driving carsalso use simulation to generate synthetic training data [4].However, such simulation is completely unguided as it doesnot consider the internals of the target DL system. Therefore,for the large input spaces of real-world DL systems ( , allpossible road conditions for a self-driving car), none of theseapproaches can hope to cover more than a tiny fraction (if anyat all) of all possible corner works on adversarial deep Learning [26,49,72]have demonstrated that carefully crafted synthetic images byadding minimal perturbations to an existing image can foolstate-of-the-art DL systems. The key idea is to create syntheticimages such that they get classified by DL models differentlythan the original picture but still look the same to the humaneye.

8 While such adversarial images expose some erroneousbehaviors of a DL model, the main restriction of such anapproach is that it must limit its perturbations to tiny invisiblechanges or require manual checks. Moreover, just like otherforms of existing DL Testing , the adversarial images onlycover a small part ( ) of DL system s logic as shown in 6. In essence, the current machine Learning Testing practicesfor finding incorrect corner cases are analogous to findingbugs in traditional software by using test inputs with low codecoverage and thus are unlikely to find many erroneous key challenges in Automated systematic Testing of large-scale DL systems are twofold: (1) how to generate inputs thattrigger different parts of a DL system s logic and uncoverdifferent types of erroneous behaviors, and (2) how to identifyerroneous behaviors of a DL system without manual label-ing/checking.

9 This paper describes how we design and buildDeepXplore to address both , we introduce the concept of neuron coverage formeasuring the parts of a DL system s logic exercised by a setof test inputs based on the number of neurons activated ( ,the output values are higher than a threshold) by the a high level, neuron coverage of DL systems is similar tocode coverage of traditional systems, a standard empiricalmetric for measuring the amount of code exercised by aninput in a traditional software. However, code coverage itselfis not a good metric for estimating coverage of DL systemsas most rules in DL systems, unlike traditional software, arenot written manually by a programmer but rather are learnedfrom training data. In fact, we find that for most of the DL(a) Input 1(b) Input 2 (darker version of 1)Figure 1: An example erroneous behavior found by DeepXplorein Nvidia DAVE-2 self-driving car platform.

10 The DNN-basedself-driving car correctly decides to turn left for image (a) butincorrectly decides to turn right and crashes into the guardrailfor image (b), a slightly darker version of (a).systems that we tested, even a single randomly picked testinput was able to achieve 100% code coverage while theneuron coverage was less than 10%.Next, we show how multiple DL systems with similar func-tionality ( , self-driving cars by Google, Tesla, and GM)can be used as cross -referencing oracles to identify erroneouscorner cases without manual checks. For example, if one self-driving car decides to turn left while others turn right for thesame input, one of them is likely to be incorrect. Such differ-ential Testing techniques have been applied successfully in thepast for detecting logic bugs without manual specifications ina wide variety of traditional software [6,11,14,15,45,86].