Practical Black-Box Attacks against Machine Learning

1 Practical Black-Box Attacks against Machine LearningNicolas PapernotPennsylvania State McDanielPennsylvania State Goodfellow JhaUniversity of Berkay CelikPennsylvania State SwamiUS Army Research Learning (ML) models, , deep neural networks(DNNs), are vulnerable to adversarial examples: maliciousinputs modified to yield erroneous model outputs, while ap-pearing unmodified to human observers. Potential attacksinclude having malicious content like malware identified aslegitimate or controlling vehicle behavior. Yet, all existingadversarial example Attacks require knowledge of either themodel internals or its training data.

2 We introduce the firstpractical demonstration of an attacker controlling a remotelyhosted DNN with no such knowledge. Indeed, the only capa-bility of our Black-Box adversary is to observe labels givenby the DNN to chosen inputs. Our attack strategy consistsin training a local model to substitute for the target DNN,using inputs synthetically generated by an adversary andlabeled by the target DNN. We use the local substitute tocraft adversarial examples, and find that they are misclas-sified by the targeted DNN. To perform a real-world andproperly-blinded evaluation, we attack a DNN hosted byMetaMind, an online deep Learning API.

3 We find that theirDNN misclassifies of the adversarial examples craftedwith our substitute. We demonstrate the general applicabil-ity of our strategy to many ML techniques by conducting thesame attack against models hosted by Amazon and Google,using logistic regression substitutes. They yield adversarialexamples misclassified by Amazon and Google at rates and We also find that this Black-Box attackstrategy is capable of evading defense strategies previouslyfound to make adversarial example crafting INTRODUCTIONA classifieris a ML model that learns a mapping betweeninputs and a set ofclasses.

4 For instance, a malware detectoris a classifier taking executables as inputs and assigning themto the benign or malware class. Efforts in the security [5, 2,9, 18] and Machine Learning [14, 4] communities exposed the Work done while the author was at rights licensed to ACM. ACM acknowledges that this contribution wasauthored or co-authored by an employee, contractor or affiliate of the United Statesgovernment. As such, the Government retains a nonexclusive, royalty-free right topublish or reproduce this article, or to allow others to do so, for Government CCS 17, April 02 - 06, 2017, Abu Dhabi, United Arab Emiratesc 2017 Copyright held by the owner/author(s).

5 Publication rights licensed to ISBN 978-1-4503-4944-4/17/04.. $ : of classifiers to integrity Attacks . Such attacksare often instantiated byadversarial examples: legitimateinputs altered by adding small, often imperceptible, perturba-tions to force a learned classifier to misclassify the resultingadversarial inputs, while remaining correctly classified by ahuman observer. To illustrate, consider the following images,potentially consumed by an autonomous vehicle [13]:To humans, these images appear to be the same: our bio-logical classifiers (vision) identify each image as a stop image on the left [13] is indeed an ordinary image of astop sign.

6 We produced the image on the right by addinga precise perturbation that forces a particular DNN to clas-sify it as a yield sign, as described in Section Here, anadversary could potentially use the altered image to causea car without failsafes to behave dangerously. This attackwould require modifying the image used internally by the carthrough transformations of the physical traffic sign. Relatedworks showed the feasibility of such physical transformationsfor a state-of-the-art vision classifier [6] and face recognitionmodel [11]. It is thus conceivable that physical adversarialtraffic signs could be generated by maliciously modifying thesign itself, , with stickers or this paper, we introduce the first demonstration thatblack-box attacksagainst DNN classifiers are Practical forreal-world adversaries withnoknowledge about the assume the adversary (a) has no information about thestructure or parameters of the DNN, and (b) does not haveaccess to any large training dataset.

7 The adversary s onlycapability is to observe labels assigned by the DNN for choseninputs, in a manner analog to a cryptographic novel attack strategy is to train a local substituteDNN with asyntheticdataset: the inputs are synthetic andgenerated by the adversary, while the outputs are labelsassigned by the target DNN and observed by the examples are crafted using the substitute param-eters, which are known to us. They are not only misclassifiedby the substitute but also by the target DNN, because bothmodels have similar decision is a considerable departure from previous work, whichevaluated perturbations required to craft adversarial exam-ples using either: (a) detailed knowledge of the DNN archi-tecture and parameters [2, 4, 9, 14], or (b) an independentlycollected training set to fit an auxiliary model [2, 4, 14].

8 [ ] 19 Mar 2017limited their applicability to strong adversaries capable ofgaining insider knowledge of the targeted ML model, or col-lecting large labeled training sets. We release assumption (a)by Learning a substitute: it gives us the benefit of having fullaccess to the model and apply previous adversarial examplecrafting methods. We release assumption (b) by replacingthe independently collected training set with a syntheticdataset constructed by the adversary with synthetic inputsand labeled by observing the target DNN s threat model thus corresponds to the real-world sce-nario of users interacting with classifiers hosted remotely bya third-party keeping the model internals secret.

9 In fact,we instantiate our attack against classifiers automaticallytrained by MetaMind, Amazon, and Google. We are ableto access them only after training is completed. Thus, weprovide the first correctly blinded experiments concerningadversarial examples as a security show that our Black-Box attack is applicable to manyremote systems taking decisions based on ML, because itcombines three key properties: (a) the capabilities requiredare limited to observing output class labels, (b) the numberof labels queried is limited, and (c) the approach appliesand scales to different ML classifier types (see Section 7),in addition to state-of-the-art DNNs.

10 In contrast, previouswork failed to simultaneously provide all of these three keyproperties [4, 14, 12, 15, 18]. Our contributions are: We introduce in Section 4 an attack against black-boxDNN classifiers. It crafts adversarial examples withoutknowledge of the classifier training data or model. To doso, a synthetic dataset is constructed by the adversaryto train a substitute for the targeted DNN classifier. In Section 5, we instantiate the attack against a re-mote DNN classifier hosted by MetaMind. The DNNmisclassifies of the adversarial inputs crafted. The attack is calibrated in Section 6 to (a) reduce thenumber of queries made to the target model and (b)maximize misclassification of adversarial examples.