Robust Physical-World Attacks on Deep Learning Visual ...

1 This paper appears at CVPR 2018 Robust Physical-World Attacks on deep Learning Visual ClassificationKevin Eykholt 1, Ivan Evtimov*2, Earlence Fernandes2, Bo Li3,Amir Rahmati4, Chaowei Xiao1, Atul Prakash1, Tadayoshi Kohno2, and Dawn Song31 University of Michigan, Ann Arbor2 University of Washington3 University of California, Berkeley4 Samsung Research America and Stony Brook UniversityAbstractRecent studies show that the state-of-the-art deep neuralnetworks (DNNs) are vulnerable to adversarial examples,resulting from small-magnitude perturbations added to theinput. Given that that emerging physical systems are us-ing DNNs in safety-critical situations, adversarial examplescould mislead these systems and cause dangerous , understanding adversarial examples in the physi-cal world is an important step towards developing resilientlearning algorithms.

2 We propose a general attack algorithm, Robust physical Perturbations (RP2), to generate robustvisualadversarial perturbations under different physicalconditions. Using the real- world case of road sign classifi-cation, we show that adversarial examples generated usingRP2achieve high targeted misclassification rates againststandard-architecture road sign classifiers in the physicalworld under various environmental conditions, includingviewpoints. Due to the current lack of a standardized testingmethod, we propose a two-stage evaluation methodology forrobust physical adversarial examples consisting of lab andfield tests. Using this methodology, we evaluate the efficacyof physical adversarial manipulations on real objects.

3 Witha perturbation in the form of only black and white stickers,we attack a real stop sign, causing targeted misclassificationin 100% of the images obtained in lab settings, and in the captured video frames obtained on a moving vehicle(field test) for the target IntroductionDeep Neural Networks (DNNs) have achieved state-of-the-art, and sometimes human-competitive, performanceon many computer vision tasks [11, 14, 36]. Based on These authors contributed successes, they are increasingly being used as partof control pipelines in physical systems such as cars [8, 17],UAVs [4, 24], and robots [40]. Recent work, however, hasdemonstrated that DNNs are vulnerable to adversarial per-turbations [5, 9, 10, 15, 16, 22, 25, 29, 30, 35].

4 These carefullycrafted modifications to the ( Visual ) input of DNNs can causethe systems they control to misbehave in unexpected andpotentially dangerous threat has gained recent attention, and work incomputer vision has made great progress in understandingthe space of adversarial examples, beginning in the digi-tal domain ( by modifying images corresponding to ascene) [9, 22, 25, 35], and more recently in the physical do-main [1, 2, 13, 32]. Along similar lines, our work contributesto the understanding of adversarial examples when pertur-bations are physically added to theobjects themselves. Wechoose road sign classification as our target domain for sev-eral reasons: (1) The relative Visual simplicity of road signsmake it challenging to hide perturbations.

5 (2) Road signsexist in a noisy unconstrained environment with changingphysical conditions such as the distance and angle of theviewing camera, implying that physical adversarial perturba-tions should be Robust against considerable environmentalinstability. (3) Road signs play an important role in trans-portation safety. (4) A reasonable threat model for transporta-tion is that an attacker might not have control over a vehicle ssystems, but is able to modify the objects in the physicalworld that a vehicle might depend on to make crucial main challenge with generating Robust physical per-turbations is environmental variability. Cyber- physical sys-tems operate in noisy physical environments that can de-stroy perturbations created using current digital-only algo-rithms [19].

6 For our chosen application area, the most dy-namic environmental change is the distance and angle [ ] 10 Apr 2018 Figure 1: The left image shows real graffiti on a Stop sign,something that most humans would not think is right image shows our a physical perturbation appliedto a Stop sign. We design our perturbations to mimic graffiti,and thus hide in the human psyche. the viewing camera. Additionally, other practicality chal-lenges exist: (1) Perturbations in the digital world can beso small in magnitude that it is likely that a camera will notbe able to perceive them due to sensor imperfections. (2)Current algorithms produce perturbations that occupy thebackground imagery of an object.

7 It is extremely difficultto create a Robust attack with background modifications be-cause a real object can have varying backgrounds dependingon the viewpoint. (3) The fabrication process ( , printingof perturbations) is by the challenges above, we designRobustPhysical Perturbations (RP2), which can generate perturba-tions Robust to widely changing distances and angles of theviewing camera. RP2creates a visible, but inconspicuousperturbation that only perturbs the object ( a road sign)and not the object s environment. To create Robust perturba-tions, the algorithm draws samples from a distribution thatmodels physical dynamics ( varying distances and an-gles) using experimental data and synthetic transformations(Figure 2).

8 Using the proposed algorithm, we evaluate the effective-ness of perturbations on physical objects, and show thatadversaries can physically modify objects using low-costtechniques to reliably cause classification errors in DNN-based classifiers under widely varying distances and example, our Attacks cause a classifier to interpret asubtly-modified physical Stop sign as a Speed Limit 45 , our final form of perturbation is a set of blackand white stickers that an adversary can attach to a physicalroad sign (Stop sign). We designed our perturbations to re-semble graffiti, a relatively common form of vandalism. Itis common to see road signs with random graffiti or coloralterations in the real world as shown in Figure 1 (the leftimage is of a real sign in a city).

9 If these random patternswere adversarial perturbations (right side of Figure 1 showsour example perturbation), they could lead to severe conse-quences for autonomous driving systems, without arousingsuspicion in human the lack of a standardized method for evaluatingFigure 2: RP2pipeline overview. The input is the target Stopsign. RP2samples from a distribution that models physicaldynamics (in this case, varying distances and angles), anduses a mask to project computed perturbations to a shapethat resembles graffiti. The adversary prints out the resultingperturbations and sticks them to the target Stop Attacks , we draw on standard techniques from thephysical sciences and propose a two-stage experiment de-sign: (1) A lab test where the viewing camera is kept atvarious distance/angle configurations; and (2) A field testwhere we drive a car towards an intersection in uncontrolledconditions to simulate an autonomous vehicle.

10 We test ourattack algorithm using this evaluation pipeline and find thatthe perturbations are Robust to a variety of distances 2 shows an overview of ourpipeline to generate and evaluate Robust physical introduce Robust physical Perturbations (RP2) togenerate physical perturbations forphysical-worldob-jects that can consistently cause misclassification in aDNN-based classifier under a range of dynamic physi-cal conditions, including different viewpoint angles anddistances (Section 3). the lack of a standardized methodology in eval-uating physical adversarial perturbations, we proposean evaluation methodology to study the effectivenessof physical perturbations in real world scenarios (Sec-tion ).