THE STATE OF DEEPFAKES

1 THES TATEOFDEEPFAKESLANDSCAPE, THREATS, AND IMPACTS eptember 2019 2019 DeeptraceContact: Henry Ajder, Giorgio Patrini, Francesco Cavalli & Laurence CullenGraphic design: Eleanor WinterCover image: Joel FilipeCite: The STATE of DEEPFAKES : Landscape, Threats, and Impact, Henry Ajder, Giorgio Patrini, Francesco Cavalli, and Laurence Cullen, September DeeptraceDeeptrace is an Amsterdam-based cybersecurity company providing deep learning and computer vision technologies for the detection and online monitoring of synthetic media. Our mission is to protect individuals and organizations from the damaging impacts of AI-generated synthetic media.

2 Deeptrace also publishes Tracer, a curated weekly newsletter covering key developments with DEEPFAKES , synthetic media, and emerging cyber would like to thank the following people for providing valuable feedback and quotes for this report: Curtis Barnes, Tom Barraclough, Danielle Citron, Jack Clark, Adam Dodge, Sam Gregory, Gabi Ivens, Hamish Ivey-Law, Jason Mancuso, Samantha North, Rodolfo Rosini, Paul Scharre, Gaurav Tanwar, Leroy Terrelonge, and Eleanor of ContentsThe STATE of DEEPFAKES : An Overview 1 The Commodification of DEEPFAKES 3 Deepfake Pornography 6 Politics & Cybersecurity 9 Concluding Remarks 15 Appendix 16 DEEPTRACE rise of synthetic media and DEEPFAKES is forcing us towards an important and unsettling realization.

3 Our historical belief that video and audio are reliable records of reality is no longer choose the word historical as this belief is a coincidence of how technology has evolved. Still today, we trust a phone call f rom a f riend or a video clip featuring a known politician, simply based on the recognition of their voices and faces. Previously, no commonly available technology could have synthetically created this media with comparable realism, so we treated it as authentic by definition.

4 With the development of synthetic media and DEEPFAKES , this is no longer the case. Every digital communication channel our society is built upon, whether that be audio, video, or even text, is at risk of being its foundation in 2018, Deeptrace has been dedicated to researching DEEPFAKES evolving capabilities and threats, providing crucial intelligence for enhancing our detection technology. In this report we share the insights f rom our most comprehensive mapping of the deepfake landscape to date, revealing DEEPFAKES real-world impact.

5 In doing so, we provide an expert overview of the current STATE of DEEPFAKES , cutting through some of the hyperbole surrounding the topic. The findings we present here are grounded in independently sourced data, accompanied by insights f rom leading experts in this research revealed that the deepfake phenomenon is growing rapidly online, with the number of deepfake videos almost doubling over the last seven months to 14,678. This increase is supported by the growing commodification of tools and services that lower the barrier for non-experts to create DEEPFAKES .

6 Perhaps unsurprisingly, we observed a significant contribution to the creation and use of synthetic media tools f rom web users in China and South Korea, despite the totality of our sources coming f rom the English-speaking key trend we identified is the prominence of non-consensual deepfake pornography, which accounted for 96% of the total deepfake videos online. We also found that the top four websites dedicated to deepfake pornography received more than 134 million views on videos targeting hundreds of female celebrities worldwide.

7 This significant viewership demonstrates a market for websites creating and hosting deepfake pornography, a trend that will continue to grow unless decisive action is are also making a significant impact on the political sphere. Two landmark cases f rom Gabon and Malaysia that received minimal Western media coverage saw DEEPFAKES linked to an alleged government cover-up and a political smear campaign. One of these cases was related to an attempted military coup, while the other continues to threaten a high-profile politician with imprisonment.

8 Seen together, these examples are possibly the most powerful indications of how DEEPFAKES are already destabilizing political processes. Without defensive countermeasures, the integrity of democracies around the world are at of politics, the weaponization of DEEPFAKES and synthetic media is influencing the cybersecurity landscape, enhancing traditional cyber threats and enabling entirely new attack vectors. Notably, 2019 saw reports of cases where synthetic voice audio and images of non-existent, synthetic people were used to enhance social engineering against businesses and STATE OF DEEPFAKES ForewordDeepfakes are here to stay, and their impact is already being felt on a global scale.

9 We hope this report stimulates further discussion on the topic, and emphasizes the importance of developing a range of countermeasures to protect individuals and organizations f rom the harmful applications of Patrini Founder, CEO, and Chief Scientist1 DEEPTRACE STATE of DEEPFAKES : An OverviewSince emerging in late 2017 the phenomenon of DEEPFAKES has developed rapidly, both in terms of technological sophistication and societal impact. In this section, we provide an overview of the current STATE of DEEPFAKES by analyzing their proliferation and The appendix details how we collected the data presented in this reportOur findings revealed that the total number of deepfake videos online is rapidly increasing, with this measurement representing an almost 100% increase based on our previous measurement (7,964)

10 Taken in December pornography accounts for a significant majority of deepfake videos online, even as other forms of non-pornographic DEEPFAKES have gained being a relatively new phenomenon (the earliest of these websites was registered in February 2018), deepfake pornography has already attracted a large viewership on the top four dedicated deepfake pornography websites number of deepfake videos online14,678percentage of deepfake videos online by pornographic and non-pornographic content96%4%134,364,438 Total number of video views across top four dedicated deepfake pornography websites2 THE STATE OF DEEPFAKES The STATE of DEEPFAKES .