A survey of research questions for robust and bene cial AI

1 A survey of research questions for robust and beneficial AI1 IntroductionArtificial intelligence (AI) research has explored a variety of problems and approaches since its inception,but for the last 20 years or so has been focused on the problems surrounding the construction ofintelligentagents systems that perceive and act in some environment. In this context, the criterion for intelligence isrelated to statistical and economic notions of rationality colloquially, the ability to make good decisions,plans, or inferences.

2 The adoption of probabilistic representations and statistical learning methods has led toa large degree of integration and cross-fertilization between AI, machine learning, statistics, control theory,neuroscience, and other fields. The establishment of shared theoretical frameworks, combined with theavailability of data and processing power, has yielded remarkable successes in various component tasks suchas speech recognition, image classification, autonomous vehicles, machine translation, legged locomotion,and question-answering capabilities in these areas and others cross the threshold from laboratory research to economicallyvaluable technologies.

3 A virtuous cycle takes hold whereby even small improvements in performance are worthlarge sums of money, prompting greater investments in research . There is now a broad consensus that AIresearch is progressing steadily, and that its impact on society is likely to increase. The potential benefitsare huge, since everything that civilization has to offer is a product of human intelligence; we cannot predictwhat we might achieve when this intelligence is magnified by the tools AI may provide, but the eradication ofdisease and poverty are not unfathomable.

4 Because of the great potential of AI, it is valuable to investigatehow to reap its benefits while avoiding potential progress in AI research makes it timely to focus research not only on making AI more capable, butalso on maximizing the societal benefit of AI. Such considerations motivated the AAAI 2008 09 PresidentialPanel on Long-Term AI Futures [61] and other projects and community efforts on AI impacts. Theseconstitute a significant expansion of the field of AI itself, which up to now has focused largely on techniquesthat are neutral with respect to purpose.

5 The present document can be viewed as a natural continuationof these efforts, focusing on identifying research directions that can help maximize the societal benefit ofAI. This research is by necessity interdisciplinary, because it involves both society and AI. It ranges fromeconomics, law, and philosophy to computer security, formal methods and, of course, various branches of AIitself. The focus is on delivering AI that isbeneficialto society androbustin the sense that the benefits areguaranteed: our AI systems must do what we want them to document is an attempt to lay out some of the research topics that we think will be most useful todo now in order to shape the future impact of AI.

6 We will surely find that some questions are less usefulor timely than others, and some important ones are missing. We hope this guide will be a helpful sourceof suggestions, but also that potential grantees won t be discouraged from approaching us with similarlyrelevant topics we didn t think of. We will try to publish future versions that are up to date with progressin the are very grateful to the many people who have contributed to this document, in particular DanielDewey, Stuart Russell, and Max Tegmark for their invaluable work on the research priorities document, LukeMuehlhauser for his list of potential strategic research projects, Nate Soares and MIRI for their technicalagenda.

7 And the MIRIxOxford research workshop analyzing and expanding on the MIRI technical people at FLI have contributed lists of additional research projects and directions, including JimBabcock, Steve Greidinger, J anos Kram ar, Richard Mallah and Max Tegmark, and many more have provided1helpful feedback and suggestions, including Anthony Aguirre, Erik Brynjolfsson, Meia Chita-Tegmark, DanielDewey, Owain Evans, Eric Gastfriend, Ales Flidr, Katja Grace, Evan Hefner, Viktoriya Krakovna, ColleenMessing, Howard Messing, Chase Moores, Luke Muehlhauser, Se an O h Eigeartaigh, Tristan Plumb, StuartRussell, Murray Shanahan, Nate Soares, Marin Soljacic, Michele Reilly, Anders Sandberg, John Sturm, JaanTallinn, Jacob Trefethen, Nick Ryder, Michael Vassar, and Alexander Wissner-Gross, Eliezer also to Nick Bostrom for writing the excellent and seminal book Superintelligence.

8 Like the restof the document, this list is at an early stage - we re looking forward to receiving your constructive feedbackand comments!2 Contents1 Introduction12 Short-term research Optimizing AI s Economic Impact .. and Forecasting Economic Impact of Automation and AI .. research .. potential adverse effects of automation and AI .. Law and Ethics research .. Computer Science research for robust AI .. ( Did I build the system right? ).. ( Did I build the right system?)

9 ( How can I prevent unauthorized access? ).. ( OK, I built the system wrong, can I fix it? )..93 Long-term research Some perspectives on the long term .. Verification .. Validity .. goal stability .. Security .. containment .. containment .. containment .. : Detection & Response .. intent to deceive .. Control .. and Domesticity .. and Unsafe Agent Architectures ..214 Motivation .. Methodology .. Forecasting AI progress .. Forecasting AI takeoff.

10 Brain emulations (uploads) ..265 Policy and Collaboration2732 Short-term research Optimizing AI s Economic ImpactThe successes of industrial applications of AI, from manufacturing to information services, demonstrate agrowing impact on the economy, although there is disagreement about the exact nature of this impact and onhow to distinguish between the effects of AI and those of other information technologies. Many economistsand computer scientists agree that there is valuable research to be done on how to maximize the economicbenefits of AI while mitigating adverse effects, which could include increased inequality and unemployment[72, 21, 39, 40, 107, 84, 69].