A survey of research questions for robust and bene cial AI

1 A survey of research questions for robust and beneficial AI1 IntroductionArtificial intelligence (AI) research has explored a variety of problems and approaches since its inception,but for the last 20 years or so has been focused on the problems surrounding the construction ofintelligentagents systems that perceive and act in some environment. In this context, the criterion for intelligence isrelated to statistical and economic notions of rationality colloquially, the ability to make good decisions,plans, or inferences. The adoption of probabilistic representations and statistical learning methods has led toa large degree of integration and cross-fertilization between AI, machine learning, statistics, control theory,neuroscience, and other fields.

2 The establishment of shared theoretical frameworks, combined with theavailability of data and processing power, has yielded remarkable successes in various component tasks suchas speech recognition, image classification, autonomous vehicles, machine translation, legged locomotion,and question-answering capabilities in these areas and others cross the threshold from laboratory research to economicallyvaluable technologies, a virtuous cycle takes hold whereby even small improvements in performance are worthlarge sums of money, prompting greater investments in research . There is now a broad consensus that AIresearch is progressing steadily, and that its impact on society is likely to increase. The potential benefitsare huge, since everything that civilization has to offer is a product of human intelligence; we cannot predictwhat we might achieve when this intelligence is magnified by the tools AI may provide, but the eradication ofdisease and poverty are not unfathomable.

3 Because of the great potential of AI, it is valuable to investigatehow to reap its benefits while avoiding potential progress in AI research makes it timely to focus research not only on making AI more capable, butalso on maximizing the societal benefit of AI. Such considerations motivated the AAAI 2008 09 PresidentialPanel on Long-Term AI Futures [61] and other projects and community efforts on AI impacts. Theseconstitute a significant expansion of the field of AI itself, which up to now has focused largely on techniquesthat are neutral with respect to purpose. The present document can be viewed as a natural continuationof these efforts, focusing on identifying research directions that can help maximize the societal benefit ofAI.

4 This research is by necessity interdisciplinary, because it involves both society and AI. It ranges fromeconomics, law, and philosophy to computer security, formal methods and, of course, various branches of AIitself. The focus is on delivering AI that isbeneficialto society androbustin the sense that the benefits areguaranteed: our AI systems must do what we want them to document is an attempt to lay out some of the research topics that we think will be most useful todo now in order to shape the future impact of AI. We will surely find that some questions are less usefulor timely than others, and some important ones are missing. We hope this guide will be a helpful sourceof suggestions, but also that potential grantees won t be discouraged from approaching us with similarlyrelevant topics we didn t think of.

5 We will try to publish future versions that are up to date with progressin the are very grateful to the many people who have contributed to this document, in particular DanielDewey, Stuart Russell, and Max Tegmark for their invaluable work on the research priorities document, LukeMuehlhauser for his list of potential strategic research projects, Nate Soares and MIRI for their technicalagenda, and the MIRIxOxford research workshop analyzing and expanding on the MIRI technical people at FLI have contributed lists of additional research projects and directions, including JimBabcock, Steve Greidinger, J anos Kram ar, Richard Mallah and Max Tegmark, and many more have provided1helpful feedback and suggestions, including Anthony Aguirre, Erik Brynjolfsson, Meia Chita-Tegmark, DanielDewey, Owain Evans, Eric Gastfriend, Ales Flidr, Katja Grace, Evan Hefner, Viktoriya Krakovna, ColleenMessing, Howard Messing, Chase Moores, Luke Muehlhauser, Se an O h Eigeartaigh, Tristan Plumb, StuartRussell, Murray Shanahan, Nate Soares, Marin Soljacic, Michele Reilly, Anders Sandberg, John Sturm, JaanTallinn, Jacob Trefethen, Nick Ryder, Michael Vassar, and Alexander Wissner-Gross, Eliezer also to Nick Bostrom for writing the excellent and seminal book Superintelligence.

6 Like the restof the document, this list is at an early stage - we re looking forward to receiving your constructive feedbackand comments!2 Contents1 Introduction12 Short-term research Optimizing AI s Economic Impact .. and Forecasting Economic Impact of Automation and AI .. research .. potential adverse effects of automation and AI .. Law and Ethics research .. Computer Science research for robust AI .. ( Did I build the system right? ).. ( Did I build the right system? ).. ( How can I prevent unauthorized access? ).. ( OK, I built the system wrong, can I fix it? )..93 Long-term research Some perspectives on the long term .. Verification .. Validity .. goal stability .. Security .. containment.

7 Containment .. containment .. : Detection & Response .. intent to deceive .. Control .. and Domesticity .. and Unsafe Agent Architectures ..214 Motivation .. Methodology .. Forecasting AI progress .. Forecasting AI takeoff .. Brain emulations (uploads) ..265 Policy and Collaboration2732 Short-term research Optimizing AI s Economic ImpactThe successes of industrial applications of AI, from manufacturing to information services, demonstrate agrowing impact on the economy, although there is disagreement about the exact nature of this impact and onhow to distinguish between the effects of AI and those of other information technologies. Many economistsand computer scientists agree that there is valuable research to be done on how to maximize the economicbenefits of AI while mitigating adverse effects, which could include increased inequality and unemployment[72, 21, 39, 40, 107, 84, 69].

8 Such considerations motivate a range of research directions, spanning areas fromeconomics to psychology. Below are a few examples that should by no means be interpreted as an Measuring and Forecasting Economic Impact of Automation and AIWhen and in what order should we expect various jobs to become automated [39]? How will this affect theemployment and wages of various professions, including less skilled workers, creatives, and different kinds ofinformation workers? Some have argued that AI is likely to greatly increase the overall wealth of humanityas a whole [21]. However, increased automation may push income distribution further towards a power law[22], and the resulting disparity may fall disproportionately along lines of race, class, and gender; researchanticipating the economic and societal impact of such disparity could be It is possible that economic measures such as real GDP per capita do not accurately capture the benefitsand detriments of heavily AI-and-automation-based economies, making these metrics unsuitable forpolicy purposes [72].

9 research on improved metrics could be useful for What has been the historical record on jobs being displaced by automation? What have the averagerate and distribution of displacement been - has it been clustered in time, industry, and geography?How long before displaced workers found new jobs? Did displacement contribute to inequality?3. Is there anything different about the advancement of artificial intelligence happening now that wouldlead us to expect a change from our centuries-long historical record on jobs being displaced by automa-tion?4. What factors make an industry more amenable or less amenable to automation? Machines histori-cally performed rote mass-production, but we ve been expanding their capabilities with advances ininformation processing and artificial Which markets are most susceptible to disruption as automation advances?

10 Significant parts of theeconomy including finance, insurance, actuarial, and many consumer markets could experienceupheaval through use of AI techniques to learn, model, and predict agent actions. These markets mightbe identified by a combination of high complexity and high rewards for navigating that complexity [69].Are there other features?6. Some jobs could be done by machines in principle but require a particular advance in artificial intelli-gence before it becomes feasible. What are some of those prerequisite advances?7. Based on the above factors, how far in advance can we predict that an industry is likely to be largelyautomated? Is this something we can predict and prepare for 20 years ahead? 10 years? 6 months?Not at all?