Understanding algorithmic decision-making

1 STUDY Panel for the Future of Science and Technology EPRS | European Parliamentary Research Service Scientific Foresight Unit (STOA) PE March 2019 EN Understanding algorithmic decision - making : Opportunities and challenges Understanding algorithmic decision - making : Opportunities and challenges While algorithms are hardly a recent invention, they are nevertheless increasingly involved in systems used to support decision - making . These systems, known as 'ADS' ( algorithmic decision systems), often rely on the analysis of large amounts of personal data to infer correlations or, more generally, to derive information deemed useful to make decisions.

2 Human intervention in the decision - making may vary, and may even be completely out of the loop in entirely automated systems. In many situations, the impact of the decision on people can be significant, such as access to credit, employment, medical treatment, or judicial sentences, among other things. Entrusting ADS to make or to influence such decisions raises a variety of ethical, political, legal, or technical issues, where great care must be taken to analyse and address them correctly. If they are neglected, the expected benefits of these systems may be negated by a variety of different risks for individuals (discrimination, unfair practices, loss of autonomy, etc.)

3 , the economy (unfair practices, limited access to markets, etc.), and society as a whole (manipulation, threat to democracy, etc.). This study reviews the opportunities and risks related to the use of ADS. It presents policy options to reduce the risks and explain their limitations. We sketch some options to overcome these limitations to be able to benefit from the tremendous possibilities of ADS while limiting the risks related to their use. Beyond providing an up-to-date and systematic review of the situation, the study gives a precise definition of a number of key terms and an analysis of their differences to help clarify the debate.

4 The main focus of the study is the technical aspects of ADS. However, to broaden the discussion, other legal, ethical and social dimensions are considered. STOA | Panel for the Future of Science and Technology AUTHORS This study has been written by Claude Castelluccia and Daniel Le M tayer (Institut national de recherche en informatique et en automatique - Inria) at the request of the Panel for the Future of Science and Technology (STOA) and managed by the Scientific Foresight Unit within the Directorate-General for Parliamentary Research Services (DG EPRS) of the Secretariat of the European Parliament.

5 ADMINISTRATOR RESPONSIBLE Mihalis Kritikos, Scientific Foresight Unit (STOA) To contact the publisher, please e-mail Acknowledgments The authors would like to thank all who have helped, in any way whatever, in this study, in particular Irene Maxwell and Cl ment H nin for their careful reading and useful comments on an earlier draft of this report. LINGUISTIC VERSION Original: EN Manuscript completed in March 2019. DISCLAIMER AND COPYRIGHT This document is prepared for, and addressed to, the Members and staff of the European Parliament as background material to assist them in their parliamentary work.

6 The content of the document is the sole responsibility of its author(s) and any opinions expressed herein should not be taken to represent an official position of the Parliament. Reproduction and translation for non-commercial purposes are authorised, provided the source is acknowledged and the European Parliament is given prior notice and sent a copy. Brussels European Union, 2019. PE ISBN: 978-92-846-3506-1 doi: QA-06-18-337-EN-N (STOA website) (intranet) (internet) (blog) Understanding algorithmic decision - making : Opportunities and challenges I Executive Summary Scope of the study: While algorithms are hardly a recent invention, they are nevertheless increasingly involved in systems used to support decision - making .

7 Known as 'ADS' ( algorithmic decision systems), ADS often rely on the analysis of large amounts of personal data to infer correlations or, more generally, to derive information deemed useful to make decisions. Human intervention in the decision - making may vary, and may even be completely out of the loop in entirely automated systems. In many situations, the impact of the decision on people can be significant, such as on access to credit, employment, medical treatment, judicial sentences, among other things.

8 Entrusting ADS to make or to influence such decisions raises a variety of different ethical, political, legal, or technical issues, where great care must be taken to analyse and address them correctly. If they are neglected, the expected benefits of these systems may be negated by the variety of risks for individuals (discrimination, unfair practices, loss of autonomy, etc.), the economy (unfair practices, limited access to markets, etc.), and society as a whole (manipulation, threat to democracy, etc.). This study reviews the opportunities and risks related to the use of ADS.

9 It presents existing options to reduce these risks and explain their limitations. We sketch some options to benefit from the tremendous possibilities of ADS while limiting the risks related to their use. Beyond providing an up-to-date and systematic review of the situation, the study gives a precise definition of a number of key terms and an analysis of their differences to help clarify the debate. The main focus of the study is the technical aspects of ADS. However, to broaden the discussion, other legal, ethical and social dimensions are considered.

10 ADS opportunities and risks: The study discusses the benefits and risks related to the use of ADS for three categories of stakeholders: individuals, the private sector and the public sector. Risks may be intentional ( to optimise the interests of the operator of the ADS), accidental (side-effects of the purpose of the ADS, with no such intent from the designer), or the consequences of ADS errors or inaccuracies ( people wrongly included in blacklists or 'no fly' lists due to homonyms or inaccurate inferences). Opportunities and risks of ADS for individuals: ADS may undermine the fundamental principles of equality, privacy, dignity, autonomy and free will, and may also pose risks related to health, quality of life and physical integrity.