The Medical Segmentation Decathlon

1 The Medical Segmentation DecathlonMichela Antonellia,1, , Annika Reinkeb,c,d,1, Spyridon Bakase,u,v, Keyvan Farahanif, AnnetteKopp-Schneiderg, Bennett A. Landmanh, Geert Litjensi, Bjoern Menzej, Olaf Ronnebergerk, Ronald , Bram van Ginnekeni, Michel Bilelloe, Patrick Bilicp, Patrick F. Christp, Richard K. G. Doq,Marc J. Gollubq, Stephan H. Heckersr, Henkjan Huismani, William R. Jarnaginm, Maureen K. McHugor,Sandy Napels, Jennifer S. Golia Pernickaq, Kawal Rhodea, Catalina Tobon-Gomeza, Eugene Vorontsovt,Henkjan Huismani, James A. Meakini, Sebastien Ourselina, Manuel Wiesenfarthg, Pablo Arbel aezy,Byeonguk Baex, Sihong Chenan, Laura Dazay, Jianjiang Fengz, Baochun Heac, Fabian Isenseeaa, YuanfengJiab, Fucang Jiaac, Namkug Kimad, Ildoo Kimae, Dorit Merhofaf,aj, Akshay Paiah,ag, Beomhee Park1,Mathias Perslevah, Ramin Rezaiifarai, Oliver Rippelaf, Ignacio Sarasuaak, Wei Shenao, Jaemin Sonx,Christian Wachingerak, Liansheng Wangab, Yan Wangap, Yingda Xiaal, Daguang Xuam, Zhanwei Xuz,Yefeng Zhengan, Amber L.

2 Simpsonl, Lena Maier-Heinb,d,w,c,2, M. Jorge Cardosoa,2aSchool of Biomedical Engineering & Imaging Sciences, King s College London, London, UKbDiv. Computer Assisted Medical Interventions, German Cancer Research Center (DKFZ), Heidelberg, GermanycHIP Helmholtz Imaging Platform, German Cancer Research Center (DKFZ), Heidelberg, GermanydFaculty of Mathematics and Computer Science, University of Heidelberg, Heidelberg, GermanyeCenter for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USAfCenter for Biomedical Informatics and Information Technology, National Cancer Institute (NIH), Bethesda, MD, USAgDiv.

3 Biostatistics, German Cancer Research Center (DKFZ), Heidelberg, GermanyhElectrical Engineering and Computer Science, Vanderbilt University, Nashville TN, USAiRadboud University Medical Center, Radboud Institute for Health Sciences, Nijmegen, The NetherlandsjQuantitative Biomedicine, University of Zurich, Zurich, SwitzerlandkDeepMind, London, UKlSchool of Computing/Department of Biomedical and Molecular Sciences, Queen s University, Kingston, ON, CanadamDepartment of Surgery, Memorial Sloan Kettering Cancer Center, New York, NY, USAnImaging Biomarkers and Computer-Aided Diagnosis Laboratory, Department of Radiology and Imaging Sciences.

4 NationalInstitutes of Health Clinical Center (NIH), Bethesda, MD, USAoDiagnostic Image Analysis Group, Radboud University Medical Center, Nijmegen, The NetherlandspDepartment of Informatics, Technische Universit at M unchenqDepartment of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY, USArDepartment of Psychiatry & Behavioral Sciences, Vanderbilt University Medical Center, Nashville, TN, USAsDepartment of Radiology, Stanford University, Stanford, CA, USAtDepartment of Computer Science and Software Engineering, Ecole Polytechnique de Montr ealuDepartment of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA.

5 USAvDepartment of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia,PA, USAwMedical Faculty, University of Heidelberg, Heidelberg, GermanyxVUNO Inc., Seoul, KoreayUniversidad de los Andes, ColombiazDepartment of Automation, Tsinghua University, Beijing, ChinaaaHIP Applied Computer Vision Lab, Division of Medical Image Computing, German Cancer Research Center (DKFZ),Heidelberg, GermanyabDepartment of Computer Science, Xiamen University, Xiamen 361005, ChinaacShenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, ChinaadAffiliation of Participating teams Corresponding authorEmail Antonelli)

6 1 These authors contributed equally to this work2 These authors contributed equally to this workPreprint submitted to Nature Digital MedicineJune 11, 2021 [ ] 10 Jun 2021aeKakao Brain, Republic of KoreaafInstitute of Imaging & Computer Vision, RWTH Aachen University, Aachen, GermanyagCerebriu A/S, Copenhagen, DenmarkahDepartment of Computer Science, University of Copenhagen, Copenhagen, , San Diego, California, USAajFraunhofer Institute for Digital Medicine MEVIS, Bremen, GermanyakLab for Artificial Intelligence in Medical Imaging (AI-Med), Department of Child and Adolescent Psychiatry, UniversityHospital, LMU M unchen, GermanyalJohns Hopkins University, Baltimore, United StatesamNVIDIA, Santa Clara, CA, USAanTencent Jarvis Lab, Shenzhen, ChinaaoMoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai, ChinaapShanghai Key Laboratory of Multidimensional Information Processing, East China Normal University, Shanghai, ChinaAbstractInternational challenges have become thede factostandard for comparative assessment of image analysisalgorithms given a specific task.

7 Segmentation is so far the most widely investigated Medical image processingtask, but the various Segmentation challenges have typically been organized in isolation, such that algorithmdevelopment was driven by the need to tackle a single specific clinical problem. We hypothesized that amethod capable of performing well on multiple tasks will generalize well to a previously unseen task andpotentially outperform a custom-designed solution. To investigate the hypothesis, we organized the MedicalSegmentation Decathlon (MSD) a biomedical image analysis challenge, in which algorithms compete in amultitude of both tasks and modalities.

8 The underlying data set was designed to explore the axis of difficultiestypically encountered when dealing with Medical images, such as small data sets, unbalanced labels, multi-sitedata and small objects. The MSD challenge confirmed that algorithms with a consistent good performanceon a set of tasks preserved their good average performance on a different set of previously unseen , by monitoring the MSD winner for two years, we found that this algorithm continued generalizingwell to a wide range of other clinical problems, further confirming our hypothesis. Three main conclusionscan be drawn from this study: (1) state-of-the-art image Segmentation algorithms are mature, accurate, andgeneralize well when retrained on unseen tasks; (2) consistent algorithmic performance across multiple tasksis a strong surrogate of algorithmic generalizability; (3) the training of accurate AI Segmentation models isnow commoditized to non AI :Image Segmentation ; Medical image; Deep learning; Grand challenge1.

9 IntroductionMachine learning is beginning to revolutionize many fields of medicine, with success stories ranging fromthe accurate diagnosis and staging of diseases [1], to the early prediction of adverse events [2] and theautomatic discovery of antibiotics [3]. In this context, a large amount of literature has been dedicated to theautomatic analysis of Medical images [4]. Semantic Segmentation refers to the process of transforming rawmedical images into clinically relevant, spatially structured information, such as outlining tumor boundaries,and is an essential prerequisite for a number of clinical applications, such as radiotherapy planning [5] andtreatment response monitoring [6].

10 It is so far the most widely investigated Medical image processing task,with about 70% of all biomedical image analysis challenges dedicated to it [7]. With thousands of algorithms2published in the field of biomedical image Segmentation per year [8], however, it has become challengingto decide on a baseline architecture as starting point when designing an algorithm for a new given challenges have become the de-facto standard for comparative assessment of image analysisalgorithms given a specific task [7]. Yet, a deep learning architecture well-suitable for a certain clinicalproblem ( , Segmentation of brain tumors) may not necessarily generalize well to different, unseen tasks( , vessel Segmentation in the liver).