The Medical Segmentation Decathlon

1 The Medical Segmentation DecathlonMichela Antonellia,1, , Annika Reinkeb,c,d,1, Spyridon Bakase,u,v, Keyvan Farahanif, AnnetteKopp-Schneiderg, Bennett A. Landmanh, Geert Litjensi, Bjoern Menzej, Olaf Ronnebergerk, Ronald , Bram van Ginnekeni, Michel Bilelloe, Patrick Bilicp, Patrick F. Christp, Richard K. G. Doq,Marc J. Gollubq, Stephan H. Heckersr, Henkjan Huismani, William R. Jarnaginm, Maureen K. McHugor,Sandy Napels, Jennifer S. Golia Pernickaq, Kawal Rhodea, Catalina Tobon-Gomeza, Eugene Vorontsovt,Henkjan Huismani, James A. Meakini, Sebastien Ourselina, Manuel Wiesenfarthg, Pablo Arbel aezy,Byeonguk Baex, Sihong Chenan, Laura Dazay, Jianjiang Fengz, Baochun Heac, Fabian Isenseeaa, YuanfengJiab, Fucang Jiaac, Namkug Kimad, Ildoo Kimae, Dorit Merhofaf,aj, Akshay Paiah,ag, Beomhee Park1,Mathias Perslevah, Ramin Rezaiifarai, Oliver Rippelaf, Ignacio Sarasuaak, Wei Shenao, Jaemin Sonx,Christian Wachingerak, Liansheng Wangab, Yan Wangap, Yingda Xiaal, Daguang Xuam, Zhanwei Xuz,Yefeng Zhengan, Amber L.

2 Simpsonl, Lena Maier-Heinb,d,w,c,2, M. Jorge Cardosoa,2aSchool of Biomedical Engineering & Imaging Sciences, King s College London, London, UKbDiv. Computer Assisted Medical Interventions, German Cancer Research center (DKFZ), Heidelberg, GermanycHIP Helmholtz Imaging Platform, German Cancer Research center (DKFZ), Heidelberg, GermanydFaculty of Mathematics and Computer Science, University of Heidelberg, Heidelberg, GermanyeCenter for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USAfCenter for Biomedical Informatics and Information Technology, National Cancer Institute (NIH), Bethesda, MD, USAgDiv. Biostatistics, German Cancer Research center (DKFZ), Heidelberg, GermanyhElectrical Engineering and Computer Science, vanderbilt University, Nashville TN, USAiRadboud University Medical center , Radboud Institute for Health Sciences, Nijmegen, The NetherlandsjQuantitative Biomedicine, University of Zurich, Zurich, SwitzerlandkDeepMind, London, UKlSchool of Computing/Department of Biomedical and Molecular Sciences, Queen s University, Kingston, ON, CanadamDepartment of Surgery, Memorial Sloan Kettering Cancer center , New York, NY, USAnImaging Biomarkers and Computer-Aided Diagnosis Laboratory, Department of Radiology and Imaging Sciences, NationalInstitutes of Health Clinical center (NIH)

3 , Bethesda, MD, USAoDiagnostic Image Analysis Group, Radboud University Medical center , Nijmegen, The NetherlandspDepartment of Informatics, Technische Universit at M unchenqDepartment of Radiology, Memorial Sloan Kettering Cancer center , New York, NY, USArDepartment of Psychiatry & Behavioral Sciences, vanderbilt University Medical center , Nashville, TN, USAsDepartment of Radiology, Stanford University, Stanford, CA, USAtDepartment of Computer Science and Software Engineering, Ecole Polytechnique de Montr ealuDepartment of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USAvDepartment of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia,PA, USAwMedical Faculty, University of Heidelberg, Heidelberg, GermanyxVUNO Inc., Seoul, KoreayUniversidad de los Andes, ColombiazDepartment of Automation, Tsinghua University, Beijing, ChinaaaHIP Applied Computer Vision Lab, Division of Medical Image Computing, German Cancer Research center (DKFZ),Heidelberg, GermanyabDepartment of Computer Science, Xiamen University, Xiamen 361005, ChinaacShenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, ChinaadAffiliation of Participating teams Corresponding authorEmail Antonelli)

4 1 These authors contributed equally to this work2 These authors contributed equally to this workPreprint submitted to Nature Digital MedicineJune 11, 2021 [ ] 10 Jun 2021aeKakao Brain, Republic of KoreaafInstitute of Imaging & Computer Vision, RWTH Aachen University, Aachen, GermanyagCerebriu A/S, Copenhagen, DenmarkahDepartment of Computer Science, University of Copenhagen, Copenhagen, , San Diego, California, USAajFraunhofer Institute for Digital Medicine MEVIS, Bremen, GermanyakLab for Artificial Intelligence in Medical Imaging (AI-Med), Department of Child and Adolescent Psychiatry, UniversityHospital, LMU M unchen, GermanyalJohns Hopkins University, Baltimore, United StatesamNVIDIA, Santa Clara, CA, USAanTencent Jarvis Lab, Shenzhen, ChinaaoMoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai, ChinaapShanghai Key Laboratory of Multidimensional Information Processing, East China Normal University, Shanghai, ChinaAbstractInternational challenges have become thede factostandard for comparative assessment of image analysisalgorithms given a specific task.

5 Segmentation is so far the most widely investigated Medical image processingtask, but the various Segmentation challenges have typically been organized in isolation, such that algorithmdevelopment was driven by the need to tackle a single specific clinical problem. We hypothesized that amethod capable of performing well on multiple tasks will generalize well to a previously unseen task andpotentially outperform a custom-designed solution. To investigate the hypothesis, we organized the MedicalSegmentation Decathlon (MSD) a biomedical image analysis challenge, in which algorithms compete in amultitude of both tasks and modalities. The underlying data set was designed to explore the axis of difficultiestypically encountered when dealing with Medical images, such as small data sets, unbalanced labels, multi-sitedata and small objects. The MSD challenge confirmed that algorithms with a consistent good performanceon a set of tasks preserved their good average performance on a different set of previously unseen , by monitoring the MSD winner for two years, we found that this algorithm continued generalizingwell to a wide range of other clinical problems, further confirming our hypothesis.

6 Three main conclusionscan be drawn from this study: (1) state-of-the-art image Segmentation algorithms are mature, accurate, andgeneralize well when retrained on unseen tasks; (2) consistent algorithmic performance across multiple tasksis a strong surrogate of algorithmic generalizability; (3) the training of accurate AI Segmentation models isnow commoditized to non AI :Image Segmentation ; Medical image; Deep learning; Grand challenge1. IntroductionMachine learning is beginning to revolutionize many fields of medicine, with success stories ranging fromthe accurate diagnosis and staging of diseases [1], to the early prediction of adverse events [2] and theautomatic discovery of antibiotics [3]. In this context, a large amount of literature has been dedicated to theautomatic analysis of Medical images [4]. Semantic Segmentation refers to the process of transforming rawmedical images into clinically relevant, spatially structured information, such as outlining tumor boundaries,and is an essential prerequisite for a number of clinical applications, such as radiotherapy planning [5] andtreatment response monitoring [6].

7 It is so far the most widely investigated Medical image processing task,with about 70% of all biomedical image analysis challenges dedicated to it [7]. With thousands of algorithms2published in the field of biomedical image Segmentation per year [8], however, it has become challengingto decide on a baseline architecture as starting point when designing an algorithm for a new given challenges have become the de-facto standard for comparative assessment of image analysisalgorithms given a specific task [7]. Yet, a deep learning architecture well-suitable for a certain clinicalproblem ( , Segmentation of brain tumors) may not necessarily generalize well to different, unseen tasks( , vessel Segmentation in the liver). Such a generalizable learner , which in this setting would representa fully automated method that can learn any Segmentation task given some training data and without theneed for human intervention, would provide the missing technical scalability to allow many new applicationsin computer-aided diagnosis, biomarker extraction, surgical intervention planning, disease prognosis, etc.

8 Toaddress this gap in the literature, we proposed the concept of theMedical Segmentation Decathlon (MSD), aninternational challenge dedicated to identifying a general-purpose algorithm for Medical image competition comprised ten different data sets with various challenging characteristics, as shown in Fig. participants were allowed to submit only one solution, able to solve all problems without changing thearchitecture or contribution of this paper is three-fold: (1) We are the first to organize a biomedical image analysischallenge in which algorithms compete in a multitude of both tasks and modalities. More specifically, theunderlying data set has been designed to feature some of the representative difficulties typically encounteredwhen dealing with Medical images, such as small data sets, unbalanced labels, multi-site data and smallobjects. (2) Based on the MSD, we released the first open framework for benchmarking Medical segmentationalgorithms with a specific focus on generalizability.

9 (3) By monitoring the winning algorithm, we show thatgeneralization across various clinical applications is possible with one single the following, we will first describe the challenge design, including the organization, mission, data setsand assessment method, in Section 2, followed by the presentation of the results in Section 3, in which wepresent the submitted methods and rankings as well as the results for the live challenge. We conclude witha discussion in Section MethodsThis section is organized according to the EQUATOR3guideline BIAS (Biomedical Image Analysis Chal-lengeS) [9], a recently published guideline specifically designed for the reporting of biomedical image analysischallenges. It comprises information on challenge organization and mission, as well as the data sets and as-sessment methods used to evaluate the submitted Challenge organizationThe Decathlon challenge was organized at the International Conference on Medical Image Computingand Computer Assisted Intervention (MICCAI) 2018, held in Granada, Spain.

10 After the main challengeevent at MICCAI, a live challenge was opened for submissions which is still open and regularly receives newsubmissions (more than 500 as of May 15th 2021).The MSD challenge aimed to test the ability of machine learning algorithms to accurately segment a largecollection of prescribed regions of interest, as defined by ten different data sets, each corresponding to adifferent anatomical structure (see Fig. 1) and to at least one Medical imagingtask[10]. The challenge itselfconsisted of two phases:3 1: Overview of the ten different tasks of the Medical Segmentation Decathlon (MSD). The challenge comprised differenttarget regions, modalities and challenging characteristics and was separated into seven known tasks (blue; the developmentphase) and three mystery tasks (gray; the mystery phase). Used abbreviations: MRI - magnetic resonance imaging, mp-MRI - multiparametric-magnetic resonance imaging, CT - computed the first phase, named thedevelopment phase, the training cases (comprising images and labels) forseven data sets were released, namely for brain, liver, heart, hippocampus, prostate, lung, and were expected to download the data, develop a general purpose learning algorithm, train thealgorithm on each task s training data independently and without human interaction (no task-specific manualparameter settings), run the learned model on each task s test data, and submit the Segmentation team was only allowed to make one submission per day to avoid model overfit, and the results werepresented in form of a live leaderboard on the challenge website,4visible to the public.