How to Exploit Weaknesses in Biomedical Challenge Design and Organization

• Authors: REINKE Annika; EISENMANN Matthias; ONOGUR Sinan; STANKOVIC Marko; SCHOLZ Patrick; FULL Peter M; BOGUNOVIC Hrvoje; LANDMAN Bennett A; MAIER Oskar; MENZE Bjoern; SHARP Gregory C; SIRINUKUNWATTANA Korsuk; SPEIDEL Stefanie; VAN DER SOMMEN Fons; ZHENG Guoyan; MÜLLER Henning; KOZUBEK Michal; ARBEL Tal; BRADLEY Andrew P; JANNIN Pierre; KOPP-SCHNEIDER Annette; MAIER-HEIN Lena
• Year of publication: 2018
• Type: Article in Proceedings
• Conference: Medical Image Computing and Computer Assisted Intervention – MICCAI 2018
• MU Faculty or unit: Faculty of Informatics
• Web: https://doi.org/10.1007/978-3-030-00937-3_45
• Doi: http://dx.doi.org/10.1007/978-3-030-00937-3_45
• Keywords: biomedical image analysis; benchmarking
• Description: Since the first MICCAI grand challenge organized in 2007 in Brisbane, challenges have become an integral part of MICCAI conferences. In the meantime, challenge datasets have become widely recognized as international benchmarking datasets and thus have a great influence on the research community and individual careers. In this paper, we show several ways in which weaknesses related to current challenge design and organization can potentially be exploited. Our experimental analysis, based on MICCAI segmentation challenges organized in 2015, demonstrates that both challenge organizers and participants can potentially undertake measures to substantially tune rankings. To overcome these problems we present best practice recommendations for improving challenge design and organization.