Timezone: »
Owing to success in the data-rich domain of natural images, Transformers have recently become popular in medical image segmentation. However, the pairing of Transformers with convolutional blocks in varying architectural permutations leaves their relative effectiveness to open interpretation. We introduce Transformer Ablations that replace the Transformer blocks with plain linear operators to quantify this effectiveness. With experiments on 8 models on 2 medical image segmentation tasks, we explore - 1) the replaceable nature of Transformer-learnt representations, 2) Transformer capacity alone cannot prevent representational replaceability and works in tandem with effective design, 3) The mere existence of explicit feature hierarchies in transformer blocks is more beneficial than accompanying self-attention modules, 4) Major spatial downsampling before Transformer modules should be used with caution.
Author Information
Saikat Roy (German Cancer Research Center (DKFZ))
Gregor Köhler (German Cancer Research Center)
Michael Baumgartner (German Cancer Research Center (DKFZ))
Constantin Ulrich (German Cancer Research Center (DKFZ))
Jens Petersen (German Cancer Research Center (DKFZ))
Fabian Isensee (German Cancer Research Center (DKFZ))
Klaus H. Maier-Hein (German Cancer Research Center (DKFZ))
More from the Same Authors
-
2022 : Metrics Reloaded »
Annika Reinke · Lena Maier-Hein · Patrick Scholz · Minu D. Tizabi · Evangelia Christodoulou · Ben Glocker · Fabian Isensee · Jens Kleesiek · Michal Kozubek · Mauricio Reyes · Michael A. Riegler · Manuel Wiesenfarth · Michael Baumgartner · Matthias Eisenmann · Doreen Heckmann-Nötzel · A. Kavur · Tim Rädsch · Laura Acion · Michela Antonelli · Tal Arbel · Spyridon Bakas · Pete Bankhead · Arriel Benis · Florian Buettner · M. Jorge Cardoso · Veronika Cheplygina · Beth Cimini · Gary Collins · Keyvan Farahani · Luciana Ferrer · Adrian Galdran · Bram van Ginneken · Robert Haase · Daniel Hashimoto · Michael Hoffman · Merel Huisman · Pierre Jannin · Charles Kahn · Dagmar Kainmueller · Alexandros Karargyris · Bernhard Kainz · Alan Karthikesalingam · Hannes Kenngott · Florian Kofler · Annette Kopp-Schneider · Anna Kreshuk · Tahsin Kurc · Bennett Landman · Geert Litjens · Amin Madani · Klaus H. Maier-Hein · Anne Martel · Peter Mattson · Erik Meijering · Bjoern Menze · David Moher · Karel G.M. Moons · Henning Mueller · Brennan Nichyporuk · Felix Nickel · Jens Petersen · Nasir Rajpoot · Nicola Rieke · Julio Saez-Rodriguez · Clarisa Sanchez · Shravya Shetty · Maarten van Smeden · Carole Sudre · Ronald Summers · Abdel Aziz Taha · Sotirios Tsaftaris · Ben Ben Van Calster · Gaël Varoquaux · Paul Jäger