The Qualifier Phase
The Qualifier Phase of the DIADEM competition took place remotely, that is, every participant downloaded the data locally, developed and tested algorithms, and submitted entries by the deadline. The data were chosen to be broadly representative of the open problem in modern neuroscience with regard to the scientific question, experimental preparation, and imaging technique.
The "rules of the game" were: algorithms had to read in image stacks of neuronal arbors and output digital reconstructions of the trees, which could then be scored against the corresponding manual gold standard. All submissions were scored and ranked, and the top performers were invited to compete in the tournament and participate in the scientific conference. Finalists were expected to optimize their algorithms and parameters — as well as prepare a research report — between the time they are selected and the opening of the Final Round of the DIADEM Challenge.
The Final Round
The Final Round of the DIADEM Challenge took place at the Janelia Farm Research Campus of the Howard Hughes Medical Institute in conjunction with an independent scientific conference organized with support from the National Institutes of Health. The conference included a small number of scientific sessions and ample time for personal and technical interactions between the finalists. Articles describing the algorithms, the data, and the contest results were collected and submitted for publication in a special issue of the journal Neuroinformatics.
Five teams were invited to participate in the Final Round of the DIADEM Challenge. Teams rotated the data set (and thus the data owner working with them) at every session. As a consequence, each of the five teams tried all data sets, and each data owner worked with all teams.
The scientific conference provided an opportunity for presentation of research reports and identification of new challenges in the development of effective tools for neuronal imaging. Research reports were subject to interdisciplinary peer review. Poster versions of all of these reports were also displayed at the conference.
The organizers produced an editorial describing the event and communicated the results to the scientific community through a variety of methods. All gold standard digital reconstructions are posted online at NeuroMorpho.Org, with the winning automated ones available on this site. The goal is to ensure that the results of the DIADEM Challenge are rapidly and extensively disseminated. This includes the scientific details of the winning algorithms, the advances and residual limits of the solutions formulated, and the practical availability and applicability of the tools for neuroscience labs.
While researchers worked independently during the qualifier and preparation phases, the Final Round and the independent scientific conference gathered leading stakeholders from both communities of data owners — cellular neuroanatomists and algorithm providers (software engineers) — to encourage intense and productive interaction. This joint expertise of these overlapping groups of scientists enabled a unique opportunity for both instantaneous technical troubleshooting and long-term scientific advancement. The Final Round mimicked real-lab dynamics, with algorithm providers and data owners working together toward the common goal of achieving the most effective and satisfactory solution.