A Framework for Automated Validation and Comparison of Models of Neurophysiological and Neurocognitive Biomarkers of Psychiatric Disorders

Research on psychiatric disorders has gradually shifted its focus fromcomplex clinical phenotypes towards the identifcation of biomarkers and endophenotypic measures. Computational approaches havegained signifcantly more attention over the last years, and this hasled to the emergence of ‘Computational Psychiatry’ as an independentdiscipline. Computational modelling of biomarkers promises to morereadily shed light on the mechanisms underlying disorders and tofacilitate the discovery of novel medications [1]. However, in order todevelop a computational model, scientists need to have an in-depthunderstanding of the current, relevant experimental data, the currentstate of computational modeling and the state-of-the-art of statisticaltesting. Based on this knowledge, they have to choose the appropriate criteria with which the model predictions and experimental observations will be compared [2]. In a feld where both the number ofexperimental and computational studies grows rapidly, as is the casefor psychiatry, this becomes more and more impracticable. Omar et al.therefore proposed a framework for automated validation of scientifcmodels, SciUnit [3]. Here, we propose to adopt this framework for thecomputational psychiatry community and to collaboratively buildcommon repositories of experimental observations, computationalmodels, test suites and tools. As a case in point, we have implementedtest suites for auditory steady-state response defcits in schizophrenicpatients, which are based on observations from several experimentalstudies [4–6], and we demonstrate how existing computational models [6, 7] can be validated against these observations and comparedagainst each other. We have included sets of observations from threeexperimental studies, which concur on most fndings but also disagreeon some. This allows us to demonstrate the usefulness of our approachin highlighting and clarifying existing, potentially conficting, experimental data. We have included computational models that not onlycomprise biophysically detailed as well as abstract models, but thatalso difer in implementation (native Python vs. Genesis vs NeuroML2),in order to demonstrate the fexibility of the approach. Furthermore, this additionally allows us to showcase the ability of the framework tocompare models against each other based on a set of experimentalobservations. Furthermore, our approach enables us to assess the variability of the produced model output, and therefore the robustness ofthe fndings, by generating a distribution of model instances wherecertain parameters, such as the precise timing of noise (however, notstrength and type of noise) or the precise connectivity (however, notthe distribution of connections) vary, which then are used to producea distribution of model outputs. This can inform on the robustness ofthe fndings and be compared against the variability of experimentalobservations.