Development of a Robustness Evaluation Procedure for Machine Learning-Based Classifiers

Introduction: Biomarkers are measurements of physiological variables used in diagnostic tests to characterize disease states1.  However, in many cases data from multiple variables are required simultaneously for a diagnosis2.  In those situations, Machine Learning (ML) algorithms are used to extract a biomarker from these multiple measurements3.  While many studies have recently been published touting high performance of certain ML-based classifiers used for diagnostic tests, considerably less research has been conducted related to investigating the robustness of the classifiers4.  Therefore, it is the aim of this study to construct a framework for evaluating the robustness of ML-based classifiers.

Methods: Given a classifier and a data set based upon which it was derived, the classifier should be evaluated for robustness. One key component is to ensure that the features that the classifier uses as inputs are relevant and robust over the dataset.  Factor analysis is used here to identify the most statistically meaningful features in a dataset. It is then checked if the inputs to the classifier are chosen from the subset of important features. Next, the dataset is corrupted with two types of noise to determine the impact of variations in the dataset on classifier performance.  Replacement noise is implemented such that a proportion p of data values is randomly removed and replaced with a value randomly drawn from the distribution of the feature which it was removed from.  Subsequently, Gaussian noise is implemented such that every value has a term added (or subtracted) to it which is randomly drawn from a Gaussian distribution of mean 0 and standard deviation s*σ, where s is a dampening factor and σ is the feature standard deviation.  Both noise methods were applied 1000 times for each value of p and s for their respective methods. Finally, the performance of classifiers on several (~1000) training/testing splits is assessed to ensure that any results obtained are not results based upon an outlier.  To verify this evaluation approach, classifiers were developed using five different machine learning algorithms (linear discriminant analysis (LDA), support vector machine (SVM), random forest (RF), partial least squares discriminant analysis (PLS), and logistic regression (LR)).  Features were selected by ranking all combinations of five metabolites by Leave One Out Cross Validation (LOOCV) accuracy.  Data used to develop the classifiers were derived from metabolomics of blood samples from children with and without an autism diagnosis. 

Figure 1. Average accuracy relative to replacement noise

Results:  Two of the five classifiers (LDA, LR) had all five features identified as statistically significant, while PLS and SVM had four significant features and RF only had two.  As such, RF and to a lesser degree PLS and SVM may contain features which are more sensitive to changes in the data, raising concerns about robustness.  Classifier accuracy for all classifiers was found to linearly decrease toward random chance with increasing replacement noise, as denoted in Fig. 1.  Therefore, if a small amount of replacement noise would decrease a classifier’s performance below the minimum acceptable performance, it is not robust. However, this seems to be more of a concern for the quality of the dataset than the ML algorithm used.  Fig. 2 shows that classifier performance variance increases significantly with respect to Gaussian noise levels for RF and PLS.  This is also in line with the expectations that not all of the features of these two classifiers come from the set that was determined significant.  Finally, the testing and LOOCV accuracies were not significantly different among all algorithms over 1000 training/testing splits, and performance variance was small for all classifiers except RF.

Figure 2. Accuracy variance relative to Gaussian noise

Conclusions:  The goal of this study was to develop a framework to evaluate the robustness of a given classifier.  This framework consists of three steps: (a) evaluation of the feature subset as potential inputs, (b) robustness analysis of the accuracy and the variance using different types and levels of noise and (c) evaluation using a variety of training/testing subsets.  This work made use of one dataset comparing five commonly used ML classification techniques. It can be concluded for this data set that the classifiers generated by LDA and LR show similar levels of robustness, while there are some concerns about PLS and more so for RF.

References: 

1.   Califf R. M. Exp Biol Med. 2018;243:213-221

2.   Krassowski M. et al. Front. Genet. 2020;11

3.   Liebal U. W. et al. Metabolites. 2020;10:243

4.   Tötsch, N., Hoffmann, D: PeerJ Comp. Sci. 2021;7

Acknowledgements:  We would like to acknowledge Dr. Uwe Kruger for aiding with the statistical analysis.