Assessment of Strategies for False Discovery Rate Control and their Implications in Computational Biology

Introduction: The analysis of measurements from metabolomics, transcriptomics, epigenetics, or metagenomics, often requires hundreds or even thousands of hypothesis tests which concurrently compare two or more groups. Multiple hypothesis testing becomes essential under those circumstances to control the overall rate of false discoveries. Commonly employed techniques for correcting the multiple hypothesis testing problem include the Bonferroni method, the Benjamini-Hochberg (BH) procedure, and the Storey-Tibshirani approach [1]. However, each method comes with its own set of limitations. The Bonferroni method utilizes a conservative readjustment of significant p-values based on the total number of hypotheses tested, regardless of dependency and sample size. The handling of dependencies among hypothesis tests is also one of the principal limitations of the Benjamini-Hochberg test, potentially leading to increased false discoveries in correlated data sets. While some studies have demonstrated the use of a leave-n-out approach within the Storey-Tibshirani procedure, a comprehensive comparison against alternative techniques is lacking [2]. This work aims to fill this gap by systematically comparing these methods under different assumptions to evaluate their performance under distinct conditions. 

Methods: This study employed MATLAB (MathWorks, Natick, MA) to create synthetic datasets simulating biological data across various conditions. A custom function was developed to generate simulated data for comparing two subsets, either drawn from the same population or alternative populations. The generative function enabled users to adjust parameters such as the simulated data distribution type, number of variables, standard deviation and mean range, maximum allowable effect size, sample size, and proportion of correlated samples. The datasets, generated through this function's modulation, underwent univariate hypothesis testing. The p-values then underwent testing using the Benjamini-Hochberg, Bonferroni, and leave-n-out methods. The accuracy of the predicted relationships between groups was evaluated across all multiple hypothesis testing correction methods, utilizing an adjusted p-value of 0.05 and FDR rate of 0.10. To investigate the impact of specific parameters, a chosen variable in the equation was isolated and manipulated, while other factors were held constant.

A preliminary analysis was performed using generated normally distributed data, evaluating the proportion of significant findings ranging from 0.01 to 0.50, the number of hypotheses tested ranging from 10 to 10,000, and the sample size ranging from 5 to 1000. Initial introduction of correlation was made only for the proportion of significant findings. A total of 100 iterations were conducted to assess possible alterations in false discovery rates, utilizing the central limit theorem to determine the average performance across conditions. For the initial analysis, a leave-1-out approach was utilized. 

Results: The number of hypothesis tested had limited impact on the accuracy of the leave-n-out approach, while the Bonferroni method had the greatest decrease in accuracy. The proportion of significant findings emerged as a pivotal factor influencing accuracy across all methods, but to varying degrees (r=0.-79, r=0.-76, r=-0.64 for Bonferroni, BH, and leave-1-out, respectively). Notably, in cases with high correlation or  proportion of significant findings, the leave-n-out approach consistently outperformed others. However, when BH and Bonferroni assumptions held true, these techniques consistently surpassed the accuracy of the leave-n-out approach. However, this is not surprising as these methods were specifically designed for these cases.

Conclusions: The findings highlight the efficacy of employing a leave-n-out approach under specific conditions, notably in data sets with high correlations or a large proportion of significant findings. This insight offers valuable considerations for the optimal application of multiple hypothesis testing techniques across diverse statistical scenarios. Future endeavors will expand the analysis protocol to enhance optimization under additional conditions and include comparisons with various real-world datasets. Such extensions aim to provide practical guidance for researchers in choosing the most appropriate technique tailored to their specific data, ensuring informed decision-making in statistical analyses.