Using Plasmode Datasets to Compare Statistical Methods for RNA-seq Analysis

RNA-seq has been rapidly adopted as the preferred technology for gene expression analysis and the development of bioinformatics and statistical methods has accompanied this process. Such methods are difficult to validate due to the complexity of RNA-seq data. Parametric simulations are commonly used to compare statistical analysis methods. However, simulated datasets can hardly mimic the complexity of real ones and they depend on model parametric assumptions. To overcome such limitations, we propose using plasmode datasets to validate analysis methods. A plasmode is a real dataset for which some truth is known. Therefore, it is possible to set scenarios consistent with null and alternative hypotheses to test analysis methods under realistic conditions. We illustrate this procedure by comparing the performance of three commonly used programs for RNA-seq data analysis. We found that all programs generated inflated type I error rates. However, once type I error rate was adjusted, DESeq and edgeR yielded similar power. Contrarily, fitting Gaussian models to transformed read counts, controlled type I error rate but yielded less power. These results were more evident using plasmodes compared to using parametric simulations. However, reference datasets needed to expand results to other scenarios are not yet available. As RNA-seq applications continue to increase, new methods will be developed and plasmodes could be used to compare established and novel analysis tools on a more realistic basis. These results emphasize the need of promoting and improving systematic data sharing across the research community for plasmode construction and methods testing.