Machine-Learning-Assisted Screening to Update a Systematic Review on Pediatric Psychotropic Medication Use

Background and Purpose: Systematic reviews provide the foundation for evidence-based social work practice. However, updating them has become increasingly labor-intensive due to the exponential growth of scientific literature. In the area of pediatric psychotropic medication use, social work researchers face longstanding concerns regarding diagnostic uncertainty, gaps in evidence about safety and effectiveness, and inconsistent prescribing practices. Updating the knowledge base is essential to child health but demands massive human labor. We addressed this methodological challenge by assessing the extent to which supervised machine learning can reduce the human workload required to update a systematic literature review on psychotropic medication use in U.S. children. Our primary objective was to develop and evaluate a predictive model that prioritizes studies for full-text review while maintaining 95% recall, i.e., excluding no more than 5% of eligible studies.

Methods: We used a training dataset of 19,567 bibliographic records from a prior systematic review (2011–2015). We de-duplicated records by comparing normalized article titles and also through an algorithm called Locality Sensitivity Hashing (LSH), yielding 16,633 unique records. Only 7.5% of the articles in the de-duplicated sample were relevant to the systematic review. We evaluated three text representation methods: Term Frequency-Inverse Document Frequency (TF-IDF; 100 variables), Doc2Vec (150 variables), and domain-specific PubMedBERT sentence embeddings (768 variables). Across the three text representations, we evaluated five base learners: penalized logistic regression, support vector machine (SVM), XGBoost, random forest, and Naive Bayes. After excluding SVM and XGBoost due to model convergence failures, we used five-fold cross-validation to train the remaining models. We also implemented a stacked ensemble approach via the SuperLearner package to optimize predictive performance. We primarily evaluated the models using Work Saved over Sampling at 95% recall (WSS@95, calculated as the true negative rate at the operating point where recall equals 95%, minus 0.05), which quantifies the reduction in manual screening relative to random sampling. We also calculated the area under the precision-recall curve (AUC-PR) to assess overall model performance.

Results: PubMedBERT embeddings significantly outperformed other representations by capturing semantic context rather than simple keyword frequency. The combination of PubMedBERT embedding and SuperLearner emerged as the optimal model, achieving a WSS@95 of 0.687 (AUC-PR 0.627), i.e., capturing 95% of relevant studies while allowing for a 68.7% reduction in manual screening effort compared to random sampling. The TF-IDF and Doc2Vec embeddings yielded a maximum WSS@95 of 0.007 and 0.127 respectively. Calibration diagnostics indicated systematic overconfidence (tendency toward false positives) across models, underscoring the necessity of using a recall-based threshold rather than a fixed probability cutoff.

Conclusions and Implications: Integrating domain-specific language models like PubMedBERT with ensemble modeling can reduce the human labor required for systematic review updates by nearly 70% without compromising study identification. These findings support timely, evidence-based practice and policy by identifying a technologically rigorous, scalable solution for social work researchers to manage the substantial labor costs associated with maintaining a current evidence base in a rapidly evolving field like pediatric mental health.