Variance Adaptive Gradient Boosted Bootstrap Modeling (VAGBBM) for Predicting Substance Use Treatment Completion

Background: Substance use conditions remain a major public health challenge in the United States, with persistently low completion of treatment among persons receiving care and substantial variability in outcomes. Existing statistical models often fail to balance predictive accuracy, calibration, stability, and clinical usefulness. This study introduces Variance Adaptive Gradient Boosted Bootstrap Modeling (VAGBBM) to address these limitations.

Research Design and Methods: A retrospective analysis was conducted using over one million treatment episodes from the Treatment Episode Data Set. VAGBBM integrates gradient boosting with bootstrap resampling and variance adaptive weighting to improve model stability. Logistic calibration was applied to enhance probability estimates. Model performance was evaluated using discrimination, calibration, and classification metrics, while decision curve analysis assessed clinical utility. Model interpretability was examined using SHapley Additive exPlanations (SHAP) values. Data management was performed in Stata v18, and modeling was conducted in RStudio.

Results: The model demonstrated acceptable discrimination (AUC = 0.717; AUPRC = 0.725) and moderate classification performance (accuracy = 0.676; balanced accuracy = 0.659). Sensitivity was 0.586 and specificity was 0.811, indicating stronger identification of noncompletion cases. Precision was 0.686, F1 score was 0.581, and the Matthews correlation coefficient was 0.335, reflecting modest overall classification strength. Calibration was moderate (Brier score = 0.202; log loss = 0.584), with significant miscalibration (p < .001), characterized by underestimation at low probabilities and overestimation at high probabilities. The optimal classification threshold was 0.438. Decision curve analysis showed the highest net benefit relative to default strategies across threshold probabilities from 0.05 to 0.45. SHAP analysis identified structural predictors as dominant, including no arrest history (0.061), public insurance (0.038), no insurance (0.022), Medicaid (0.020), and private insurance (0.014). SHAP further revealed substantial individual level heterogeneity, with predicted probabilities ranging from 0.317 to 0.409 relative to a baseline of 0.444, driven primarily by structural and socioeconomic factors. Overall, SHAP confirms high interpretability and consistent explanation of model behavior across observations.

Conclusion and Implications: The proposed VAGBBM represents a methodological advancement that integrates predictive accuracy, calibration, stability, and clinical utility. By explicitly addressing variance and probability reliability, the framework overcomes key limitations of conventional ensemble approaches in health research. The model provides a scalable and interpretable approach for identifying individuals at risk of treatment noncompletion, supporting targeted intervention strategies and more efficient allocation of resources within substance use treatment systems. More broadly, this approach offers a generalizable template for developing robust, interpretable, and clinically relevant predictive models in large observational datasets.