Toward Rigorous Data Science: A Systematic Review and Evaluation of AI and Data Science Methods In Social Work and Welfare Policy

Background and Purpose: The growing interest in data science within social welfare has increased significantly. However, guidance on appropriate methods and substantive perspectives that address multidisciplinary audiences remains limited. This study evaluates the current state of AI and data science methods in social welfare and related fields by mapping the primary research topics, data sources, and analytical methods. Building on this evaluation, we assess the quality of existing data science scholarship, identify strengths and gaps in the literature, and propose a framework for developing rigorous data science research.

Methods: We employed a two-stage approach to identify and assess data science/AI literature. The first stage combined the terms "social work," "social welfare," and "data science," restricted to peer-reviewed sources, yielding 147 original articles published between 2018 and 2024. According to their main thrust, retrieved articles were categorized into one of four headings: i) data, ii) methods, iii) substantive areas, and iv) conceptual/normative foundations and applications. In the second stage, 59 articles were selected based on three core data science criteria: the incorporation of nontraditional or unstructured data, the use of machine learning methods, and a substantive social science application. We constructed an analytic framework anchored in the key stages of the machine learning pipeline and applied it to assess the methodological completeness of each study. The framework follows sequential steps encompassing data collection, preprocessing, feature engineering, model selection, hyperparameter tuning, training and validation, model evaluation, and policy decision-making.

Findings: Nearly a third of reviewed articles (30%, 44 out of 147) engaged primarily at the conceptual or policy level, consistent with the social work profession's values-driven identity. Social work scholars notably frontload discussions around algorithmic discrimination, ethics, and fairness, critically reflecting on how the profession adopts these technologies. Substantive areas represented the second most common category at 26%, with child welfare as the dominant sub-area. Text documentation is the leading form of nontraditional data, most often utilized through administrative data linkages. Among the 59 articles examined in the second stage, social work research using data science methods consistently addressed model selection and evaluation. Supervised machine learning for risk prediction, particularly the Random Forest algorithm, was the predominant approach, and comparisons across ML algorithms were limited. Studies broadly fell short in reporting data preprocessing procedures, hyperparameter tuning, evaluation metric trade-offs, and policy evaluation beyond model predictions.

Conclusions and Implications: Social work and welfare literature places greater emphasis on conceptual and ethical frameworks than on technical innovation and the adaptation of AI and data science approaches. Analytic work is most established in predictive risk analytics in child welfare using text data, reflecting the field's priorities and its relative narrow methodological scope. The field stands to benefit from diversifying topics, data sources, and analytical methods. Future research should attend more carefully to underreported stages of the data science pipeline and bring policy implications beyond prediction to the forefront of empirical work. The proposed framework provides a guide for researchers and practitioners seeking to systematically incorporate data science methods into social work research.