Training Task-Specific Language Models for Social Work Research: Active Learning and Uncertainty Sampling for Efficient Text Analysis

Social work researchers increasingly use language models to analyze unstructured text, such as classifying case narratives, extracting themes from clinical notes, screening administrative records, and coding open-ended survey responses. Yet methodological training in the field has largely treated these models as off-the-shelf tools to prompt rather than systems that can be trained on a researcher’s own data. This distinction matters: a small model trained on task-specific labeled examples can match or exceed the performance of a model many times its size used out of the box. Training transforms modest, locally deployable models into tools that rival far larger systems—without requiring specialized hardware, cloud infrastructure, or transmitting confidential data to external servers. This workshop teaches participants to train (fine-tune) task-specific language models for text analysis, emphasizing active learning and uncertainty sampling for building high-performing models with minimal labeling effort. Because trained models are small enough to run locally, they also carry only a fraction of the energy and carbon cost of relying on large cloud-hosted systems for the same tasks.

Training a task-specific model is straightforward when the right data are in the right shape. The more consequential methodological challenge—and the central focus of this workshop—is how researchers can strategically select which cases to label to maximize performance with the fewest human-coded examples. Active learning addresses this challenge by using the model’s own uncertainty to identify the most informative cases for human review. Uncertainty sampling selects cases where the model is least confident—the cases with the greatest potential to improve the model when labeled. This iterative cycle of training, identifying uncertain cases, labeling, and retraining enables strong performance with substantially fewer labeled examples than random sampling, considerably reducing annotation burden while maintaining accuracy.

The workshop covers four components:

1. Conceptual Overview: An accessible introduction to two model families for text classification—small generative models and encoder classifiers—covering when each architecture is appropriate, their trade-offs in speed, flexibility, and hardware requirements, and why both benefit from training on labeled data.
2. Hands-On Fine-Tuning: Demonstration of preparing labeled data and fine-tuning a language model on a social work classification task, including data formatting, training configuration, and performance evaluation.
3. Active Learning Pipeline: The core segment, in which participants implement uncertainty sampling to interpret model confidence scores, rank unlabeled cases by uncertainty, and design efficient annotation workflows that minimize labeling effort.
4. Evaluation and Learning Curves: Participants evaluate model performance using cross-validation and learning curve analysis, interpreting how accuracy scales with training set size and when additional labeling yields diminishing returns.

Pedagogical techniques include worked examples using de-identified case narratives, interactive exercises comparing random versus uncertainty-based selection, and visual learning curve analyses illustrating active learning’s efficiency gains. All materials use open-source tools deployable on standard hardware without specialized infrastructure.

Participants will leave equipped with the understanding and technical foundation to train and deploy task-specific language models with active learning on their own data. The approach has direct implications for research involving confidential text records, resource-constrained settings, and corpus-scale analyses where reproducibility and local deployment are essential.