A Human-In-the-Loop AI Approach to Identifying Self-Care Content In Virtual Trainings

Background and Purpose. After the COVID-19 pandemic, online training became a common way to mobilize knowledge for practice, overcoming geographic limitations and the related costs of in-person training. This generated extensive virtual meeting data across social service settings. Yet these transcripts remain largely underutilized in research because manual qualitative analysis is labor-intensive. Meanwhile, large language models (LLMs) offer promising tools to address this gap by enabling researchers to process large volumes of conversational text more efficiently. This study examined the feasibility of using LLMs to classify the presence of self-care content in Zoom meeting transcripts and assessed whether a human-in-the-loop (HITL) collaborative framework enhances classification performance relative to fully automated approaches.

Methods. Transcript data were drawn from virtual trainings on SEL for education-sector leaders held between 2023 and 2024 (total sentences: N = 6,932). Transcripts were generated using automated speech recognition, de-identified, and parsed into individual sentences using a custom Python script. An open-source local LLM classified each sentence as self-care-related or not. A human-annotated validation dataset (n = 458) was used to evaluate AI-only performance and to calibrate a HITL workflow in which sentences with model confidence below 90% were flagged for human review. TF-IDF vectorization and Mann-Whitney U tests were applied to compare linguistic patterns across label groups.

Results. AI-only classification achieved an accuracy of 0.82 and an F1 score of 0.73. The HITL approach, in which humans reviewed 30.1% of validation sentences, improved accuracy to 0.92 and F1 to 0.89. When scaled to the full corpus, 1,480 sentences were identified as self-care-related. The proportion of sentences requiring human review varied across transcripts, ranging from 11.21% to 30.90% (mean = 25.02%), with longer meetings generally requiring less human intervention. TF-IDF analyses revealed significant linguistic differentiation between groups (unigrams: U = 485,606, p < .001; bigrams: U = 48,860, p < .001), with terms such as mindful, breathe, and overwhelmed most characteristic of self-care content, while administrative and logistical language dominated non-self-care sentences.

Conclusions and Implications. Findings demonstrate that human-AI collaboration substantially improves classification accuracy while limiting the proportion of cases requiring human review, offering a scalable approach to text analysis. This approach enables researchers and practitioners to assess the extent to which professional development convenings incorporate emotional support and self-care content, providing empirical grounding for fidelity monitoring and continuous quality improvement. However, AI is not a shortcut; the investment required for prompt design and iterative validation still demands considerable time in the early stages.