Applications of Large Language Models In Social Science Research: A Qualitative Case Study

Background and Purpose: Qualitative data analysis can provide researchers with rich and nuanced information about the experiences of specific populations; however, the process for coding and analyzing qualitative can require extensive time and resources, particularly for large datasets. This study examines the potential applications of Large Language Models (LLMs) to enhance qualitative data analysis in mental health and social science. The presentation will use interview data from a study analyzing social anxiety triggers in student veterans. Methods: The current study compared LLM analysis using Google’s NotebookLM of interview data against previously published expert coding of interview transcripts. Analyses evaluated a number of dimensions including agreement on trigger identification, intensity, and response categorization using both quantitative (Cohen’s Kappa, Pearson correlation) and qualitative methods. Results: Overall, we found substantial agreement between LLM-identified and expert-identified anxiety triggers (Cohen’s Kappa = 0.68). There also was a strong correlation between LLM and expert assessments of anxiety intensity, with a Pearson correlation of 0.952 (p < 0.05). Interestingly, the LLM provided more granular categorization while maintaining clinical relevance and revealed patterns not previously identified through traditional coding by researchers. While expert analysis grouped responses into 8 broad, clinically oriented behavioral categories, the LLM identified 46 distinct, phenomenologically specific behavioral responses. This suggests the LLM excels at bottom-up, descriptive identification, while expert analysis provides a top-down, theoretical synthesis. However, LLMs showed several limitations that should be considered- in particular the LLM was less effective in capturing nuanced emotional contexts and veteran-specific cultural references. Conclusions and Implications: LLMs can serve as valuable complementary tools in social work research, reducing analysis time and uncovering new insights. The current methodology, while effective, could be refined to better support both AI-assisted and expert analysis, potentially improving the depth and efficiency of clinical research. However, their integration requires careful consideration of ethical issues and methodological rigor, underscoring the indispensable role of human expertise in interpreting complex clinical data. Overall, the results suggest that LLMs are most appropriate for coding interviews and focus groups that semi-structured protocols that allowed for both flexibility and consistent coverage of key domains. Additionally, real-time digital documentation integrated into the interview process can help to provide structured data points that complement the narrative transcript. Finally, contextual information within the transcription should be systematically captured and preserved through the documentation process to augment the coding process with LLMs.