Evaluating Locally Deployed Small Language Models for Classifying Intimate Partner Violence In Suicide Narratives

Background and Purpose: Qualitative analysis plays a critical role in understanding intimate partner violence (IPV) within the context of suicide. While qualitative coding enables social work researchers to capture nuanced insights from suicide narratives and related IPV experiences, it is often labor-intensive and may expose researchers to emotionally distressing or traumatic content. These challenges are amplified when analyzing large-scale narrative data. Large language models (LLMs) offer a promising approach by improving efficiency and reducing exposure to sensitive material. However, cloud-based LLMs are often restricted in social work research due to data privacy and confidentiality concerns.

Locally deployed small-scale LLMs provide a viable alternative by enabling secure, offline analysis that aligns with data protection requirements and can be implemented on standard personal computers. This study aims to (1) evaluate the feasibility of using locally deployed LLMs to classify IPV in suicide narratives and (2) examine whether multi-model approaches improve classification performance.

Methods: The dataset consisted of 774 suicide cases involving immigrant women drawn from the National Violent Death Reporting System (2013–2019). Four open, small-sized LLMs (Llama 3.2, Gemma 3, Qwen 2.5, and Phi-3 Mini) were deployed on a personal computer via Ollama. Using 10% of the data, we conducted a pilot test for prompt development through an iterative three-stage process to improve consistency. Models were instructed to classify the presence of physical IPV that was recent or related to circumstances preceding the suicide, based on explicit evidence in the narratives. All analyses were conducted in R within an offline environment. We first evaluated the performance of individual models, then applied two multi-model approaches (agreement across three or more models; high recall + accurate filtering) by combining outputs across models. Performance was assessed against human-coded data using accuracy (the proportion of correctly classified cases), precision (accuracy in identifying IPV cases), recall (sensitivity in identifying true IPV cases), F1 score (harmonic mean of precision and recall), and Cohen’s kappa (agreement between human and LLM coding).

Results: Substantial variation across models was observed. Qwen achieved the highest accuracy (0.92) but very low recall (0.07), indicating a conservative approach that missed most IPV cases. Gemma showed very high recall (0.95) but low precision (0.13), suggesting over-identification. Llama demonstrated the most balanced performance, with the highest F1 score (0.32) and Cohen’s kappa (0.24). Multi-model approaches yielded modest improvements, with the best combination (high recall + filter) improving agreement with human coding. Review of misclassified cases suggests patterns in LLM errors.

Conclusions and Implications: Overall, findings suggest that small LLMs showed limited performance. The observed trade-offs between sensitivity and precision suggest only modest agreement with human coding, indicating that small LLMs are not yet sufficiently reliable as standalone tools for complex qualitative classification tasks. Multi-model approaches showed potential to improve classification performance by leveraging complementary model strengths. Future research can further examine the utility of multi-model approaches based on the strengths and weaknesses of each model, explore the potential of hybrid human–AI workflows, and leverage model fine-tuning to improve reliability and applicability in social work research.