Development of a Person-Centered Care Dictionary for Clinical Notes: An LLM-Assisted Approach

Background and Purpose: Person-centered care (PCC) is a recognized quality indicator in behavioral health, yet no validated dictionary exists for measuring PCC language in clinical documentation. Manual chart review remains the primary assessment method, limiting scalability across large electronic health record systems. Dictionary-based text analysis offers a transparent, reproducible alternative that requires fewer computational resources than machine learning classifiers, but constructing a domain-specific dictionary for a construct as broad as PCC is costly and time-intensive when done through traditional expert compilation or corpus-based statistical extraction alone. This study developed a PCC dictionary for behavioral health clinical notes using a multi-model large language model (LLM) approach, treating cross-model agreement as a form of inter-rater reliability and combining LLM-generated candidates with expert adjudication. This design reduces the time and cost of initial candidate generation while preserving human judgment.

Methods: First, two domain experts defined PCC pillars grounded in the patient-centered and person-centered care literature. Shared principles across these paradigms were identified and synthesized into five core PCC domains: Holistic (with six sub-domains), Individualized, Partnership, Service User Perspective, and Strengths-Based. Second, the pillar definitions and a list of seed words were used as prompts for six LLMs, each independently generating words clinicians might use in progress notes documenting care consistent with each pillar. Third, inter-model agreement was calculated as the number of LLMs that independently generated each word. Fourth, three expert annotators independently reviewed all 432 candidate words using a custom swipe-based annotation tool, coding each as include or exclude. Inter-rater agreement was assessed using Gwet's AC1, which corrects for prevalence imbalance in binary coding tasks. The group conducted iterative annotation with discussion of pillar definitions and re-rating. Response time per annotation was recorded as a secondary indicator of coding difficulty.

Results: Six LLMs generated 432 candidate words across five PCC pillars, with the Holistic pillar containing six sub-domains (Community, Health/Physical, Occupation, Safety & Security, Social, Spiritual). In the initial annotation round, 331 words (76.6%) received unanimous inclusion from all three annotators, and 2 received unanimous exclusion. Gwet's AC1 was 0.817 overall, indicating substantial agreement. Of the 99 words with disagreement, 97 were re-annotated by four raters after team discussion. In the second round, 86.6% reached a strong majority (three or more of four raters). The combined Gwet's AC1 across all 453 evaluated words was 0.906, indicating excellent agreement. Items with disagreement took longer to annotate than items with unanimous agreement (Mdn = 3.7s vs. 2.0s, p < .001). The final dictionary retained 408 words.

Conclusions and Implications: This study presents a replicable, cost-efficient method for constructing domain-specific dictionaries by using multiple LLMs to generate candidate terms and expert review to make final inclusion decisions. The LLM-based generation step produced a large initial candidate set at minimal cost compared to traditional expert-only compilation, while the multi-model consensus approach provided an empirical basis for prioritizing terms before human review. The resulting PCC dictionary, once validated, can support large-scale assessment of person-centered language in behavioral health documentation, both for research purposes and clinic-level quality improvement.