Can Persona-Based LLM Agents Approximate Published Health Preference Findings? a Social Work-Relevant Replication Using Synthetic Respondents

Background and Purpose: Across many areas of social work research, particularly those involving health, services, and public decision-making, researchers need practical, low-cost ways to pilot survey instruments, test scenario framing, and examine responses to complex choices before investing in large human-subject studies. Large language models are increasingly being used for simulation, yet evidence remains limited on whether persona-based LLM agents can reproduce established findings from real human studies. This study is especially relevant to social work because researchers frequently study preference-sensitive issues such as treatment choices, caregiving decisions, service access, and broader policy priorities, often under substantial time and funding constraints. We examined that question by replicating a published U.S. willingness-to-pay per quality-adjusted life-year study using synthetic respondents. Our goal was to assess whether this approach may have practical value for social work and behavioral health research, including survey pretesting, scenario modeling, and early-stage preference research.

Methods: We created a synthetic sample of 1,000 U.S. adult personas based on demographic distributions reported in the original study. Each persona was assigned to an LLM agent and completed the same four valuation scenarios used in the published survey: immediate benefit to self, delayed benefit to self, benefit to a family member, and broader societal coverage. We then compared the aggregate estimates from the agents with the published U.S. benchmark estimates. The analysis focused on how closely the synthetic-agent results reproduced the overall level and ordering of preferences reported in the original study.

Results: The LLM-agent estimates were generally close to the published benchmarks (e.g., immediate self: 62.2k vs. 62.0k; family-member: 71.7k vs. 69.0k). Replication was strongest for immediate self-valuations and family-member valuations, where the synthetic results closely matched the original study. The model also reproduced the overall pattern that societal valuations were high relative to personal valuations. At the same time, performance was weaker for delayed self-benefit, suggesting that time-related preferences were harder for the agents to capture accurately (62.1k vs. 52.0k). Overall, the findings suggest that persona-based LLM agents can recover broad population-level patterns in stated health preferences, even without access to the original respondent-level data, though important gaps remain.

Conclusions and Implications: Persona-based LLM agents may be useful as a low-cost, early-stage tool for social work research when investigators want to pilot survey instruments, test framing effects, or explore how preferences related to health, services, and policy choices shift across scenarios. This potential is especially relevant where resource constraints limit rapid empirical testing. At the same time, synthetic agents should not be treated as replacements for human participants, particularly when research questions depend on nuanced temporal preferences or lived experience. This is especially important in social work research, where decision-making is often shaped by caregiving roles, structural inequality, culture, trauma, disability, and other contextual factors that synthetic personas may not fully capture. Taken together, the findings suggest that LLM-based simulation may serve as a useful support tool for early-stage social work research, but its use should remain cautious, transparent, and grounded in real-world validation.