Covariance Structure, Reliability, and Validity Under Bot Contamination

Background: Online survey research in social work is increasingly vulnerable to bot-generated responses, creating significant threats to data quality, measurement integrity, and the validity of substantive inferences. While prior work has focused largely on identifying suspicious cases, less attention has been paid to how automated responding alters the psychometric performance of survey measures themselves. This study examined whether bot contamination distorts covariance patterns, factor structure, reliability, and validity in a widely used measure of intimate partner violence.

Methods: We analyzed 218 completed responses from an open-access online survey collected during the COVID-19 pandemic. Responses were classified as bot or human using a two-pronged strategy that combined a manual review with an unsupervised algorithmic detection method. We compared bot and human cases on the Revised Conflict Tactics Scale (CTS2) using descriptive statistics, group comparisons, regression models, and confirmatory factor analysis. Psychometric evaluation focused on model fit, inter-item covariance patterns, internal consistency, convergent validity, and composite reliability across the full sample and by classification group.

Results: Manual and algorithmic classification showed fair-to-moderate agreement, indicating that a multi-method approach is necessary for identifying suspicious survey responses. Importantly, bot and human cases differed substantially in their psychometric profiles. Bot responses produced unusually dense and uniform covariance matrices, especially within the physical assault subscales, suggesting patterned responding rather than construct-driven variation. In contrast, human responses showed more differentiated covariance structures and the expected domain-specific clustering. Multigroup CFA demonstrated poor global fit when bot and human cases were modeled together, indicating that the latent structure of the CTS2 was disrupted by automated cases. Although internal consistency estimates such as Cronbach’s alpha and McDonald’s omega were often acceptable across groups, these coefficients did not capture the measurement instability introduced by bot data. Human responses showed stronger average variance extracted and more stable composite reliability, whereas bot responses showed weaker convergent validity and lower shared variance, despite sometimes high reliability coefficients.

Implications: These findings suggest that bot contamination is not simply a data-cleaning problem; it is also a psychometric problem that can distort covariance structure, weaken construct validity, and compromise interpretation of scale scores. The study highlights the need for social work researchers to go beyond standard reliability checks and examine covariance matrices, factor loadings, and validity indices when evaluating online survey data. This is especially important in open-access or incentivized surveys, where automated responding may be more likely to occur. For research design and measurement in social work, the results underscore the value of layered detection strategies, transparent screening criteria, and psychometric diagnostics that can identify whether collected data truly support valid measurement of the intended construct. These detection strategies are imperative to ensure that the evidence base upon which policy is designed accurately reflects lived experiences and does not cause further harm.