The Map Is Not the Territory: Limits of Generalization In Algorithmic Risk Assessment across Criminal Justice Contexts

Background and Purpose:

Risk assessment algorithms are constantly used in the U.S. criminal legal system to inform decisions related to bail, sentencing, and probation. These tools are often presented as objective and as transferable across jurisdictions, despite being developed within specific institutional contexts. At the same time, the literature on criminology emphasizes that crime and justice outcomes are shaped by place, institutions, and the processes of data collection. The question at the center of this study is whether predictors of recidivism vary across contexts, or whether similar patterns emerge despite differences in place and data production. The analysis also examines what these differences mean for the use of algorithmic tools in criminal justice policy and practice.

Methods:
This study uses the COMPAS dataset from Broward County, Florida, and the NIJ Georgia Recidivism dataset. Although representing different populations, they both include measures of recidivism, with recidivism defined as a new arrest within two years. They also contain comparable demographic and criminal history variables. I begin the analysis by using descriptive statistics to characterize each dataset and identify differences in population composition. Then, I use logistic regressions for each dataset to examine the association between recidivism and other independent variables, such as age, prior criminal history, race, and gender. Finally, I compare the direction, magnitude, and statistical significance of coefficients across models, with particular attention to differences between structural and demographic variables.

Findings:

Results indicate an inconsistent pattern across datasets. Age stands out as a relatively stable predictor, with younger individuals more likely to reoffend. In contrast, other variables vary substantially. Associations involving race are not consistent in either direction or statistical significance. The effects of prior criminal history and gender are significant in both datasets but vary considerably in magnitude. These results suggest that while some predictors reflect broader and known patterns, others are more influenced by local institutional dynamics than we initially thought – including differences in policing practices, court processes, and data construction.

Conclusions and Implications:

These findings have important implications for the use of algorithmic tools in criminal justice decision-making. When algorithms designed in one jurisdiction are implemented elsewhere without adjusting parameters, the predictions are unlikely to reflect local realities, which risks reinforcing existing inequalities. This has direct consequences for democratic governance, as data-driven tools increasingly influence decisions about resource allocation, community safety, and carceral policies. For social work, the results reinforce the importance of critically questioning its effects on local communities. Ensuring that predictive tools are used responsibly requires not only technical accuracy but also attention to equity, human dignity, and the broader democratic goals of the criminal justice system.