Enabling Historical Longitudinal Studies with Algorithmic Record Linkage

Traditional methods of identifying and linking different instances of the same individual found in historical records can be labour intensive and time consuming, but it is an important tool for historical research. Adopting an algorithmic approach – where a set of rules created by the historian are applied to the data in order to identify matches – can accelerate this process and facilitate this type of research on a much broader scale, whilst also leading to more easily reproducible findings. However, such an approach also creates a new set of challenges – particularly in terms of dealing with edge cases where human intuition is not easily replicated, and may result in false positives or negatives.

Through the lens of our recent work on linking individuals between decadal censuses for England and Wales, we discuss the practical implications of employing such techniques in historical research – namely, the significant challenges of reconciling the varied reporting of supposedly immutable characteristics (such as birthplace or birth year), differentiating between records with very similar individual profiles, and finding suitable methods for evaluating the success rate of the matching process.

With the application of this approach, one major research area has been the in-flows and out-flows of individuals from different occupations – particularly in relation to industrial change and mechanisation. By undertaking this process at scale, we have been able to demonstrate and explore the place-specific nature of socio-demographic profiles associated with emerging industries.