Estimating the false negatives due to blocking in record linkage
Section 7. Conclusions and future work

• Previous
• Next

A new finite mixture has been proposed for estimating the false negatives due to a standard blocking procedure, when linking a file to a register or a census with complete coverage, when both sources are free of duplicate records. An empirical study with social data gives encouraging results. Yet future work must address the issues of variance estimation and statistical inference about the number of classes. Extensions are also required to account for undercoverage and duplicate records.

Disclaimer

The content of this paper represents the authors’ opinions and not necessarily those of Statistics Canada. It describes theoretical methods that might not reflect those currently implemented by the Agency.

Acknowledgements

The authors express their gratitude towards Dr. Jonnagada Rao for his insight and towards the Public Service Commission for access to the data.

Appendix A

For the E-step, the equations are as follows.

$$\begin{array}{ll}P(n_i|c_{ig}=1)\hfill & =I(n_i=0)(1-p_g)e^{-{\lambda }_g}+I(n_i>0)\left(p_g+(1-p_g)\frac{{\lambda }_g}{n_i}\right)\frac{e^{-{\lambda }_g}{\lambda }_g^{n_i-1}}{(n_i-1)!}\hfill \\ P(c_{ig}=1|n_i)\hfill & =\frac{{\alpha }_{g}P(n_i|c_{ig}=1)}{{\displaystyle {\sum }_{g^{\prime }=1}^G{\alpha }_{g\text{'}}P(n_i|c_{i{g}^{\prime }}=1)}}\hfill \\ P(n_{i|M}=1|n_i,c_{ig}=1)\hfill & =\frac{p_g{n}_i}{p_g{n}_i+(1-p_g){\lambda }_g}\hfill \\ P(n_{i|U}=n_i|n_i,c_{ig}=1)\hfill & =I(n_i=0)+I(n_i>0)\frac{(1-p_g){\lambda }_g}{p_g{n}_i+(1-p_g){\lambda }_g}\hfill \\ P(n_{i|U}=n_i-1|n_i,c_{ig}=1)\hfill & =\frac{p_g{n}_i}{p_g{n}_i+(1-p_g){\lambda }_g}\hfill \end{array}$$

and

$$\begin{array}{ll}E[c_{ig}{n}_{i|M}|n_i]\hfill & =P(c_{ig}=1|n_i)P(n_{i|M}=1|n_i,c_{ig}=1)\hfill \\ E[c_{ig}{n}_{i|U}|n_i]\hfill & =P(c_{ig}=1|n_i)E[n_{i|U}|n_i,c_{ig}=1]\hfill \\ E[n_{i|U}|n_i,c_{ig}=1]\hfill & =\left(\frac{p_g(n_i-1)+(1-p_g){\lambda }_g}{p_g{n}_i+(1-p_g){\lambda }_g}\right)n_i.\hfill \end{array}$$

References

Armstrong, M., and Mayda, J. (1993). Model-based estimation of record linkage error rates. Survey Methodology, 19, 2, 137-147. Paper available at https://www150.statcan.gc.ca/n1/en/pub/12-001-x/1993002/article/14459-eng.pdf.

Belin, T., and Rubin, D. (1995). A method for calibrating false-match rates in record linkage. Journal of the American Statistical Association, 90, 694-707.

Billingsley, P. (1995). Probability and Measure, Wiley.

Blakely, T., and Salmond, C. (2002). Probabilistic record linkage and a method to calculate the positive predicted value. Journal of Epidemiology, 31, 1246-1252.

Bohensky, M. (2016). Bias in data linkage studies. In Methodological Developments in Data Linkage, (Eds., K. Harron, H. Goldstein and C. Dibben), Chichester: Wiley, 63-82.

Chambers, R., and Kim, G. (2016). Secondary analysis of linked data. In Methodological Developments in Data Linkage, (Eds., K. Harron, H. Goldstein and C. Dibben), Chichester: Wiley, 83-108.

Chipperfield, J., Hansen, N. and Rossiter, P. (2018). Estimating precision and recall for deterministic and probabilistic record linkage. International Statistical Review, 86, 219-236.

Christen, P. (2012). Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution and Duplicate Detection, Springer.

Daggy, J., Xu, H., Hui, S., Gamache, R. and Grannis, S. (2013). A practical approach for incorporating dependence among fields in probabilistic record linkage. BMC Medical Informatics and Decision Making, 13, 1-8.

Dempster, A., Laird, N. and Rubin, D. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society Series B, 39, 1-38.

D’Orazio, M., Di Zio, M. and Scanu, M. (2006). Statistical Matching, Theory and Practice, Wiley.

Fellegi, I., and Sunter, A. (1969). A theory of record linkage. Journal of the American Statistical Association, 64, 1183-1210.

Han, Y., and Lahiri, P. (2018). Statistical analysis with linked data. International Statistical Review, 87, special issue, 139-157, 1C19 doi:10.1111/insr.12295.

Herzog, T., Scheuren, F. and Winkler, W. (2007). Data Quality and Record Linkage Techniques, Springer.

Hoeffding, W. (1948). A class of statistics with asymptotically normal distributions. Annals of Statistics, 19, 293-325.

Lee, A.J. (1990). U-Statistics: Theory and Practice, Marcel Dekker.

Newcombe, H. (1988). Handbook of Record Linkage, Oxford University Press.

Sariyar, M., Borg, A. and Pommerening, K. (2011). Controlling false match rates in record linkage using extreme value theory. Journal of Biomedical Informatics, 44, 648-654.

Statistics Canada (2017a). Record Linkage Project Process Model, Catalogue No. 12-605-X.

Statistics Canada (2017b). Income Reference Guide, Census of Population, 2016, Catalogue No. 98-500-X2016004.

Thibaudeau, Y. (1993). The discrimination power of dependency structures in record linkage. Survey Methodology, 19, 1, 31-38. Paper available at https://www150.statcan.gc.ca/n1/en/pub/12-001-x/1993001/article/14477-eng.pdf.

Varin, C., Reid, N. and Firth, D. (2011). An overview of composite likelihood methods. Statistica Sinica, 21, 5-42.

Winkler, W. (1993). Improved decision rules in the Fellegi-Sunter model of record linkage. In Proceedings of the 1993 Joint Statistical Meetings, American Statistical Association, 274-279.

Winkler, W.E. (2016). Probabilistic linkage. In Methodological Developments in Data Linkage, (Eds., K. Harron, H. Goldstein and C. Dibben), Chichester: Wiley, 7-35.

• Previous
• Article main page
• Next