Estimating the false negatives due to blocking in record linkage
Section 4. Finite mixture model

• Previous
• Next

The linkage of the two sources is of interest when it is a viable option even $N$ is very large. To capture the essence of such situations, the following two regularity conditions are assumed.

1. Two matched records are neighbours with a probability that is bounded away from 0 regardless of  $N.$
2. Two unmatched records are accidental neighbours with a probability of $O(1/N).$

These assumptions imply that each record has a bounded expected number of neighbours and that $O(N)$ pairs (instead of $O(mN)$ pairs and even $O(N^2)$ pairs if $m=O(N)$ ) are selected by the blocking criteria. They also imply that there is enough linkage information to identify matched records with a success probability, which is bounded away from zero, regardless of the population size. The above assumptions further imply a particular limiting distribution for the number of neighbours $n_i.$ Indeed, let $n_i=n_{i|M}+n_{i|U},$ where $n_{i|M}$ is the number of matched neighbours and $n_{i|U}$ is the number of unmatched neighbours. Note that these latter variables are not directly observed expect when $n_i=0$ or $n_i=N$ (see Table 3.1). They are also conditionally independent given $v_i$ and such that $n_{i|M}|v_i~\text{Bernoulli}\text{(}p\text{(}{v}_i\text{)),}$ $n_{i|U}|v_i~\text{Binomial}\text{(}N-1,\lambda (v_i)/(N-1)\text{),}$ if an unmatched record is a neighbour with the probability $\lambda (v_i)/(N-1)$ independently of the other unmatched records. When the functions $p(.)$ and $\lambda (.)$ do not depend on $N$ and $N$ is large, we have $n_{i|U}|v_i\dot{~}\text{Poisson}\text{(}\lambda \text{(}{v}_i\text{))}$ (Billingsley, 1995), where  $\dot{~}$ means approximately distributed as. Hence, $n_i|v_i\dot{~}\text{Bernoulli}\text{(}p\text{(}{v}_i\text{))}\text{*}\text{Poisson}(\lambda (v_i)),$ where $*$ is the convolution operator. Note that, in general, the functions $p(.)$ and $\lambda (.)$ are unknown high-dimensional parameters. To simplify, further assume that $(p(.),\lambda (.))$ is (well approximated by) a piecewise constant function with $G$ levels, such that we have the finite mixture model $n_i~{\displaystyle {\sum }_{g=1}^G{\alpha }_g}(\text{Bernoulli}(p_g)*\text{Poisson}({\lambda }_g))$ holds approximately. When $G$ is fixed, the unknown model parameters are given by the vector $\psi ={\left[({\alpha }_g,p_g,{\lambda }_g)\right]}_{1\le g\le G}$ that may be estimated with the Expectation-Maximization (EM) procedure in the next section.

The connection between the error rates and model parameters is made by first noting that the FNR and FPR definitions imply

$$\begin{array}{ll}\text{FNR}\hfill & =\frac{1}{m}{\displaystyle \sum _{i=1}^m(1-n_{i|M})},\hfill \\ (N-1)\text{FPR}\hfill & =\frac{1}{m}{\displaystyle \sum _{i=1}^m{n}_{i|U}}.\hfill \end{array}(4.1)$$

When $m=N$ almost surely, the above equations imply that

$$\begin{array}{ll}E[\text{FNR}]\hfill & =1-E[n_{i|M}],\hfill \\ \hfill & =1-E[p(v_i)],\hfill \\ (N-1)E[\text{FPR]}\hfill & =E[n_{i|U}]\hfill \\ \hfill & =E[\lambda (v_i)],\hfill \end{array}(4.2)$$

where $E[p(v_i)]={\displaystyle {\sum }_{g=1}^G{\alpha }_g{p}_g}$ and $E[\lambda (v_i)]={\displaystyle {\sum }_{g=1}^G{\alpha }_g{\lambda }_g}$ with the finite mixture model. When $m$ is random and such that

$$\begin{array}{ccc}\frac{1}{m}{\displaystyle \sum _{i=1}^m{n}_{i|M}}& \stackrel{p}{\to }& E[n_{i|M}],\\ \frac{1}{m}{\displaystyle \sum _{i=1}^m{n}_{i|U}}& \stackrel{p}{\to }& E[n_{i|U}],\end{array}(4.3)$$

as $N\to \infty ,$ the error rates and the model parameters are related as follows

$$\begin{array}{rrr}\hfill \text{FNR}& \hfill \stackrel{p}{\to }& \hfill 1-E[p(v_i)],\\ \hfill (N-1)\text{FPR}& \hfill \stackrel{p}{\to }& \hfill E[\lambda (v_i)].\end{array}(4.4)$$

• Previous
• Article main page
• Next