Multiple-frame surveys for a multiple-data-source world
Section 4. Multiple-frame surveys and data integration
Rao (2021) reviewed a number of data integration methods for
combining information from a probability sample
assumed to come
from a frame with complete coverage, with information from a nonprobability
sample
often a census
of part of the population as in Section 3.4. Rao considered two cases for
making inferences about
:
(1)
is observed in
both samples, and (2) auxiliary information
is observed in
both samples but
is observed
only in
In this section
I examine various data integration methods from the perspective of the
multiple-frame paradigm and the assumptions in Section 2.2.
4.1 Small area estimation
Small area estimation can
be considered to be a special case of a dual-frame estimation problem in which
Assumption (A6) is not met. Here,
is a
probability sample from Frame 1 and Frame 2 is often an
administrative data source. Both frames are assumed to have complete coverage
of the population, but the variable of interest
is measured
only in
Auxiliary
information
used to predict
is measured in
both samples. Beaumont and Rao (2021) discussed integrating probability and
nonprobability samples through the use of the Fay-Herriot (1979) estimator with
small area estimation techniques.
A composite small area
estimator (Rao and Molina,
2015) of the population mean
in area
is of the form
where
is the direct estimator for the sample mean in
area
from
(which may have large variance or may not
exist),
is a predicted value from a regression model,
and
is a compositing factor. For the Fay-Herriot
estimator,
depends on the relative precision of the two
estimators under an assumed regression model whose parameters are estimated
from
For the estimator
the variable
is measured differently in the two frames
predicted values are
used for Frame 2
and different
compositing factors are used in different areas.
4.2 Mass imputation and sample matching
Suppose that
is a
full-response probability sample from Frame 1, but the variable of
interest
is not measured
in
However,
is measured in
from Frame 2,
and auxiliary variables
are measured in
both samples. Let
be the
predicted value of
from an
imputation model, relating
to
that is
developed on
and let
and
be the
estimated population and domain-
totals from
using the
imputed values.
Similarly to small area
estimation, mass imputation fits into the dual-frame context by relaxing
Assumption (A6) of no measurement error. Kim and Rao (2012) and Chipperfield,
Chessman and Lim (2012) considered the situation where both frames are
complete and
and
are both
probability samples. The frames can differ
Frame 1, for example, might be an area
frame and Frame 2 might be a population register
but both are assumed to have full coverage. Chipperfield et al. (2012) used a composite estimator
where the optimal value of the compositing factor
minimizes the variance (considering both the
sampling and imputation variability). Kim and Rao (2012) proposed adding a correction
for bias with the estimator
this estimator is of the same form as (4.1) with
if the estimated parameters in the imputation
model are required to satisfy
If the imputation model
produces unbiased and accurate predictions for
combining the
samples augments the effective sample size for calculating estimates. When both
samples are probability samples with full coverage, it is possible to perform
model diagnostics on
Chipperfield et al. (2012) suggested several diagnostics, including testing the
imputation model on small areas, investigating whether it is possible to
predict survey membership from the value of
(for
or
(for
and studying
the sensitivity of the mean squared error to different levels of bias in
The sensitivity
of the diagnostics, however, depends on the quality and size of
If
is small
relative to
may contain
subpopulations that are not well represented in
and are poorly
fit by the imputation model.
The situation becomes
more complicated when Frame 2 is incomplete or when
has selection
bias. When domain
is nonempty as
in Figure 2.2(a), then the composite estimator with imputed values becomes
The properties of the estimator in (4.2) depend on how well the imputation
model predicts the values of
in
Several imputation methods have been proposed.
With sample matching (Rivers, 2007),
for observation
in
is set equal to the value of
of the observation’s nearest neighbor (with
respect to the values of
in
Rivers (2007), considering the situation in which
is a convenience sample, took
in (4.2) and used the information in
for the sole purpose of finding the imputed
values
for
Yang, Kim and Hwang (2021) studied theoretical properties of
mass-imputed estimators that employ nearest neighbor methods.
Chen, Li and Wu (2020), building on the work of Lee (2006),
Lee and Valliant (2009), and Valliant and Dever (2011) on using propensity score weighting to estimate
population characteristics from a nonprobability sample, proposed a
“doubly-robust” estimator for the situation where
is measured in
both surveys but
is measured
only in nonprobability sample
Let
if population
unit
is in
and 0
otherwise. Under strong assumptions that (1)
and
are independent
given covariates
(2)
for all
population units
and (3)
and
are
conditionally independent given
they estimated
as a function
of
using
information in
and proposed
the estimator
where
is an imputation prediction for the unknown
values of
in
(developed using the information in
The estimator
is approximately unbiased for
if either the imputation model or the model
predicting
is correct. If the imputation model is
correct, then the first term of
is approximately unbiased for
and the second term has expected value 0. If
the model predicting
is correct, then
is approximately unbiased for
and
If neither model is correct, however,
may have large bias.
Kim and Tam
(2021) considered an extension of the
situation in Section 3.4 in which
is not measured
in
or is measured
differently than in
and proposed
substituting an imputed value
for
in the
estimators from
in (3.6),
obtaining the estimator in (4.2) with
they calibrated
this estimator to the known domain size
4.3 Imputation and the NSAF
The estimators in Section 4.2
impute a predicted value
for the unknown
value of
in
All have the
strong assumption that the imputation model developed on
applies to the
units in domain
.
As Lu (2014b)
noted when studying regression for dual-frame surveys, relationships between
and
may differ
across domains. Thus, an imputation model developed on a sample from an
incomplete frame, or on a sample with selection bias, may provide poor
predictions for
in other parts
of the population. Moreover, without data on
in the part of
the population that is imputed, it may not be possible to assess the quality of
the predictions.
A dual-frame survey was
taken for the NSAF because of concern that characteristics of interest might
differ for telephone and nontelephone households. Let
if child
is in a
household that is below 200 percent of the poverty threshold, and 0 otherwise.
Using the full sample from both frames (Urban Institute and Child Trends, 2007) an estimated 42.2 percent of children lived in households below 200
percent of the poverty threshold, with standard error 0.5 percent. The
estimated percentage from the RDD sample was 38.6 percent and the estimated
percentage from the area sample was 93.4 percent. Children in the nontelephone
households, sampled from the area frame, were much more likely to be living in
poverty.
Now suppose that the NSAF
had not measured poverty and income variables in the area sample, and
was imputed
using regression relationships developed in the RDD sample. In many surveys,
the only information available for developing an imputation model is
demographic variables. Fitting a logistic regression model to the RDD sample
that predicts
from race (with
categories white, black, and other), and assigning each child in the area
sample to the category with highest predicted probability, results in an
estimate of 30.5 percent of children in the area sample living in poverty
a lower value than in the RDD sample. Adding an
indicator variable for living in a single-parent household to the model, the
estimated percentage for the area sample goes up to 51.9 percent. Both of these
estimates, and estimates calculated using cell-mean imputations, are far below
the percentage of 93.4 percent from the real data.
The problem, of course,
is that the auxiliary information is not rich enough to provide a good
prediction of poverty in the area sample. The key feature of the data, and the
reason that Waksberg and his colleagues used a dual-frame survey, is that being
without a telephone is highly associated with poverty. That association cannot
be estimated from the RDD sample where all households have telephones. It might
be possible to develop an imputation model using information from other surveys
such as the Current Population Survey, where both telephone and non-telephone
households are sampled, but I could not find an imputation model predicting
from non-income
variables in the RDD sample that provided good predictions.
The nontelephone
households were a small part of the population for the NSAF, but the
differences between the multivariate relationships in the telephone and
nontelephone households were so great that the imputation only slightly reduced
bias. If poverty had not been measured for the nontelephone sample, however,
and the published statistics had relied only on the imputations, there would
have been no way to detect the bias.
4.4 Domain misclassification
One major challenge for
combining data using a multiple-frame approach is identifying the domain
membership (or multiplicity) of units in the data sources. This is challenging
even for surveys that are designed to make use of multiple frames.
The NSAF was designed as
a screening survey where telephone households were excluded from the area
sample. All households sampled from Frame 2, the RDD frame, were correctly
classified since they were contacted by telephone. The more difficult part was
obtaining the correct domain classification for households in the area-frame
sample. Initial prescreening questions asked whether the household had any
working telephones; those that answered no were transferred to the telephone
interviewer who conducted the detailed interview. The telephone interviewer
administered another brief screening interview and asked again about telephone service.
An additional 7 percent of households were excluded after answering the more
detailed questions about telephone ownership. Some had told the in-person
interviewer that they did not have a telephone because they thought the
interviewer wanted to borrow it. Others had misunderstood the question about
telephone ownership
one respondent, answering the prescreening
questions in the living room, thought they applied only to telephones in the
living room and did not mention the telephone in the bedroom (Cunningham,
Shapiro and Brick, 1999). Although the second screening interview may have
corrected for misclassification from respondents who mistakenly said they did
not have a telephone during the prescreening, there was no remedy for potential
misclassification from respondents who responded in prescreening that they had
a telephone when in fact they did not. Misclassification in this direction may
have been part of the reason the investigators had a smaller sample size of
nontelephone households than they had anticipated.
In dual-frame telephone surveys, the domain for Figure 2.2(b)
(cell only, landline only, or both) is usually determined by asking the
respondent about other available telephones and, sometimes, the relative amount
each type of telephone is used. Brick, Flores-Cervantes, Lee and Norman (2011)
found that their landline samples and cell samples both had smaller estimated
proportions of dual users than expected from statistics collected on telephone
ownership in the National Health Interview Survey. They conjectured that this
was because of persons who had access to both types of telephones but rarely
used one of them.
Domain membership may be unknown or difficult to
estimate when combining existing data sources. In some cases, as when administrative
lists are combined, it may be possible to link records, or the data files may
contain information that indicates whether the unit is in other frames. In
others, there may be little or no information available on domain membership.
How can one know whether a participant in an opt-in panel survey is also in a
frame of Medicare recipients if no questions about Medicare are asked in the
survey?
Lohr (2011) found that even a small amount of domain misclassification could create
large biases in dual-frame estimators; moreover, calibration to domain counts
that were based on misclassifications could worsen the bias. She proposed a
method for adjusting for bias due to domain misclassification, assuming that
misclassification probabilities
(observation classified in domain
observation
actually in domain
are known or
can be accurately estimated for different population subgroups. Lin, Liu and
Stokes (2019) studied a similar method using misclassification
probabilities
(observation actually in domain
observation
classified in domain
It may be possible to use multiple-frame methods when
domain membership is unknown if the probability that unit
is in domain
can be
estimated from auxiliary information
known for all
sampled units. Kim and Tam
(2021) proposed substituting an estimator
for the unknown domain membership for the situation in Section 3.4 where
is a census of
a subset of the population. They set
if the
predicted probability that unit
was in domain
exceeded 1/2,
and estimated the population total for domain
as
When domain membership is imputed, the mean squared
error depends on the accuracy of the domain imputations as well as design
features and nonresponse bias in
More research
is needed to establish statistical properties of estimators when domain
membership is estimated. It may also be desired to study alternative estimators
that use the predicted probabilities directly to estimate the total in domain
as
Dever (2018) used sample matching to evaluate the frame overlap for a probability
sample
taken from an
address-based sampling frame, and a nonprobability sample
recruited from
social media sites. She investigated the percentage of respondents in
who had no
close match in
Although this
procedure does not provide an unbiased estimate of the size of domain
a large
percentage of unmatched cases for large samples can indicate that
represents a
different population than
4.5 Indirect sampling and capture-recapture estimation
Sections 4.2 to 4.4
looked at extensions of multiple-frame estimators that relaxed Assumptions (A2),
(A4), and (A6). All of these, though, assumed that at least one of the frames,
or their union, had full coverage. Let’s now look at an example where Assumption
(A1) of full coverage is relaxed, and the multiple frames are used to estimate
the population size.
In indirect sampling, the
target population consists of units that are linked to units in the sampling
frame but are not necessarily in the frame (Lavallée, 2007)
units
in the target population are sampled indirectly through the links to the
sampling units in the frame. Lavallée
and Rivest (2012) extended the idea to
multiple-frame sampling. As an example, suppose the target population consists
of home care workers, who provide paid care for elderly, ill, or disabled
persons in their homes. Frame 1 might be a list of persons receiving
Medicare benefits, and Frame 2 might be a list of home health care aides
from employment or licensing agencies. Persons in the Frame-1 sample are asked
to identify workers who provide them with home care, who are then interviewed.
A sample of workers from Frame 2 is also interviewed. The home care
workers identified from the Frame-1 sample may have links to multiple persons
in Frame 1 and may also be in Frame 2. Similarly, persons in the
Frame-2 sample may also have links to units in Frame 1. An example of
linkage structure is shown in Figure 4.1.

Description for Figure 4.1
Figure illustrating an example of linkage structure. In the example, there is an indirect sampling with two frames (Frame 1 and Frame 2) linked to the target population. Units in the dark shaded area have links to both frames. A unit in the Target Population may have links to multiple units in Frame 1 and/or Frame 2. Furthermore, it is possible that the units of the Target Population can only be found in one of the two frames.
With indirect sampling,
the
frames can contain
different types of units (the situation with different types of units was also considered
by Hartley, 1974). We are not interested in overlap of the sampling frames
(shown as nonoverlapping in Figure 4.1 because they contain different
types of units) but in the overlap for the units in the target population.
Sampled units in the target population have multiple chances of selection if
they are linked to multiple units in one or both sampling frames.
Let
if unit
from Frame
is linked to
unit
in the target
population, and let
be the total
number of links between unit
in the target
population and Frame
(assumed to be
knowable from asking unit
Then an
estimator
can be found
for each frame using the links as
where
In the context of our
example, person
in
would say they
receive paid home care from provider
resulting in
Then the linked
home care provider would be asked about how many other persons they work for
who receive Medicare (assume they would know this or it could be determined
from other sources), giving the value
The quantity
sums the
weights of the units in
with links to
unit
adjusting for
the multiplicity of the links to that frame. If
then
where
if target population member
is linked to at least one unit in Frame
and 0 otherwise.
Multiple-frame methods
may then be used to estimate characteristics of the population of home care
providers, assuming that unit
linked from
provides
accurate information on (1) the number of links to members of Frame
needed for
multiplicity adjustments with Frame
and (2) whether
they are also linked to the other frame(s)
for
needed to
adjust for the multiplicity of linkage from different frames.
Lavallée and
Rivest (2012) noted that if the union of
the two frames has incomplete coverage
Assumption (A1) is violated
the samples from the two frames can be used to
estimate the size of the target population. Let
for
Then
is the number
of target population members that can be linked from Frame
. Each sample also provides an estimate of the number
of target population units that can be linked from both frames:
and
These can be
composited to obtain an estimator
of the number
of persons in the target population who can be captured from both frames.
The Lincoln-Petersen
capture-recapture estimator of population size can then be used. Under the
strong assumption that being captured by Frame 1 is independent of being
captured by Frame 2, the total number of home care providers can be
estimated by
In some cases
where the independence assumption is not met for the entire population, it may
be approximately met on subpopulations whose estimated numbers can be summed.
If there are more than two frames, loglinear models may be used to explore
associations among the frames (Lohr,
2022, Chapter 14); Zhang (2019)
presented a model for the situation in which frames may contain misclassified
units.
Alleva, Arbia, Falorsi,
Nardelli and Zuliani (2020) proposed using indirect multiple-frame sampling to
estimate the number of people infected by SARS-CoV-2 during the early stages of
the COVID-19 pandemic in 2020
information needed for estimating
transmissibility and infection parameters in epidemiologic models. In this
application, Frame 1 consists of persons with verified infections (perhaps
obtained from hospitals, quarantine centers, or clinics), and Frame 2
consists of other persons; the persons in
are
administered a test for SARS-CoV-2. The linked sample consists of persons who
had contact within the past 14 days with anyone in
or with a
member of
who tested
positive.