Are probability surveys bound to disappear for the production of official statistics?
Section 3. Design-based approaches

• Previous
• Next

Design-based approaches yield design-consistent estimators of $\theta$ even when the non-probability source produces estimates with a significant selection bias. In this context, the purpose of using a non-probability sample is to reduce the variance of estimators of $\theta .$ The efficiency gains achieved can be used to justify a reduction of the probability sample size, thereby a reduction of the data collection costs and respondent burden. The methods that we consider in Sections 3.1 and 3.2 require collecting the values of the variable of interest $y$ in the probability sample, just like small area estimation methods described in Section 4.4. However, the efficiency gains are usually expected to be more modest than those obtained using small area estimation methods. In Section 3.1, we consider the scenario $y_k^{*}=y_k$ whereas in Section 3.2, we consider the scenario $y_k^{*}\ne y_k.$

3.1  Weighting by the inverse of the probability of inclusion in the combined sample

 The ideal case occurs when the non-probability sample is a census, i.e., $s_{\text{NP}}=U.$ In that case, the value of the parameter of interest $\theta ={\displaystyle {\sum }_{k\in U}{y}_k}$ can be directly calculated without worrying about bias or variance since $y_k^{*}=y_k$ is assumed in this section. In general, we expect under-coverage in the sense that $s_{\text{NP}}$ is smaller than the population $U.$ In a design-based approach, the potential under-coverage bias can be addressed by selecting a probability sample $s_P$ from $U$ and collecting the values of the variable $y$ for the sample units. Ideally, the probability sample is drawn from $U-s_{\text{NP}}$ but it is possible that the units in $s_{\text{NP}}$ cannot be linked to those of the sampling frame $U$ to establish the set $U-s_{\text{NP}}.$ In general, the larger the non-probability sample, the more it is possible to reduce the size of the probability sample without jeopardizing the desired precision of the estimates.

It seems desirable to estimate $\theta$ using all the data collected in the combined sample $s=s_P\cup s_{\text{NP}}.$ The inclusion indicator in $s$ can be defined as ${\tilde{I}}_k={\delta }_k+\left(1-{\delta }_k\right)I_k.$ To obtain a design-unbiased estimator of $\theta ,$ each unit $k\in s$ is weighted by ${\tilde{w}}_k={\tilde{\pi }}_k^{-1}$ where ${\tilde{\pi }}_k=E\left({\tilde{I}}_k|{\Omega }_P\right).$ Under assumptions 1 and 2, $E\left(I_k|{\Omega }_P\right)={\pi }_k$ and we obtain

$${\tilde{\pi }}_k=E\left({\tilde{I}}_k|{\Omega }_P\right)={\delta }_k+\left(1-{\delta }_k\right){\pi }_k.$$

The resulting estimator is written:

$$\widehat{\theta }={\displaystyle {\sum }_{k\in s}{\tilde{w}}_k{y}_k}={\displaystyle {\sum }_{k\in s_{\text{NP}}}{y}_k}+{\displaystyle {\sum }_{k\in s_P}\frac{1}{{\pi }_k}}\left(1-{\delta }_k\right)y_k.(3.1)$$

Note that estimator (3.1) requires the indicator ${\delta }_k$ to be available for all units in the sample $s_P.$ For the units $k\in s_P\cap s_{\text{NP}},$ we have two values: $y_k$ and $y_k^{*}.$ In principle, we should have $y_k^{*}=y_k,$ but it is possible that this relationship is not exactly satisfied. These units can be used to validate the assumption $y_k^{*}\approx y_k.$ If significant differences are observed, it may be preferable to not consider this approach and to rely on the methods in Section 3.2 that use data from the non-probability source as auxiliary data. If we trust the data quality of the non-probability source, it may be advisable not to collect the variable $y$ in the probability sample for the units also present in the non-probability sample in order to reduce the data collection costs and respondent burden.

We can view the problem as if we had two sampling frames: $U$ and $s_{\text{NP}}.$ A sample $s_P$ is drawn randomly from $U$ and a census is taken from $s_{\text{NP}}$. The probability of selection in the sample $s,$ $\mathrm{Pr}\left(k\in s|{\Omega }_P\right),$ can then be calculated for each unit $k\in U,$ and the estimator (3.1) is recovered by weighting each unit $k\in s$ by the inverse of that probability. This approach was proposed by Bankier (1986) to address the problem of multiple sampling frames. In the context of integrating a probability and non-probability sample, estimator (3.1) was proposed by Kim and Tam (2020).

The last sum of (3.1) is a design-unbiased estimator of ${\displaystyle {\sum }_{k\in U}\left(1-{\delta }_k\right)y_k}={\displaystyle {\sum }_{k\in U-s_{\text{NP}}}{y}_k}.$  If a vector of auxiliary variables, $x_k,$  is available for $k\in s_P$  as well as the total $T_x={\displaystyle {\sum }_{k\in U}{x}_k}$  then the weight $1/{\pi }_k$  in (3.1) can be replaced with a calibrated weight $w_k$  (e.g., Deville and Särndal, 1992; Haziza and Beaumont, 2017). The calibrated weights minimize a distance function between $w_k$  and $1/{\pi }_k,$   $k\in s_P,$  under the constraint of satisfying the calibration equation ${\displaystyle {\sum }_{k\in s_P}{w}_k{x}_k}=T_x.$  Ideally, the calibration is done only on the portion not covered by the non-probability sample, $U-s_{\text{NP}};$  i.e., the calibration vector $\left(1-{\delta }_k\right)x_k$  is used, and the calibration equation becomes: ${\displaystyle {\sum }_{k\in s_P}{w}_k\left(1-{\delta }_k\right)x_k}={\displaystyle {\sum }_{k\in U-s_{\text{NP}}}{x}_k}.$  This is not possible when ${\displaystyle {\sum }_{k\in U-s_{\text{NP}}}{x}_k}$  is unknown.

Remark: If assumption 2 is not appropriate, then $E\left(I_k|{\Omega }_P\right)\ne E\left(I_k|Z\right)={\pi }_k.$ To get around this problem, all the units for which the data were collected after selecting the sample $s_P$ can be removed from $s_{\text{NP}}$. Assumption 2 is then satisfied, but a lot of available data may be omitted. To take advantage of the full set $s_{\text{NP}},$ it is necessary to make a few assumptions and partially depart from the design-based approach. Assuming that $E\left(I_k|{\Omega }_P\right)=\mathrm{Pr}\left(I_k=1|{\delta }_k,Y,\Omega \right),$ we can use Bayes’ theorem to show that

$$\mathrm{Pr}\left(I_k=1|{\delta }_k=0,Y,\Omega \right)=\frac{1-\mathrm{Pr}\left({\delta }_k=1|I_k=1,Y,\Omega \right)}{1-\mathrm{Pr}\left({\delta }_k=1|Y,\Omega \right)}{\pi }_k,$$

for the units $k\in U-s_{\text{NP}}.$ Therefore, estimating $E\left(I_k|{\Omega }_P\right)$ requires postulating a model for ${\delta }_k.$ Under some assumptions, $\mathrm{Pr}\left({\delta }_k=1|I_k=1,Y,\Omega \right)$ can be estimated using the data from the probability sample and, for example, a logistic regression model. Estimating $\mathrm{Pr}\left({\delta }_k=1|Y,\Omega \right)$ can be done using the methods described in Section 4.3 that do not rely on the validity of assumption 2, such as the method by Chen, Li and Wu (2019). These methods require that the auxiliary variables used to model this probability be available for all units of the combined sample $s=s_P\cup s_{\text{NP}}.$ Unlike in Section 4.3, here we can take advantage of the availability of $y_k$ for all units of both samples, and we can use the variable of interest as an auxiliary variable. Then, $\theta$ is estimated by replacing ${\pi }_k$ in (3.1) with an estimate of $\mathrm{Pr}\left(I_k=1|{\delta }_k=0,Y,\Omega \right).$ Similar approaches were proposed by Beaumont, Bocci and Hidiroglou (2014) to take into account late respondents in Statistics Canada’s National Household Survey, i.e., households that responded to the initial questionnaire after the follow-up probability sample of non-respondents was drawn.

3.2 Calibration of the probability sample to the non-probability source

Data from non-probability sources, such as those provided by web panel respondents, can be fraught with measurement errors large enough to cast doubt on the assumption that $y_k^{*}\approx y_k.$ Therefore, such data cannot be used to directly replace the values of the variable $y.$ However, they can be used as auxiliary data to enhance the probability survey using the calibration technique. The non-probability source contains the values $y_k^{*}$ for $k\in s_{\text{NP}}$ and potentially the values of other variables. From all these variables, it is possible to form a vector of auxiliary variables $x_k^{*},$ available for $k\in s_{\text{NP}},$ that could include an intercept. Its total is denoted as $T_{x^{*}}={\displaystyle {\sum }_{k\in s_{\text{NP}}}{x}_k^{*}}={\displaystyle {\sum }_{k\in U}{\delta }_k{x}_k^{*}}.$ Another vector of auxiliary variables, $x_k,$ may also be available for $k\in s_P,$ as well as its total for the entire population $U,$ $T_x={\displaystyle {\sum }_{k\in U}{x}_k}.$ The calibrated weights $w_k,$ $k\in s_P,$ are obtained by minimizing a distance function between $w_k$ and $1/{\pi }_k,$ $k\in s_P,$ under the constraint of satisfying the calibration equation

$${\displaystyle {\sum }_{k\in s_P}{w}_k\left(\begin{array}{c}{x}_k\\ {\delta }_k{x}_k^{*}\end{array}\right)}=\left(\begin{array}{c}{T}_x\\ T_{x^{*}}\end{array}\right).$$

Note that this calibration can be done only if $x_k^{*}$ is available in the probability sample for all units $k\in s_P\cap s_{\text{NP}}.$  The estimator of $\theta$ is again written as $\widehat{\theta }={\displaystyle {\sum }_{k\in s_P}{w}_k{y}_k},$  where $w_k$  is the calibrated weight satisfying the above calibration equation. No model assumption is required for the validity of the approach, and the resulting estimates remain design-consistent regardless of the strength of the relationship between $y_k$  and the auxiliary variables $x_k$  and $x_k^{*}.$ A strong relationship will help reduce the design variance of $\widehat{\theta },$$\mathrm{var}\left(\widehat{\theta }|{\Omega }_P\right).$ Kim and Tam (2020) discuss the use of such calibration.

Canada’s Labour Force Survey (LFS) provides an example of a potential application for this calibration method. The unemployment rate, defined as the number of unemployed persons divided by the number of persons in the labour force, is a key parameter of interest that the LFS estimates. To improve the precision of the LFS estimates, a calibration variable indicating whether an individual is receiving employment insurance could be effective because there is definitely a connection between receiving employment insurance and being unemployed. The total of this calibration variable, the number of employment insurance beneficiaries, is needed for implementing this calibration and is available from an administrative source. However, applying this method would require adding a question to the LFS to identify LFS respondents who are receiving employment insurance. This information could also be obtained through a linkage between the LFS and the administrative source. It remains to be determined whether such a calibration variable could yield significant gains in the LFS.

• Previous
• Article main page
• Next