Statistical inference with non-probability survey samples
Section 7. Assumptions revisited

• Previous
• Next

Our discussions on estimation procedures for non-probability survey samples are under the assumptions A1-A4 and the focuses are on the validity and efficiency of estimators for the finite population mean under three inferential frameworks. The theoretical results on model-based prediction, inverse probability weighting and doubly robust estimation have been rigorously established under those assumptions. It seems that researchers are triumphant in dealing with the emerging area of non-probability data sources. However, as pointed out by the 2021 ASA President Robert Santos in his opinion article entitled “Using Our Superpowers to Contribute to the Public Good” (Amstat News, May 2021), “Our superpowers are only as good as their underlying assumptions, assumptions that are all too often embraced with aplomb, yet cannot be proven.” How to check assumptions A1-A4 in practical applications of the methods is a question that can never be fully answered, and yet there are steps to follow to boost the confidence in using the theoretical results. It is also important to understand the potential consequences when certain assumptions become seriously questionable.

7.1  Assumption A1

Assumption A1 states that ${\pi }_i^A=P\left(R_i\mathrm{<mo>=</mo>}1|x_i,y_i\right)=P\left(R_i=1|x_i\right).$ It is the most crucial assumption for the validity of the pseudo maximum likelihood estimator of Chen et al. (2020) and the nonparametric kernel smoothing estimator presented in Section 4.1.3 for the propensity scores, although all other assumptions are also involved. It is equivalent to the missing at random (MAR) assumption in the missing data literature. It is well understood that the MAR assumption cannot be tested using the sample data itself. The same statement holds for assumption A1 with non-probability survey samples.

In a nutshell, assumption A1 indicates that the auxiliary variables $x$ included in the non-probability sample fully characterize the participation behaviour or the sample inclusion mechanism for units in the population. Sufficient attention should be given at the study design stage before data collection, if such a stage exists, to investigate potential factors and features of units which might be related to participation and sample inclusion. For human populations, the factors and features may include demographical variables, social and economic indicators, and geographical variables.

Assumption A1 leads to the conclusion that the conditional distribution of $y$ given $x$ for units in the non-probability sample is the same as the conditional distribution of $y$ given $x$ for units in the target population. It implies that the auxiliary variables $x$ should include relevant predictors for the study variable $y.$ With the given datasets $S_A$ and $S_B,$ sensitivity analysis through comparisons of marginal distributions and conditional models can be helpful in building confidence on assumption A1. For variables which are available in both $S_A$ and $S_B,$ one can compare the empirical distribution functions (or moments) from $S_A$ to the survey weighted empirical distribution functions (or moments) from $S_B.$ Marked differences between the two indicate that $S_A$ is a non-probability sample with unequal propensity scores. One possible sensitivity analysis on assumption A1 is to select a variable $z$ which has certain similarities to $y,$ and a set of auxiliary variables $u$ with both $z$ and $u$ available from $S_A$ and $S_B.$ We fit a conditional model $z|u$ using data from $S_A$ and a survey weighted conditional model $z|u$ using data from $S_B.$ If $u$ includes all the key auxiliary variables for assumption A1, we should see the two versions of fitted models to be similar to each other. Drastic differences between the two fitted models are a strong sign that either the $z$ is itself an important auxiliary variable for assumption A1 or the assumption is questionable.

7.2  Assumption A2

A casual look at assumption A2 may have people believe that it should easily be satisfied in practice, since a similar assumption is widely used in missing data analysis and causal inference. It turns out that the assumption can be highly problematic, and for scenarios where the assumption fails to hold, the target population is different from the one assumed for the estimation methods. It is similar to the frame undercoverage and nonresponse problems which are discussed extensively in probability sampling.

Assumption A2 states that ${\pi }_i^A=P\left(R_i=1|x_i,y_i\right)>0$ for all $i.$ It is equivalent to stating that every unit in the target population has a non-zero probability to be included in the non-probability sample. If the sample was taken by a probability sampling method, this would be the scenario where the sampling frame is complete and there are no hardcore nonrespondents. For most non-probability samples, the concept of “sampling frame” is often irrelevant or simply a convenient list, and the selection and inclusion of units for the sample may not have a structured process. In her presentation at the 2021 CANSSI-NISS Workshop, Mary Thompson pointed out that “the statement that the sample inclusion indicator $R$  is a random variable is itself an assumption” for non-probability survey samples.

Let $U$ be the set of $N$ units for the target population. Let $U_0=\left\{i|i\in U\text{and}{\pi }_i^A>0\right\}.$ It is apparent that $U_0\subset U$ and $U_0\ne U$ when assumption A2 is violated. There are two typical scenarios in practice. The first can be termed as stochastic undercoverage, where the non-probability sample $S_A$ is selected from $U_0$ and $U_0$ itself can be viewed as a random sample from $U.$ For example, the contact list of an existing probability survey is used to approach units in the population for participation in the non-probability sample. In this case $U_0$ consists of units from the probability sample. Another example is a volunteer survey where the target population consists of adults in a specific city/region but the participants are recruited from visitors to major shopping centers in the region over certain period of time. The subpopulation $U_0$ includes visitors to the chosen locations over the sampling period and it is reasonable to assume that $U_0$ is a random sample from the target population. Let $D_i=1$ if $i\in U_0$ and $D_i=0$ otherwise, $i=\mathrm{1,}\mathrm{2,}\dots \mathrm{,}N.$ We have

$$P\left(R_i=1|x_i\mathrm{,}{y}_i,D_i=1\right)>0\text{and}P\left(R_i=1|x_i\mathrm{,}{y}_i,D_i=0\right)=0$$

for $i=\mathrm{1,}\mathrm{2,}\dots \mathrm{,}N.$ If the subpopulation $U_0$ is formed with an underlying stochastic mechanism such that $P\left(D_i=1|x_i\mathrm{,}{y}_i\right)\text{>}\text{0}$ for all $i\in U,$ we have

$${\pi }_i^A=P\left(R_i=1|x_i\mathrm{,}{y}_i\right)=P\left(R_i=1|x_i\mathrm{,}{y}_i,D_i=1\right)P\left(D_i=1|x_i\mathrm{,}{y}_i\right)>0$$

for $i=\mathrm{1,}\mathrm{2,}\dots \mathrm{,}N.$ In other words, the assumption A2 is valid under the scenario of stochastic undercoverage for non-probability samples.

The second scenario is termed as deterministic undercoverage where units with certain features will never be included in the non-probability sample. Suppose that participation in the non-probability survey requires internet access and a valid email address, and 20% of the population have neither access to the internet nor an email address, we have an example where the 20% of the population have zero propensity scores. There is no simple fix to the inferential procedures developed under A2. Yilin Chen’s PhD dissertation at University of Waterloo (Chen, 2020) contained one chapter dealing with some specific aspects of the scenario.

7.3  Assumption A3

Among all the assumptions, this one is less crucial to the validity of the proposed inferential procedures. Under assumption A3, the full likelihood function for the propensity scores is given in (4.1). For any parametric model on ${\pi }_i^A=\pi \left(x_i\mathrm{,}\alpha \right),$ the quasi log-likelihood function ${\text{ℓ}}^{*}\mathrm{<mo\; stretchy="false">(</mo>}\alpha \mathrm{<mo\; stretchy="false">)</mo>}$ given in (4.2) leads to the quasi score functions $U\mathrm{<mo\; stretchy="false">(</mo>}\alpha \mathrm{<mo\; stretchy="false">)</mo>}=\partial {\text{ℓ}}^{*}\mathrm{<mo\; stretchy="false">(</mo>}\alpha \mathrm{<mo\; stretchy="false">)</mo>}/\partial \alpha ,$ which remains unbiased even if assumption A3 is violated. There might be some efficiency loss without assumption A3 in estimating the model parameters $\alpha$ but the estimation methods are still valid under the other three assumptions.

7.4  Assumption A4

It is not difficult to find an existing probability sample from the same target population. It might be very hard, however, to have a probability survey sample which contains the desirable auxiliary variables. Existing probability surveys are designed with specific aims and scientific objectives, and the auxiliary variables included in the survey are not necessarily relevant to the analysis of a particular non-probability survey sample. The ultimate goal for satisfying assumption A4 is to identify and gain access to an existing probability survey sample with a rich collection of demographical variables, social and economic indicators, and geographical variables.

A rich-people’s problem (when one has too much money) for assumption A4 may also occur in practice when two or more existing probability survey samples are available. How to combine all of them for more efficient analysis of non-probability survey samples is a research topic that deserves further attention. Some practical guidances on choosing one reference probability sample from available alternatives include following considerations.

(i)   Check for availability of important auxiliary variables which are relevant to characterizing the participation behavor or having prediction power to the study variables in the non-probability sample;

(ii)  Give first preference to the one with a larger set of variables that are common to the non-probability sample;

(iii) Assign second preference to the probability sample with a larger sample size;

(iv) And lastly, use the probability sample for which the mode of data collection is the same as the one for the non-probability sample.

It was shown by Chen et al. (2020) that two reference probability survey samples with the same set of common auxiliary variables tend to produce very similar IPW estimators but the one with a larger sample size leads to better mass imputation estimators.

• Previous
• Article main page
• Next