Comments on “Statistical inference with non-probability survey samples”
Section 2. Model assumptions and diagnostics

• Previous
• Next

Probability sampling gained widespread use after the theory was developed in the 1930s and 1940s because it provided a mathematically justified solution to the problem of how to generalize from a sample to a population. Under minimal assumptions, a full-response probability sample produces approximately unbiased estimates of population quantities, accompanied by confidence intervals that have approximately correct coverage probabilities. It is the only method that is guaranteed to produce accurate confidence intervals without making assumptions about the unsampled members of the population. A probability sample is representative because of the procedure by which it is drawn.

All other methods require huge assumptions. The major assumptions for the predictive and IPW methods, given in Section 2.1 of Wu’s article, are: (A1) $y$ and the random variable indicating participation in $S_A$ are independent given $x,$ (A2) every unit in the population has ${\pi }_i^A>0,$ and (A3) the random variables indicating participation in $S_A$ are independent given $x.$ These assumptions imply that the auxiliary information $x$ is rich enough to develop inverse propensity weights that remove selection bias for $y,$ and that a model developed on $S_A$ to predict $y$ from $x$ will also apply to units not in $S_A.$

Statistical properties of the estimators are developed assuming that (A1)-(A3) are true and that the models adopted for weighting or imputation are correctly specified. Under those conditions, the estimated population mean is approximately unbiased with variance given by the appropriate theorem. But, as Wu points out, that variance estimate is conditional on the assumptions being satisfied; if the assumptions are not met, it will severely underestimate the true mean squared error and give a misleading impression of the estimate’s trustworthiness. If $n_A$ and $n_B$ are large but (A1) is violated, the bias might be 10 percentage points but the reported standard error of an MI or IPW estimate will be close to zero. In practice, many nonprobability samples will violate the assumptions: Mercer, Lau and Kennedy (2018) found, when weighting online opt-in samples with rich auxiliary information, that “even the most effective adjustment strategy was only able to remove about 30% of the original bias”.

The assumptions cannot be fully tested because they involve missing data ‒ population members missing from $S_A$ and $y$ values missing from $S_B.$ But, as with nonresponse adjustments in probability samples (Lohr, 2022, Chapter 8), one can perform model checks and diagnostics using available information, with the recognition that these might not catch all model deficiencies.

Compare statistics from the nonprobability sample with those from other data sources

Wu suggests comparing empirical distribution functions of variables in $x$ from $S_A$ with the survey-weighted empirical distribution functions from $S_B.$ Differences may indicate that observations in $S_A$ have unequal propensity scores or that the $x$ variables are measured differently in $S_A$ than in $S_B$ (see Section 3). One can also compare empirical distributions from $S_A$ with those from another probability survey $S_C.$

If IPW is used, one can also compare propensity-score-weighted empirical distribution functions from $S_A$ with those from $S_B$ and other surveys. This should be done only for variables not used in the weighting, since the propensity score weights have already adjusted for imbalances in weighting variables. Dutwin and Buskirk (2017), for example, constructed propensity weights for a nonprobability sample through raking on marginal totals and then compared the cross-tabulations of those raking variables.

Wu also suggests treating a variable $z$ that is measured in both $S_A$ and $S_B$ as a response variable, and comparing conditional models for $z|u$ fitted on $S_A$ and $S_B,$ where $u$ is a subset of $x$ (excluding $z).$ Differences in the two models can indicate that $z$ is needed as an auxiliary variable, and may also raise questions of how well the set of measured auxiliary variables satisfy assumption (A1).

In an example from Kim, Park, Chen and Wu (2021), the estimated percentage of persons who volunteer was 24.8% from the Current Population Survey (the gold-standard estimate), but the MI and IPW estimates from $S_A$ were both close to 50% with reported standard error less than one percentage point. The standard error, computed under the model assumptions, did not account for the selection bias of $S_A$  with respect to volunteerism ‒ a bias that could not be removed using demographics, home ownership, and medical insurance as model covariates.

Compare results from the IPW and MI approaches

An alternative to using the doubly robust estimator for analysis is to use each model to identify potential deficiencies of the other. Possible investigations include comparing the empirical distribution of $y$ from $S_A$ (using the inverse propensity weights) with the empirical distribution of $y^{\text{*}}$ from $S_B$ (using the imputed values and the survey weights). Similarly, as suggested by Chipperfield, Chessman and Lim (2012), one can compare estimated domain means from $S_A$ and $S_B$ for a set of domains $d=1,\dots ,D.$ One might also compare imputations for $y$ fit to the unweighted data set $S_A$ with imputations developed on $S_A$ with inverse propensity weights.

Simulation studies are valuable for checking the small-sample behavior when the assumptions are met, but are of limited value for exploring sensitivity to model assumptions. These explore model deviations devised by the investigators, but real surveys can diverge from the model in many unanticipated ways.

Perform model diagnostics

Of course, for either the IPW or model-predictive approach, analysts should employ standard regression diagnostics such as examining residuals and influential observations to examine model fit and sensitivity to outliers, and document the checks that were done.

For the IPW approach, it is also desirable to examine characteristics of the final weights. The coefficient of variation of the weights provides a rough measure of the amount of adjustments that were needed to make sample $S_A$ “representative”. A low coefficient of variation, however, does not necessarily mean the sample is representative; this may merely reflect inadequacy of the available auxiliary information for developing weights. For example, suppose a quota sample from an opt-in internet panel is drawn to match the population with respect to the auxiliary variables. The inverse propensity weights will have little variation because the $x$ variables were used to form the quota classes, but the sample may still produce biased estimates of $y$ variables such as internet usage or volunteering.

The graphical methods proposed by Makela, Si and Gelman (2014) for assessing weight adjustments in surveys can be used with IPW as well. Brick (2015) suggested looking at the magnitude of the IPW adjustments in the weighting cells. One can also examine the distribution of the weights within domains of interest.

The inverse propensity weights can also provide information about assumption (A2). A domain that has high weights relative to other domains may have undercoverage in $S_A.$ Dever (2018) proposed investigating assumption (A2) by identifying individuals in $S_B$ who have no close match in $S_A.$

Bondarenko and Raghunathan (2016) reviewed and proposed graphical and numerical diagnostic tools for assessing and improving imputation models. None of these diagnostics, however, will test the assumption that the regression model fit on $S_A$ applies to units not in $S_A.$ Just as ${\overline{y}}_A$ may be a biased estimator of ${\mu }_y,$ regression coefficients derived from $S_A$ may also be biased, and the model constructed from $S_A$ to predict $y$ from $x$ might not apply to other parts of the population.

Take a small probability sample to investigate assumptions

The preceding steps can identify some model deficiencies, but cannot fully test assumptions (A1) and (A2). But one can test the imputation model by obtaining data about $y$ on a probability subsample of $S_B.$ Similarly, one could take a probability sample from population members not in $S_A$ to check inferences from the IPW approach, or observe $y$ on a subsample of units in $S_B$ that are similar to those with high weights in $S_A,$ or that have no close match in $S_A.$

• Previous
• Article main page
• Next