Comments on “Statistical inference with non-probability survey samples”
Section 4. Unverifiable assumptions: Recent developments in sensitivity analysis

• Previous
• Next

Wu provides four key assumptions required to correct for selection bias in non-probability surveys using data from probability surveys: they can be roughly summarized as “selection at random” or SAR (covariates in the non-probability sample explain the probability of selection in the non-probability sample); “positivity” (all elements in the population have a non-zero probability of selection into the non-probability sample); “independence” (elements are selected independently into the non-probability sample); and “common covariates” (there exists a probability survey with covariates whose subset matched the covariates required for the MAR assumption to hold). It might be worth noting that the first two assumptions basically require the non-probability survey to be a probability survey “in disguise” $–$ that is, there really are non-zero probabilities of selection into the non-probability survey for all elements in the population, but we as analysts just do not know what they are.

In practice neither of these assumptions probably hold precisely. Some recent work has focused on the failure of the first, the SAR assumption. Some existing measures borrowed from the non-response literature have been repurposed here: for example, the R-indicator measure (Schouten, Cobben and Bethlehem, 2009), which in this context is the measure of the variability in the probabilities of selection in the non-probability sample:

$$\widehat{R}\mathrm{<mo>=</mo>}1-2\sqrt{\frac{1}{n_a-1}{\displaystyle \sum _{i\mathrm{<mo>=</mo>}1}^{n_A}}{\left({\widehat{\pi }}_i^A-{\displaystyle \sum _{j\mathrm{<mo>=</mo>1}}^{n_A}}{\widehat{\pi }}_j^A/n_a\right)}^2}$$

$\widehat{R}$ can range between 0 and 1, where 1 is achieved when probabilities of selection are constant $–$ suggesting something akin to a simple random sample, with less chance for selection bias $–$ and 0 $–$ suggesting all elements are either included with probability 1 or 0, maximizing the risk of selection bias.

Of course, in the absence of the outcome $Y$ in the probability sample, there is no way to directly assess selection bias. Hence recent work has extended Andridge and Little (2011), which develops a sensitivity analysis using a pattern-mixture model, wherein selection into non-probability sample is allowed to depend entirely on a scalar reduction to the covariates $X,$ entirely on the outcome $Y,$ or some convex combination thereof. Little, West, Boonstra and Hu (2020), Andridge, West, Little, Boonstra and Alvarado-Leiton (2019), and West, Little, Andridge, Boonstra, Ware, Pandit and Alvarado-Leiton (2021) consider sensitivity to this assumption in the estimation of the mean of a normally distributed variable, the mean of a binary outcome, and the regression parameters in a linear regression model, respectively, in non-probability samples. By varying the convex mixing parameter $\varphi ,$ sensitivity to the SAR assumption can be assessed. Boonstra, Little, West, Andridge and Alvarado-Leiton (2021) finds that these “standard measures of bias” (SMB) compare favorably with alternatives such as $\widehat{R}$ in a simulation study. An important point to note is that the methods that extend Andridge and Little (2011) do not depend on assumption of common covariates in a probability sample. This suggests that methods that use information available in the probability sample to assess SAR are an open area for development.

The second assumption $–$ positivity $–$ is also unlikely to exist precisely in many practical settings. My own work in this area has focused on naturalistic driving studies, which typically involve convenience samples in a limited geographical area: for example, the Second Strategic Highways Research Program (SHRP2) recruited drivers in six specific geographic regions across the United States (Transportation Research Board (TRB) of the National Academy of Sciences, 2013). This corresponds to the second scenario given by Wu in Section 7.2, where only a subpopulation has any chance of being selected into the non-probability sample, which as he notes has “no simple fix”. Following his notation of $D$ providing an indicator of membership in the subpopulation, it would seem that if $D_i\perp X_i\mathrm{,}{Y}_i|{\pi }_i^A$ $–$ that is, if the distribution of $X\mathrm{,}Y$ is the same for $D\mathrm{<mo>=</mo>}0$ and $D\mathrm{<mo>=</mo>}1$ after weighting for ${\pi }_i^A$ within the $D\mathrm{<mo>=</mo>}1$ stratum $–$ then lack of positivity would have no impact on inference. This is likely a tall order in the most general settings but might be reasonably well approximated if the analysis of interest involves a subset of $X\mathrm{,}Y$ that is only weakly associated with $D$ even before adjustment.

Finally, regarding the fourth assumption $–$ existence of a probability sample with available $X$ $–$ I very much second Wu’s observation that methods to take advantage of multiple probability surveys need more development. However, it remains more likely that a researcher will struggle to find a single probability sample with sufficient covariates than struggle with a surfeit of options (Wu’s “rich person’s problem”). To this end I will conclude with a call to action by the survey community.

• Previous
• Article main page
• Next