Comments on “Statistical inference with non-probability survey samples” – Miniaturizing data defect correlation: A versatile strategy for handling non-probability samples
Section 6. Counterbalancing sub-sampling

• Previous
• Next

6.1  The devastating impact of data defect on effective sample size

A key finding, which has surprised many, from studying the data quality issue is how small the size of our “big data” is when we take into account the data defect. To prove this mathematically, we can equate the mean-squared error (MSE) of ${\overline{G}}_W$ in (2.1), with the MSE of a simple random sampling estimator of size $n_{\text{eff}}.$ This yields (see Meng (2018) for derivation):

$$n_{\text{eff}}\approx \frac{f_W}{1-f_W}\frac{1}{\text{E}\mathrm{<mo\; stretchy="false">[</mo>}{\rho }_{\tilde{R},G}^2\mathrm{<mo\; stretchy="false">]</mo>}}\approx \frac{f_W}{1-f_W}\frac{1}{{\rho }_{\tilde{R},G}^2}\mathrm{,}(6.1)$$

where $f_W=n_W/N$ and the expectation $\text{E}$ is with respect to the conditional distribution of $\tilde{R}$ given $n_W.$ It is worthwhile to note that this (conditional) distribution can involve all three types of probability discussed in Section 1.2 because the variations in $\tilde{R}$ can come from multiple sources. For example, in typical opinion surveys, there will be (1) design probability in the sampling indicator, (2) divine probability in formulating the non-response mechanism, and (3) device probability for estimating the mechanism and the weights.

Expression (6.1) is the weighted version/extension of the expression given in Meng (2018) with equal weights, which reveals the devastating impact of a seemingly tiny ddc. Suppose our sample is 1% of the population, and it suffers from a half-percent ddc. Applying (6.1) (with equal weights) with $f_W=$ 0.01 and ${\rho }_{\tilde{R},G}=$ 0.005 yields $n_{\text{eff}}\approx$ 404 regardless of the sample size $n_R.$ In the case of the 2020 US presidential election, 1% of the voting population is about 1.55 million people, and hence the loss of sample size due to a half percent ddc is about 1 - (404 / 1,550,000) > 99.97%. Such seemingly impossible losses have been reported in both election studies (Meng, 2018) and COVID vaccination studies (Bradley et al., 2021). A most devastating consequence of such losses is the “big data paradox”: the larger the (apparent) data size, the surer we fool ourselves because our false confidence (in both technical and literal sense) goes up with the erroneous data size, while the actual coverage probability of the incorrectly constructed confidence intervals become vanishingly small (Meng, 2018; Msaouel, 2022).

A positive implication from this revelation, however, is that we can trade much data quantity for data quality, and still end up having statistically more accurate estimates. Of course, in order to reduce the bias, we will need some information about it. If we have reliable information on the value of ddc, we can directly adjust for the bias in estimating the population average corresponding to the ddc, for example by a Bayesian approach, similar to that taken by Isakov and Kuriwaki (2020) in their scenario analysis. Furthermore, if we have sufficient information to construct reliable weights, we can use the weights to adjust for selection biases as commonly done. Nevertheless, even in such cases, it may still be useful to create a representative miniature of the population out of a biased sample for general purposes, which for example can eliminate many practitioners’ anxiety and potential mistakes for not knowing how to properly use the weights. Indeed, few really know how to deal with weights, because “Survey weighting is a mess” (Gelman, 2007).

However, creating a representative miniature out of a biased sample in general is a challenging task, especially because ddc can (and will) vary with the variable of interest. Nevertheless, just as weighting is popular tool despite it being far from perfect, let us explore representative miniaturization and see how far we can push the idea. The following example therefore is purely for brainstorming purposes, by looking into a common but challenging scenario, where we have reasonable information or understanding on the direction of the bias, that is, the sign of the ddc, but rather vague information about its magnitude. A good example is non-representativeness of election polls because voters tend to not want to disclose their preferences when they plan to vote for a socially unpopular candidate; we therefore know the direction of the bias, but not much about its degree other than some rough guesses (e.g., a range of 10 percentage points).

6.2  Creating a less biased sub-sample

The basic idea is to use such partial information about the selection bias to design a biased sub-sampling scheme to counterbalance the bias in the original sample, such that the resulting sub-samples have a high likelihood to be less biased than the original sample from our target population. That is, we create a sub-sampling indicator $S_I,$ such that with high likelihood, the correlation between $S_I{R}_I$ and $G_I$ is reduced, compared to the original ${\rho }_{R,G},$ to such a degree that it will compensate for the loss of sample size and hence reduce the MSE of our estimator (e.g., the sample average). We say with high likelihood, in its non-technical meaning, because without full information on the response/recording mechanism, we can never guarantee such a counterbalance sub-sampling (CBS) would always do better. However, with judicious execution, we can reduce the likelihood of making serious mistakes.

To illustrate, consider the case where $y$ is binary. Let $\Delta =r_1-r_0,$ where $r_y$ is the propensity of responding/reporting for individuals whose responses will take value $y:$ $r_y={\mathrm{Pr}}_I\mathrm{<mo\; stretchy="false">(</mo>}{R}_I=1|y_I=y\mathrm{<mo\; stretchy="false">)</mo>.}$ If the sample is representative, then like ${\rho }_{R,G},$ $\Delta$ is miniaturized, meaning that it is on the order of $N^{-1/2}.$ This is most clearly seen via the easily verifiable identity (see (4.1) of Meng, 2018)

$$\Delta =\frac{{\text{Cov}}_I\mathrm{<mo\; stretchy="false">(</mo>}{y}_I\mathrm{,}{R}_I\mathrm{<mo\; stretchy="false">)</mo>}}{p\mathrm{<mo\; stretchy="false">(</mo>1}-p\mathrm{<mo\; stretchy="false">)</mo>}}={\rho }_{R,y}\sqrt{\frac{f_R\mathrm{<mo\; stretchy="false">(</mo>1}-f_R\mathrm{<mo\; stretchy="false">)</mo>}}{p\mathrm{<mo\; stretchy="false">(</mo>1}-p\mathrm{<mo\; stretchy="false">)</mo>}}}\mathrm{,}(6.2)$$

where $p={\mathrm{Pr}}_I\mathrm{<mo\; stretchy="false">(</mo>}{y}_I=\mathrm{1<mo\; stretchy="false">)</mo>}$ and $f_R={\mathrm{Pr}}_I\mathrm{<mo\; stretchy="false">(</mo>}{R}_I=\mathrm{1<mo\; stretchy="false">)</mo>},$ which is the original sampling rate. A key ingredient of CBS is to determine $s_y\mathrm{<mo>=</mo>}{P}_I\mathrm{<mo\; stretchy="false">(</mo>}{S}_I=1|y_I=y\mathrm{,}{R}_I=\mathrm{1<mo\; stretchy="false">)</mo>}$ for $y=\text{0,}\text{1,}$ that is, the sub-sampling probabilities of individuals who reported $y=1$ and $y=0,$ respectively.

To determine the beneficial choices, let $f_S={\mathrm{Pr}}_I\mathrm{<mo\; stretchy="false">(</mo>}{S}_I=1|R_I=1\mathrm{<mo\; stretchy="false">)</mo>}$ be the sub-sampling rate, and ${\Delta }_S=s_1{r}_1-s_0{r}_0.$ Then by applying (2.2) (with equal weights) and (6.2) to both the sample average and the sub-sample average, we see that the sub-sample average has smaller (actual) error in magnitude if and only if

$${\left(\frac{{\Delta }_S}{f_S{f}_R}\right)}^2<{\left(\frac{\Delta }{f_R}\right)}^2\iff f_S^2>{\left(\frac{{\Delta }_S}{\Delta }\right)}^2\mathrm{.}(6.3)$$

Writing $r=r_1/r_0$ and $s=s_1/s_0,$ the right-hand side of (6.3) becomes

$${\left[s{p}^{*}+\mathrm{<mo\; stretchy="false">(</mo>1}-p^{*}\mathrm{<mo\; stretchy="false">)</mo>}\right]}^2>{\left(\frac{rs-1}{r-1}\right)}^2\mathrm{,}(6.4)$$

where $p^{*}={\mathrm{Pr}}_I\mathrm{<mo\; stretchy="false">(</mo>}{y}_I=1|R_I=\mathrm{1<mo\; stretchy="false">)</mo>}$ is observed in the original sample, which should remind us that $p^{*}$ may be rather different from the $p$ we seek, because of the biased $R$ -mechanism.

An immediate choice to satisfy (6.4) is to set $s=r^{-1},$ which of course typically is unrealistic because if we know the value of $r,$ then the problem would be a lot simpler. To explore how much leeway we have in deviating from this ideal choice, let $\delta =r-1,$ we can then show that (6.4) is equivalent to

$$\mathrm{<mo\; stretchy="false">(</mo>}s-\mathrm{1<mo\; stretchy="false">)</mo>}\left\{\mathrm{<mo\; stretchy="false">[</mo>1}+\mathrm{<mo\; stretchy="false">(</mo>1}+p^{*}\mathrm{<mo\; stretchy="false">)</mo>}\delta \mathrm{<mo\; stretchy="false">]</mo>}\mathrm{<mo\; stretchy="false">(</mo>}s-\mathrm{1<mo\; stretchy="false">)</mo>}+2\delta \right\}<0.(6.5)$$

This tells precisely the permissible choices of $s$ without over-correcting (in the magnitude of the resulting bias): 

(i)  When $r>1,$ i.e., $\delta >0,$ we can take any $s$ such that

$$\frac{{\mathrm{<mo\; stretchy="false">[</mo>1}-\mathrm{<mo\; stretchy="false">(</mo>1}-p^{*}\mathrm{<mo\; stretchy="false">)</mo>}\delta \mathrm{<mo\; stretchy="false">]</mo>}}_{+}}{1+\mathrm{<mo\; stretchy="false">(</mo>1}+p^{*}\mathrm{<mo\; stretchy="false">)</mo>}\delta }\le s<\mathrm{1<mo>;</mo>}(6.6)$$

(ii)  When $r<1,$ i.e., $\delta <0,$ we can take any $s$ such that

$$1<s\le \frac{1-\mathrm{<mo\; stretchy="false">(</mo>1}-p^{*}\mathrm{<mo\; stretchy="false">)</mo>}\delta }{{\mathrm{<mtext>\hspace{0.17em}</mtext><mo\; stretchy="false">[</mo>1}+\mathrm{<mo\; stretchy="false">(</mo>1}+p^{*}\mathrm{<mo\; stretchy="false">)</mo>}\delta \mathrm{<mo\; stretchy="false">]</mo>}}_{+}}\mathrm{.}(6.7)$$

This pair of results confirms a number of our intuitions, but also offers some qualifications that are not so obvious. Since we sub-sample to compensate for the bias in the original sample, $s$ and $r$ must stay on the opposite side of 1, i.e., $\mathrm{<mo\; stretchy="false">(</mo>}s-\mathrm{1<mo\; stretchy="false">)</mo>}\mathrm{<mo\; stretchy="false">(</mo>}r-\mathrm{1<mo\; stretchy="false">)</mo>}=\mathrm{<mo\; stretchy="false">(</mo>}s-\mathrm{1<mo\; stretchy="false">)</mo>}\delta <0,$ as seen in (6.6)-(6.7). To prevent over corrections, some limits are needed, but it is also possible that the initial bias is so bad that no sub-sampling scheme can make things worse, which is reflected by the positivizing function ${[x]}_{+}$  in the two expressions above. However, the expressions for the limits as well as for the thresholds to activate the positivizing functions are not so obvious. Nor is it obvious that these expressions depend on the unknown $p$ indirectly via the observed $p^{*},$ and hence only prior knowledge of $r$ is required for implementing or assessing CBS.

This observation suggests that it is possible to implement a beneficial CBS when we can borrow information from other surveys (or studies) where the response/recording behaviors are of similar nature. For example, we may learn that a previous similar survey had $r=$ 1.5 (e.g., those with $y=1$ had 6% of chance to be recorded, and those with $y=0$ had only 4% chance). Taking into account the uncertainty in the similarity between the two surveys, we might feel comfortable to place (1.2, 1.8) as the plausible range for $r$ in the current study. Suppose we observe $p^{*}=$ 0.6, this means that the maximum ‒ over the range $r\in$ (1.2, 1.8) ‒ of the lower bound on the permissible $s$ as given in (6.6) is

Therefore, as long as we choose $s\in \mathrm{<mo\; stretchy="false">[</mo>0.7,}\mathrm{1<mo\; stretchy="false">)</mo>},$ we are unlikely to over-correct. The price we pay for this robustness is that the resulting sub-sample is not as good quality as it can be, for example, when the underlying $r$ for the current study is indeed 1.5 (in expectation). Choosing any $s\in \mathrm{<mo\; stretchy="false">[</mo>0.7,}\mathrm{1<mo\; stretchy="false">)</mo>}$ will not provide the full correction as provided by $s=1/r=$ 0.67, that is, the sub-sample average will still have a positive bias but with a smaller MSE compared to the original sample average. Of course both the feasibility and effectiveness of such CBS need to be carefully investigated before it can be recommended for general consumption, especially going beyond binary $y.$ The literature on inverse sampling (Hinkins, Oh and Scheuren, 1997; Rao, Scott and Benhin, 2003) is of great relevance for such investigations, because it also aims to produce simple random samples via subsampling, albeit with a different motivation (to turn complex surveys into simple ones for ease of analysis).

• Previous
• Article main page
• Next