Comments on “Statistical inference with non-probability survey samples”
Section 3. When should one use nonprobability samples?

• Previous

Wu describes methods for combining information from probability and nonprobability samples after the decision has been made to do so. A first question, however, is whether the operation should be done at all. It may be desired to use a nonprobability sample because no high-quality probability sample measures $y,$ and it is thought that “any information is better than no information”. But is that true?

Suppose that, despite the careful model-fitting and model-checking, key statistics are still biased. Could reporting a flawed statistic be worse than reporting no statistic? Bad statistics, once published, can circulate for a long time ‒ even after more rigorous studies show that they are biased. In 1975, advice columnist Ann Landers asked her readers to respond to the question “If you had it to do over again, would you have children?” About 70% of the 10,000 persons who mailed a response said they would not have children in a do-over. This statistic is still cited, even though it is from a convenience sample, has been contradicted by numerous other studies, and is nearly 50 years old (Lohr, 2022). It is also unlikely that predictive modeling or IPW would have corrected the selection bias affecting Landers’ statistic, which occurred within all demographic groups.

With these issues in mind, here are some questions that could be asked when deciding whether to use estimates from a nonprobability sample and, if so, which statistical method to use for making inferences.

• How will the statistics be used? Estimates from the nonprobability sample might serve well for developing a marketing strategy or for an exploratory sociological study, but might not be deemed reliable enough for estimating unemployment or the number of persons requiring food assistance. Statistics from a nonprobability sample should be accompanied by evidence that the estimates are fit for use.
• What is the quality of the data in $S_A?$ Administrative records such as tax records have a different quality profile than a survey of volunteers recruited through an internet advertisement.

If the population for $S_A$ is well-defined (for example, tax filers), it may be better to report statistics for that population than to attempt to generalize to the population of $S_B.$ For tax records, many persons below preset income thresholds have ${\pi }_i^A=0$ and assumption (A2) is violated. Instead, a multiple-frame approach might be adopted, where a different data source is used to estimate ${\mu }_y$ for the parts of the population not in $S_A$ (Lohr, 2021).

Since all of the models rely on auxiliary information $x,$ it is important to have $S_A$ and $S_B$ measure the $x$ variables the same way. If income is used as an auxiliary variable, the same questions should be used to define income in both surveys, and income should be measured for the same unit (person or household).

Kennedy (2022) suggested that some respondents to opt-in online surveys may provide incorrect demographic information or bogus answers to questions; if that occurs, model predictions will be flawed. It may even be possible for outsiders desiring a specific outcome to manipulate the data in $S_A$  ‒ for example, an organization might arrange for the survey to be taken by a set of volunteers whose claimed demographic characteristics match those of the population but who give the “desired” answer for $y.$ Some proponents of nonprobability samples argue that low-response-rate probability samples also require weighting adjustments or imputation, but there is one important difference: the probability survey may have nonresponse, but the initial sample is selected randomly and cannot be manipulated by outside organizations.

If the data in $S_A$ are low-quality, is it worth spending the time to construct models? As Louis (2016) said, “Space-age procedures will not rescue stone-age data”.

• How detailed is the auxiliary information? If $S_A$ is large, and the auxiliary information is specific enough to be able to identify specific records, then linking records between $S_A$ and $S_B$ would be a better method for combining the data. Imputation or IPW would be used if the auxiliary information $x$ is rich enough to give good predictions of $y_i$ or ${\pi }_i^A,$  but not rich enough to permit accurate linkage. If there is little auxiliary information, however, then one would expect low variation in the propensity scores or imputed values, and the methods may give poor predictions ‒ with little information to diagnose potential problems.
• What analyses are desired? Wu discusses estimating the population mean, but the analyst may also want to look at relationships between $y$ and other variables, or estimate means or medians for subgroups. The choice of method depends in part on the variables that are available in $S_A$ and $S_B.$ If $S_A$ contains many response variables whose relationship is of interest, the IPW approach might be preferred.

If it is desired to explore relationships between $y$ and variables measured only in $S_B,$  imputation might be a better choice. Here, though, the analyst should be careful to acknowledge the imputation when presenting results ‒ if, say, linear regression is used for the imputation, the correlation calculated on $S_B$ is not between variable $u$ and variable $y,$ but between $u$ and $x^T\widehat{\beta }.$

• What are the implications for data equity? Jagadish, Stoyanovich and Howe (2021) defined “representation equity” as “increasing the visibility of underrepresented groups that have been historically disadvantaged or suppressed in the data record”.

  Nonprobability samples have the potential to improve data equity. They can increase the sample size and visibility of rare population subgroups ‒ a large data set $S_A$ might contain 10,000 members of the subgroup, while even a full-response probability survey with $n_B=$ 60,000 might contain only ten. Or the nonprobability sample may contain population members who are underrepresented in the probability survey because they are out of scope, undercovered in the sampling frame, or prone to nonresponse. In these situations, $S_A$ provides information about groups that are not as well represented in the probability survey.

On the other hand, historically disadvantaged groups may be underrepresented in all data sources, including $S_A.$ For example, a large nonprobability sample of electronic health records will be able to generate estimates for more population subgroups than a small probability sample about health. But persons without health insurance or access to medical care are underrepresented. In this situation, relying on $S_A$ to produce population estimates may reinforce inequities. If the estimates are used to distribute resources, then, as the program is implemented, more data will be collected in the areas getting those resources and will validate their needs, but no such follow-up will be done in areas that are inaccurately determined to receive no resources. The feedback loop will propagate the inequitable representation in data sources.

The MI and IPW methods have different data equity implications. Imputation assigns a predicted value of $y$ to each observation in $S_B,$ and the imputed $y$ value may differ from the $y$  value the respondent would have supplied if asked ‒ particularly if the respondent is in a subgroup that is unrepresented or misrepresented in $S_A.$ Will the model give accurate predictions for historically underrepresented subgroups? Did the respondents to $S_B$ give informed consent for $y$ to be imputed?

IPW assumes that the propensity scores can be estimated from auxiliary information. Is that information rich enough to give accurate weights? Are some subgroups unrepresented in $S_A?$ It may be useful to compare the results from the two methods, and from other data sources if available, for historically underrepresented population subgroups.

Wu’s critical review raises many important issues for persons interested in using nonprobability samples to make inferences about the population. I especially appreciate his assessment of the strong assumptions needed for the model-based methods, and applaud the emphasis on addressing these problems during the survey design stage.

References

Bondarenko, I., and Raghunathan, T. (2016). Graphical and numerical diagnostic tools to assess suitability of multiple imputations and imputation models. Statistics in Medicine, 35(17), 3007-3020.

Brick, J.M. (2015). Compositional model inference. In Proceedings of the Survey Research Methods Section, Alexandria, VA: American Statistical Association, 299-307.

Chipperfield, J., Chessman, J. and Lim, R. (2012). Combining household surveys using mass imputation to estimate population totals. Australian & New Zealand Journal of Statistics, 54, 223-238.

Dever, J.A. (2018). Combining probability and nonprobability samples to form efficient hybrid estimates: An evaluation of the common support assumption. In Proceedings of the 2018 Federal Committee on Statistical Methodology (FCSM) Research Conference. https://nces.ed.gov/FCSM/pdf/A4_Dever_2018FCSM.pdf.

Dutwin, D., and Buskirk, T.D. (2017). Apples to oranges or gala versus golden delicious? Comparing data quality of nonprobability internet samples to low response rate probability samples. Public Opinion Quarterly, 81(S1), 213-239.

Jagadish, H.V., Stoyanovich, J. and Howe, B. (2021). COVID$–$19 brings data equity challenges to the fore. Digital Government: Research and Practice, 2(2), 1-7.

Kennedy, C. (2022). Exploring the assumption that online opt-in respondents are answering in good faith. Paper presented at the 2022 Morris Hansen Lecture, March 1, 2022.

Kim, J.-K., Park, S., Chen, Y. and Wu, C. (2021). Combining non-probability and probability survey samples through mass imputation. Journal of the Royal Statistical Society, Series A, 184, 941-963.

Lohr, S.L. (2021). Multiple-frame surveys for a multiple-data-source world. Survey Methodology, 47, 2, 229-263. Paper available at https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2021002/article/00008-eng.pdf.

Lohr, S.L. (2022). Sampling: Design and Analysis, Third Edition. Boca Raton, FL: CRC Press.

Louis, T.A. (2016). Discussion of combining information from survey and non-survey data sources: Challenges and opportunities. 130^th CNSTAT Meeting Public Seminar; Washington, DC. https://sites.nationalacademies.org/cs/groups/dbassesite/documents/webpage/dbasse_172505.pdf.

Makela, S., Si, Y. and Gelman, A. (2014). Statistical graphics for survey weights. Revista Colombiana de Estadística, 37(2), 285-295.

Mercer, A., Lau, A. and Kennedy, C. (2018). For Weighting Online Opt-In Samples, What Matters Most? Washington, DC: Pew Research.

• Previous
• Article main page