Are probability surveys bound to disappear for the production of official statistics?
Section 5. Conclusion
In this paper, we presented several methods that use data from a non-probability source while preserving a statistical framework that allows for valid inferences. This, in our view, is essential for national statistical agencies because, without this framework, the usual measures of the quality of the estimates, such as variance or mean square error estimates, disappear and it becomes difficult to draw accurate conclusions. Using data from a non-probability source is not without risk. For model-based approaches, it seems unavoidable to plan enough time and resources for modelling. The literature on classical statistics is replete with tools for validating model assumptions. Although this topic was not adequately covered in the previous sections, careful validation of the assumptions is still a critical step in the success of these approaches (Chambers, 2014) and is one of the recommendations made by Baker et al. (2013).
Estimating the variance or mean square error of the estimators described in the previous sections is also an important topic that we omitted. Yet, this problem does not pose any particular difficulties, in general, and a number of methods exist for variance or mean square error estimation. For design-based approaches, the topic has been extensively covered in the literature (see, for example, Wolter, 2007). This is also true for small area estimation methods (see Rao and Molina, 2015) and for the calibration approach (see Valliant, Dorfman and Royall, 2000). Nevertheless, it might be useful that research be undertaken to adequately address this issue in some specific cases, such as weighting by inverse propensity score or statistical matching by nearest donor.
We assumed that the non-probability source was a subset of the population of interest and that it may be subject to measurement errors. However, there are other potential flaws with non-probability sources. For example, they may contain duplicates or units outside the population. This could make some of the methods discussed in this article unusable, especially the design-based methods. Therefore, it might be useful to tackle these problems in the future.
We mainly limited ourselves to describing several methods that use data from a non-probability sample, whether or not combined with data from a probability survey, once all the data have been collected and processed. There are a number of other methods that use data from non-probability sources during the various stages of a probability survey. For example, one or more non-probability sources can be used to create a sampling frame or improve its coverage. These sources can also be used in a multi-frame sampling context, to replace data collection for certain variables, or to impute the missing values in a probability survey. These topics were not covered in this article, but are reviewed in Lohr and Raghunathan (2017).
The literature on integrating data of a probability and non-probability sample is quite recent. However, there are a number of methods that combine data from two probability surveys (e.g., Hidiroglou, 2001; Merkouris, 2004; Ybarra and Lohr, 2008; Merkouris, 2010; and Kim and Rao, 2012). Such methods may be used to first combine two probability surveys before integrating them with a non-probability source using one of the methods in Section 4. For example, if the total is unknown, it may be possible to estimate it using more than one probability survey and then use this estimated total in the calibration approach. It still needs to be assessed whether such a strategy would yield significant efficiency gains.
Are probability surveys bound to disappear for the production of official statistics? The question is relevant in the current context of surveys conducted by national statistical agencies where high data collection costs and increasingly lower response rates are observed. In our opinion, the time has not yet come because the alternatives are not reliable and general enough to eliminate the use of probability surveys without severely sacrificing the quality of the estimates. In Section 4, we mentioned that calibration and weighting by inverse propensity score could eliminate the use of a probability survey, provided that a vector of population totals is available from a census or a comprehensive administrative source. In general, these known totals will not be numerous and effective enough to sufficiently reduce the selection bias of a non-probability sample. To get around this problem, the suggestion has been made in the literature to complement with other totals estimated using a good-quality probability survey. It seems to us that this is the way to significantly reduce bias and to really take advantage of calibration and weighting by inverse propensity score methods presented in Section 4. Of course, some probability surveys with very low response rates and/or data of questionable quality could occasionally be eliminated in favour of data from non-probability sources. In our view, most surveys conducted by Statistics Canada do not fall into this category. Although they are not perfect, they continue to provide reliable information to meet users’ needs and to make informed decisions. The complete elimination of probability surveys seems highly unlikely in the short or medium term. However, it can be expected that their use will be reduced in the future in order to control costs and respondent burden.
References
Baker, R., Brick, J.M., Bates, N.A., Battaglia, M., Couper, M.P., Dever, J.A., Gile, K. and Tourangeau, R. (2013). Summary report of the AAPOR task force on non-probability sampling. Journal of Survey Statistics and Methodology, 1, 90-143.
Bankier, M.D. (1986). Estimators based on several stratified samples with applications to multiple frame surveys. Journal of the American Statistical Association, 81, 1074-1079.
Beaumont, J.-F., and Bissonnette, J. (2011). Variance estimation under composite imputation: The methodology behind SEVANI. Survey Methodology, 37, 2, 171-179. Paper available at https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2011002/article/11605-eng.pdf.
Beaumont, J.-F., and Bocci, C. (2009). Variance estimation when donor imputation is used to fill in missing values. Canadian Journal of Statistics, 37, 400-416.
Beaumont, J.-F., and Bocci, C. (2016). Small Area Estimation in the Labour Force Survey. Paper presented at the Advisory Committee on Statistical Methods, May 2016, Statistics Canada.
Beaumont, J.-F., Bocci, C. and Hidiroglou, M. (2014). On weighting late respondents when a follow-up subsample of nonrespondents is taken. Paper presented at the Advisory Committee on Statistical Methods, May 2014, Statistics Canada.
Beaumont, J.-F., Haziza, D. and Bocci, C. (2014). An adaptive data collection procedure for call prioritization. Journal of Official Statistics, 30, 607-621.
Berg, E., Kim, J.-K. and Skinner, C. (2016). Imputation under informative sampling. Journal of Survey Statistics and Methodology, 4, 436-462.
Bethlehem, J. (2009). The rise of survey sampling. Discussion paper (09015), Statistics Netherlands, The Hague.
Bethlehem, J. (2016). Solving the nonresponse problem with sample matching. Social Science Computer Review, 34, 59-77.
Brick, J.M. (2011). The future of survey sampling. Public Opinion Quarterly, 75, 872-888.
Chambers, R. (2014). Survey sampling in official statistics – Some thoughts on directions. Proceedings of the 2014 International Methodology Symposium, Statistics Canada, Ottawa, Canada.
Chen, Y., Li, P. and Wu, C. (2019). Doubly robust inference with non-probability survey samples. Journal of the American Statistical Association (published online).
Chen, J.K.T., Valliant, R.L. and Elliott, M.R. (2018). Model-assisted calibration of non-probability sample survey data using adaptive LASSO. Survey Methodology, 44, 1, 117-144. Paper available at https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2018001/article/54963-eng.pdf.
Citro, C.F. (2014). From multiple modes for surveys to multiple data sources for estimates. Survey Methodology, 40, 2, 137-161. Paper available at https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2014002/article/14128-eng.pdf.
Couper, M.P. (2000). Web surveys: A review of issues and approaches. Public Opinion Quarterly, 64, 464-494.
Couper, M.P. (2013). Is the sky falling? New technology, changing media, and the future of surveys. Survey Research Methods, 7, 145-156.
Deville, J.-C. (1991). A theory of quota surveys. Survey Methodology, 17, 2, 163-181. Paper available at https://www150.statcan.gc.ca/n1/en/pub/12-001-x/1991002/article/14504-eng.pdf.
Deville, J.-C. (1998). La correction de la non-réponse par calage ou par échantillonnage équilibré. Proceedings of the Survey Methods Section, Statistical Society of Canada, Sherbrooke, Canada.
Deville, J.-C., and Dupont, F. (1993). Non-réponse : principes et méthodes. Actes des Journées de Méthodologie Statistique, 53-69, December 15 and 16, 1993, INSEE, Paris.
Deville, J.-C., and Lavallée, P. (2006). Indirect sampling: The foundations of the generalized weight share method. Survey Methodology, 32, 2, 165-176. Paper available at https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2006002/article/9551-eng.pdf.
Deville, J.-C., and Särndal, C.-E. (1992). Calibration estimators in survey sampling. Journal of the American Statistical Association, 87, 376-382.
D’Orazio, M., Di Zio, M. and Scanu, M. (2006). Statistical Matching: Theory and Practice. New York: John Wiley & Sons, Inc.
Dutwin, D., and Buskirk, T.D. (2017). Apples to oranges or gala versus golden delicious? Comparing data quality of nonprobability internet samples to low response rate probability samples. Public Opinion Quarterly, 81, 213-249.
Elliott, M., and Valliant, R. (2017). Inference for non-probability samples. Statistical Science, 32, 249-264.
Eltinge, J.L., and Yansaneh, I.S. (1997). Diagnostics for formation of nonresponse adjustment cells, with an application to income nonresponse in the U.S. Consumer Expenditure Survey. Survey Methodology, 23, 1, 33-40. Paper available at https://www150.statcan.gc.ca/n1/en/pub/12-001-x/1997001/article/3103-eng.pdf.
Fay, R.E., and Herriot, R.A. (1979). Estimates of income for small places: An application of James-Stein procedures to census data. Journal of the American Statistical Association, 74, 269-277.
Gelman, A., and Little, T.C. (1997). Poststratification into many categories using hierarchical logistic regression. Survey Methodology, 23, 2, 127-135. Paper available at https://www150.statcan.gc.ca/n1/en/pub/12-001-x/1997002/article/3616-eng.pdf.
Haziza, D., and Beaumont, J.-F. (2007). On the construction of imputation classes in surveys. International Statistical Review, 75, 25-43.
Haziza, D., and Beaumont, J.-F. (2017). Construction of weights in surveys: A review. Statistical Science, 32, 206-226.
Haziza, D., and Lesage, É. (2016). A discussion of weighting procedures for unit nonresponse. Journal of Official Statistics, 32, 129-145.
Hidiroglou, M.A. (2001). Double sampling. Survey Methodology, 27, 2, 143-154. Paper available at https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2001002/article/6091-eng.pdf.
Hidiroglou, M.A., Beaumont, J.-F. and Yung, W. (2019). Development of a small area estimation system at Statistics Canada. Survey Methodology, 45, 1, 101-126. Paper available at https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2019001/article/00009-eng.pdf.
Hinkins, S., Oh, H.L. and Scheuren, F. (1997). Inverse sampling design algorithms. Survey Methodology, 23, 1, 11-22. Paper available at https://www150.statcan.gc.ca/n1/en/pub/12-001-x/1997001/article/3101-eng.pdf.
Iannacchione, V.G., Milne, J.G. and Folsom, R.E. (1991). Response probability weight adjustments using logistic regression. Proceedings of the Survey Research Methods Section, American Statistical Association, 637-642, Alexandria, VA.
Kalton, G. (2019). Developments in survey research over the past 60 years: A personal perspective. International Statistical Review, 87, S10-S30.
Kim, J.K., and Fuller, W. (2004). Fractional hot deck imputation. Biometrika, 91, 559-578.
Kim, J.K., and Rao, J.N.K. (2012). Combining data from two independent surveys: A model-assisted approach. Biometrika, 99, 85-100.
Kim, J.K., and Tam, S.M. (2020). Data integration by combining big data and survey data for finite population inference. Unpublished manuscript.
Kim, J.K., and Wang, Z. (2019). Sampling techniques for big data analysis. International Statistical Review, 87, S177-S191.
Kim, J.K., Wang, Z., Zhu, Z. and Cruze, N.B. (2018). Combining survey and non-survey data for improved sub-area prediction using a multi-level model. Journal of Agricultural, Biological and Environmental Statistics, 23, 175-189.
Kott, P.S. (2006). Using calibration weighting to adjust for nonresponse and coverage errors. Survey Methodology, 32, 2, 133-142. Paper available at https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2006002/article/9547-eng.pdf.
Kott, P.S. (2019). A partially successful attempt to integrate a Web-recruited cohort into an address-based sample. Survey Research Methods, 13, 95-101.
Laflamme, F., and Karaganis, M. (2010). Development and implementation of responsive design for CATI surveys at Statistics Canada. Proceedings of the European Conference on Quality in Official Statistics, Helsinki, Finland, May 2010.
Lavallée, P. (2007). Indirect Sampling. New York: Springer.
Lavallée, P., and Brisbane, J. (2016). Sample matching: Towards a probabilistic approach for web surveys and big data? Paper presented at the Advisory Committee on Statistical Methods, May 2016, Statistics Canada.
Lee, S. (2006). Propensity score adjustment as a weighting scheme for volunteer panel Web survey. Journal of Official Statistics, 22, 329-349.
Lesage, É. (2017). Combiner des données d’enquêtes probabilistes et des données massives non probabilistes pour estimer des paramètres de population finie. Unpublished manuscript.
Little, R.J.A. (1986). Survey nonresponse adjustments for estimates of means. International Statistical Review, 54, 139-157.
Lohr, S., and Raghunathan, T.E. (2017). Combining survey data with other data sources. Statistical Science, 32, 293-312.
Lundquist, P., and Särndal, C.-E. (2013). Aspects of responsive design with applications to the swedish living conditions survey. Journal of Official Statistics, 29, 557-582.
Meng, X.-L. (2018). Statistical paradises and paradoxes in big data (I): Law of large populations, big data paradox, and the 2016 US presidential election. Annals of Applied Statistics, 12, 685-726.
Mercer, A.W., Kreuter, F., Keeter, S. and Stuart, E.A. (2017). Theory and practice in nonprobability surveys: Parallels between causal inference and survey inference. Public Opinion Quarterly, 81, 250-271.
Merkouris, T. (2004). Combining independent regression estimators from multiple surveys. Journal of the American Statistical Association, 99, 1131-1139.
Merkouris, T. (2010). Combining information from multiple surveys by using regression for efficient small domain estimation. Journal of the Royal Statistical Society: Series B, 72, 27-48.
Miller, P.V. (2017). Is there a future for surveys? Public Opinion Quarterly, 81, 205-212.
Neyman, J. (1934). On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection. Journal of the Royal Statistical Society, 97, 558-625.
Phipps, P., and Toth, D. (2012). Analyzing establishment nonresponse using an interpretable regression tree model with linked administrative data. Annals of Applied Statistics, 6, 772-794.
Rancourt, E. (2019). Admin-first as a statistical paradigm for Canadian statistics: Meaning, challenges and opportunities. Proceedings of Statistics Canada’s 2018 International Methodology Symposium (to appear).
Rao, J.N.K. (2005). Interplay between sample survey theory and practice: An appraisal. Survey Methodology, 31, 2, 117-138. Paper available at https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2005002/article/9040-eng.pdf.
Rao, J.N.K. (2020). Making inference by combining data from multiple sources: An appraisal. Sankhyā (under review).
Rao, J.N.K., and Fuller, W. (2017). Sample survey theory and methods: Past, present and future directions. Survey Methodology, 43, 2, 145-160. Paper available at https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2017002/article/54888-eng.pdf.
Rao, J.N.K., and Molina, I. (2015). Small Area Estimation. Second Edition, Hoboken, New Jersey: John Wiley & Sons, Inc.
Rao, J.N.K., Scott, A.J. and Benhin, E. (2003). Undoing complex survey data structures: Some theory and applications of inverse sampling (with discussion). Survey Methodology, 29, 2, 107-128. Paper available at https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2003002/article/6787-eng.pdf.
Rässler, S. (2012). Statistical Matching: A Frequentist Theory, Practical Applications, and Alternative Bayesian Approaches. Lecture Notes in Statistics, New York: Springer, 168.
Rivers, D. (2007). Sampling from web surveys. Proceedings of the Survey Research Methods Section, American Statistical Association, Alexandria, VA.
Rivest, L.-P., and Belmonte, E. (2000). A conditional mean squared error of small area estimators. Survey Methodology, 26, 1, 67-78. Paper available at https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2000001/article/5179-eng.pdf.
Royall, R.M. (1970). On finite population sampling theory under certain linear regression models. Biometrika, 57, 377-387.
Royall, R.M. (1976). The linear least-squares prediction approach to two-stage sampling. Journal of the American Statistical Association, 71, 657-664.
Särndal, C.-E., Lumiste, K. and Traat, I. (2016). Reducing the response imbalance: Is the accuracy of the survey estimates improved? Survey Methodology, 42, 2, 219-238. Paper available at https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2016002/article/14663-eng.pdf.
Schonlau, M., and Couper, M.P. (2017). Options for conducting Web surveys. Statistical Science, 32, 279-292.
Schouten, B., Calinescu, M. and Luiten, A. (2013). Optimizing quality of response through adaptive survey designs. Survey Methodology, 39, 1, 29-58. Paper available at https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2013001/article/11824-eng.pdf.
Squire, P. (1988). Why the 1936 Literary Digest Poll failed. Public Opinion Quarterly, 52, 125-133.
Tourangeau, R., Brick, J.M., Lohr, S. and Li, J. (2017). Adaptive and responsive survey designs: A review and assessment. Journal of the Royal Statistical Society, 180, 201-223.
Valliant, R., and Dever, J.A. (2011). Estimating propensity adjustments for volunteer web surveys. Sociological Methods & Research, 40, 105-137.
Valliant, R., Dorfman, A. and Royall, R.M. (2000). Finite Population Sampling: A Prediction Approach. New York: John Wiley & Sons Inc.
Wolter, K.M. (2007). Introduction to Variance Estimation. Second Edition, New-York: Springer.
Wu, C., and Sitter, R.R. (2001). A model-calibration approach to using complete auxiliary information from survey data. Journal of the American Statistical Association, 96, 185-193.
Ybarra, L.M., and Lohr, S.L. (2008). Small area estimation when auxiliary information is measured with error. Biometrika, 95, 919-931.
Zhang, L.C. (2012). Topics of statistical theory for register-based statistics and data integration. Statistica Neerlandica, 66, 41-63.
- Date modified: