Bayesian hierarchical weighting adjustment and survey inference
Section 1. Introduction

• Previous
• Next

1.1  Background

Design-based and model-based approaches have long been contrasted in survey research (Little, 2004). The former automatically takes into account survey design, while the latter can yield robust inference for small sample estimation. Rao (2011) provides an appraisal of frequentist and Bayesian methods on survey sampling practice. Classical design-based approaches use weights to adjust the sample to the population; see Chen, Elliott, Haziza, Yang, Ghosh, Little, Sedransk and Thompson (2017) for a review of various weighted estimators for a population mean. However, classical survey weighting usually relies on many user-defined choices so that the process of weighting can be difficult to codify in real-world surveys (Gelman, 2007). The Bayesian approach for finite population inference (Ghosh and Meeden, 1997) allows prior information to be incorporated, when appropriate, but is subject to model misspecification.

In the present paper we combine Bayesian prediction and weighting in a unified approach to survey inference, applying scalable and robust Bayesian regression models to account for complex design features under the framework of multilevel regression and poststratification (MRP, Gelman and Little (1997); Park, Gelman and Bafumi (2005); Ghitza and Gelman (2013); Si, Pillai and Gelman (2015)). MRP adjusts for complex design and response mechanisms and improves small area estimation (Fay and Herriot, 1979; Rao and Molina, 2015). We deal with different complex issues caused by real-life applications and much finer levels for subdomain inference of interest. Our method yields efficient and valid finite population inference, especially for subgroups, and constructs model-based weights after smoothing.

The contributions of this paper are two folded: 1) as innovative Bayesian methodology developments we develop a new structured prior setting to handle high-order interaction terms; and 2) to improve survey research and operation, we combine Bayesian prediction and weighting as a unified approach to survey inference, accounting for design features in the Bayesian modeling. We generalize MRP for finite population inferences and construct stable and calibrated model-based weights to solve the problems of classical weights. We disseminate the R package rstanarm implementing the proposed methods for public use, promoting the model-based approaches in survey research and operational practice. More importantly, the paper builds the groundwork to use MRP in the survey weighting adjustment and data integration, for example, to make inferences with nonprobability surveys. Our proposed methods offer one important and practical tool for designing and weighting survey samples (Valliant, Dever and Kreuter, 2018).

1.2  Framework

For a finite population of $N$ units, we denote the variable of interest as $y\mathrm{<mo>=</mo>}\left(y_1\mathrm{,}\dots \mathrm{,}{y}_N\right)$ and the inclusion indicator variable as $I\mathrm{<mo>=</mo>}\left(I_1\mathrm{,}\dots \mathrm{,}{I}_N\right),$ where $I_i\mathrm{<mo>=</mo>}1$ if unit $i$ is included in the sample and $I_i\mathrm{<mo>=</mo>}0$ otherwise. Here, inclusion refers to selection and response. The general inference framework considers the joint distribution for $I$ and $y.$ Design-based inference considers the distribution of $I$ and treats $y$ as fixed. Under probability sampling, model-based inferences can be based on the distribution of $y$ alone given the variables that affect the inclusion mechanisms are included in the model (Royall, 1968), that is, under the ignorable inclusion mechanism when the distribution of $I$ given $y$ is independent of the distribution of $y$ (Rubin, 1976, 1983).

To account for the factors that affect inclusion, classical design weights adjust for unequal probabilities of sampling, with subsequent weighting adjustments accounting for coverage problems and nonresponse during data collection or data cleaning. Classical weights are thus generated as a product of multiple adjustment factors: inverse probability of selection, inverse propensity score of response, and poststratification (also called calibration or benchmarking; Holt and Smith (1979)). Each of these adjustments can be approximate when the probability of selection, the probability of response, or population totals are estimated from data. Beyond any approximation issues, even if the inclusion model is known exactly, extreme values of weights will cause high variability and then inferential problems, especially when the weights are weakly correlated with the survey outcome variable (Rao, 1966a, b; Hájek, 1971; Särndal, Swensson and Wretman, 1992). When the weighting process involves poststratification or nonresponse adjustment – where the weights themselves are random variables – the variance estimation will be different from the cases only with fixed design weights. It is nontrivial to analytically derive a variance estimator under the multi-stage weighting adjustment or complex sampling design.

In practice, the construction of survey weights requires somewhat arbitrary decisions of the selection of variables and interactions, pooling of weighting cells, and weight trimming. It can be unclear whether and how to incorporate auxiliary information (Groves and Couper, 1995). Discussion of smoothing and trimming in the survey weighting literature (e.g., Potter, 1988, 1990; Elliott and Little, 2000; Elliott, 2007; Xia and Elliott, 2016) has focused on estimating the finite population total or mean, with less attention to subdomain estimates. Beaumont (2008) proposes to regress weights on the survey variables and use the predicted values as smoothed weights, where the direction is inspiring but tangential to the inference objective where good inference properties are desired for the survey variable of interest rather than the weights. Borrowing information on survey outcomes potentially increases efficiency and calls for a general framework.

Gelman (2007) recommends regression models including as covariates any variables that affect selection and response, including stratification variables, clusters, and auxiliary information. Any of these approaches can be sensitive to the prior specification for stable estimation; this is the model-based counterpart to the decisions required for smoothing or trimming classical survey weights. Flexible prediction techniques, such as spline functions, penalized regression and tree-based models, have been proposed to accommodate model-assisted survey estimation (Särndal et al., 1992; Wu and Sitter, 2001; Breidt and Opsomer, 2017; McConville and Toth, 2018).

Model-based and model-assisted weighting adjustment methods for finite population total estimation have been compared by Henry and Valliant (2012). The model-based weighting methods in the superpopulation perspective (Valliant, Dorfman and Royall, 2000) use predictions from regression models to derive case weights, where the predictions are based on hierarchical linear regression models with various bias corrections (Chambers, Dorfman and Wehrly, 1993; Firth and Bennett, 1998). Based on the finite population total estimation, model-assisted methods derive case weights mainly from calibration on benchmark variables (Kott, 2009) via the generalized regression estimator (GREG, Deville and Särndal (1992)). However, the case weights derived from regression predictions can be highly variable and even negative and may damage some domain estimates. Model-based approaches play a vital role in small area estimation but are subject to misspecification and need new developments when the number of domains is large and the inclusion mechanism is not simply random.

To protect against model misspecification, Little (1983) recommends modeling differences in the distribution of outcomes across classes defined by differential probabilities of inclusion. Si et al. (2015) construct poststratification cells based on the unique values of inclusion probabilities and build hierarchical models to smooth cell estimates as advocated by Little (1991, 1993).

We propose to use Bayesian hierarchical models accounting for survey design to generate weights that can be used in design-based inference. The inference is well calibrated and valid with good frequentist properties (Little, 2011). For large samples, the inference will parallel with design-based inference. For small samples, the hierarchical model smoothing will stabilize domain estimation and generate robust weighting adjustment.

We use the intrinsic variables that are used for design weight construction, nonresponse adjustment and calibration, assume they are discretized, and construct poststratification cells based on the cross-tabulation. Weights are derived through the regressing survey outcome on variables used for weighting given the poststratification. The inclusion of the outcome variable into weighting and poststratification can avoid model misspecification and potentially increase efficiency (Fuller, 2009). Multilevel model estimates shrink the cell estimates towards the prediction from the regression model. The MRP framework combines multilevel regression and poststratification, accounts for design features in the Bayesian paradigm, and is then well equipped to handle complex design features. Our proposal distinguishes from the model-based weights in the literature by using the poststratification cell structure and improves by smoothing, thus avoiding negative weight values.

Si et al. (2015) incorporate weights into MRP, increasing flexibility and efficiency comparing to the pseudo-likelihood approach (Pfeffermann, 1993). In the present paper we go further, starting from the variables that are used for weighting and constructing model-based weights as byproducts under MRP. We develop a novel prior specification for the regularization to handle potentially large numbers of poststratification cells. The prior setting allows for variable selection and keeps the hierarchical structure among main effects and high-order interaction terms for categorical variables. That is, if one variable is not predictive, then the high-order interactions involved with this variable are also likely to be not predictive, to facilitate model interpretation. McConville and Toth (2018) use tree-based methods to automatically select poststrata based on auxiliary variables that are potentially correlated with the survey outcome. Our proposed structured prior plays a similar role with the recursive partitioning algorithm to facilitate poststrata selection but improves efficiency by partial pooling. We use the smoothed weights and estimates that are more stable than the regression tree estimator, and the Bayesian framework propagates all sources of uncertainty while McConville and Toth (2019) ignore the variance for tree growing and use mean squared error to approximate the variance.

We have implemented the computation in the R package rstanarm (Goodrich and Gabry, 2017). The fully Bayesian inference is realized via Stan (Stan Development Team, 2018, 2017), which uses Hamiltonian Monte Carlo sampling with adaptive path lengths (Hoffman and Gelman, 2014). Stan promotes robust model-based approaches by reducing the computational burden of building and testing new models. The rstanarm package allows for efficient Bayesian hierarchical modeling and weighting inference. The codes are publicly available and reproducible. Our developed computation software provides the accessible platform and has the potential to support the unified framework for survey inference.

Section 2 introduces the motivating problem of weighting for an ongoing social science survey. We discuss the method in detail Section 3. Section 4 describes the statistical evaluation of model-based prediction and weighting inference, and demonstrate the efficiency gains in comparison with classical weighting. We apply the proposal to the real-life survey in Section 5. Section 6 summarizes the improvement and discusses further extension.

• Previous
• Article main page
• Next