Bayesian hierarchical weighting adjustment and survey inference
Section 6. Discussion

Table of contents

We combine Bayesian prediction and weighting as a unified approach to survey inference. Multilevel regression with structured prior distributions and poststratification on the population inference yield efficient estimation when accounting for the design feature. The computation is implemented via Stan and disseminated through the R package rstanarm for public use, and the software development promotes the model-based approaches in survey research and operational practice. We construct stable and calibrated model-based weights to solve the problems of classical weights. This article builds up the model-based prediction and weighting framework and serves as the first contribution to evaluate the statistical properties of model-based weights and compare the performances with classical weighting. Model-based weights are smoothed across poststratification cells and improve small domain estimation.

The structured prior uses the hierarchical structure between the main effects and high-order interaction terms to introduce multiplicative constraints on the corresponding scale parameters and informs variable selection. Model improvement can be done after post-processing the posterior inferences. The Bayesian structural model yields more stable inference than that with independent prior distributions. Such hierarchy assumption may not be valid for special cases, such as the Exclusive-Or problem where two variables show no main effects but a perfect interaction. However, we do not have strong evidence in the application studies against the plausibility of hierarchy. Furthermore, the unified prediction and weighting approach is well equipped to deal with complex survey designs and big data in surveys, such as streaming data and combining multiple survey studies.

The general MRP framework is open to flexible modeling strategies. In this article, we illustrate by a regression model with all variables of interest and the high-order interactions and incorporate structured prior distributions for regularization. Other approaches, such as nonparametric models and machine learning tools, can be implemented under the MRP framework, being robust against model misspecification. Si et al. (2015) use Gaussian process regression models to borrow information across poststratification cells based on the distances between the inverse inclusion probability weights. Further extensions include applying such flexible approaches to weight smoothing and deriving the model-based weights.

The broad application opportunities come with various challenges that need further investigation. The model-based weights are outcome dependent, which improves the efficiency but potentially reduces the robustness. Survey organizers prefer a set of weights that can be used for general analysis purpose, without being sensitive to outcome selection. We can compare different weights constructed by several important outcomes and conduct sensitivity analysis. When the model-based weights give different inference conclusions, we recommend choosing the set of weights that generate the most reasonable results, with scientific reasoning and be consistent with the population inference.

The weighted marginal distributions of the calibration variables are a bit different from the population inferences, as in Section 5, which does not meet the usual weighting diagnosis standard of survey organizers. The model-weights tend to match the joint distribution to that in the population, but weight smoothing may bring in bias. Tradeoff constraints can be induced to the model to match the marginal distributions.

Another practical challenge is that the population distribution of the calibration variables may be unknown, that is, the population poststratification cell sizes N j ’ s MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacH8rrps0l bbf9q8WrFfeuY=Hhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaacaWGobWaaSbaaSqaaiaadQgaaeqaaG qaaOGaa8xgGiaa=nhaaaa@351A@ are unknown. A supplemental model is needed to allow estimation of this information from the sample and integrated with MRP to propagate all sources of uncertainty as an extension, similar to the framework in Si and Zhou (2020) by incorporating known margins. The model-based predictions and weighting inferences need further extensions to handle discrete outcomes, inference on regression coefficients and non-probability or informative sampling designs (Kim and Skinner, 2013). It will be useful to link these ideas on survey inference with the biostatistical and econometrics literature on inverse propensity score and doubly robust weighting (Kang and Schafer, 2007).

Acknowledgements

We thank the National Science Foundation, National Institutes of Health, Office of Naval Research, Institute of Education Sciences, and Sloan Foundation for grant support.

Appendix

A. Example code

Here we present code for the application described in the data. We have written a function model_based_cell_weights to calculate the model-based weights from a fitted rstanarm model.

model_based_cell_weights <− function(object, cell_table) {
stopifnot(
is.data.frame(cell_table),
colnames(cell_table) == c("N", "n")
)
draws <− as.matrix(object)
Sigma <− draws[, grep("^Sigma\\[", colnames(draws)), drop = FALSE]
sigma_theta_sq <− rowSums(Sigma)
sigma_y_sq <− draws[, "sigma"]^2
Ns <− cell_table[["N"]] # population cell counts
ns <− cell_table[["n"]] # sample cell counts
J <− nrow(cell_table)
N <− sum(Ns)
n <− sum(ns)
# implementing equation 7 in the paper (although i did some algebra first to
# simplify the expression a bit)
Nsy2 <− N * sigma_y_sq
ww <− matrix(NA, nrow = nrow(draws), ncol = J)
for (j in 1:J) {
ww[, j] <−
(Nsy2 + n * Ns[j] * sigma_theta_sq) / (Nsy2 + N * ns[j] * sigma_theta_sq)
}
return(ww)
}
# prepare population data: acs_ad has age, eth, edu and inc
acs_ad %>%
mutate(
cell_id = paste0(age, eth, edu, inc)
) −> acs_ad
acs_design <− svydesign(id = ~1, weights = ~perwt, data = acs_ad)
agg_pop <−
svytable( ~ age + eth + edu + inc, acs_design) %>%
as.data.frame() %>%
rename(N = Freq) %>%
mutate(
cell_id = paste0(age, eth, edu, inc)
) %>%
filter(cell_id %in% acs_ad$cell_id)
# prepare data to pass to rstanarm
# SURVEYdata has 4 variables used for weighting: age, eth, edu and inc; and outcome Y
dat_rstanarm <−
SURVEYdata %>%
mutate(
cell_id = paste0(age, eth, edu, inc)
)%>%
group_by(age, eth, edu, inc) %>%
summarise(
sd_cell = sd(Y),
n = n(),
Y = mean(Y),
cell_id = first(cell_id)
) %>%
mutate(sd_cell = if_else(is.na(sd_cell), 0, sd_cell)) %>%
left_join(agg_pop[, c("cell_id", "N")], by = "cell_id")
# Stan fitting under structured prior in rstanarm
fit <−
stan_glmer(
formula =
Y ~ 1 + (1 | age) + (1 | eth) + (1 | edu) + (1 | inc) +
(1 | age:eth) + (1 | age:edu) + (1 | age:inc) +
  (1 | eth:edu) + (1 | eth:inc) +
  (1 | age:eth:edu) + (1 | age:eth:inc),
data = dat_rstanarm, iter = 1000, chains = 4, cores = 4,
prior_covariance =
rstanarm::mrp_structured(
cell_size = dat_rstanarm$n,
cell_sd = dat_rstanarm$sd_cell,
group_level_scale = 1,
  group_level_df = 1
),
seed = 123,
prior_aux = cauchy(0, 5),
prior_intercept = normal(0, 100, autoscale = FALSE),
adapt_delta = 0.99
)
# model-based weighting
cell_table <− fit$data[,c("N","n")]
weights <− model_based_cell_weights(fit, cell_table)
weights <− data.frame(w_unit = colMeans(weights),
cell_id = fit$data[["cell_id"]],
Y = fit$data[["Y"]],
n = fit$data[["n"]]) %>%
mutate(
w = w_unit / sum(n / sum(n) * w_unit), # model-based weights
Y_w = Y * w
)
with(weights, sum(n * Y_w / sum(n)))# mean estimate

B. Simulation designs

Here we present the simulation designs, coefficient values, and comparison on the weighted distributions of socio-demographics as a supplement to Sections 4 and 5.


Table B.1
Covariates in the outcome (O) and selection (S) models for slightly unbalanced design
Table summary
This table displays the results of Covariates in the outcome (O) and selection (S) models for slightly unbalanced design Case 1, Case 2, Case 3, Case 4, Case 5, Case 6 and Case 7 (appearing as column headers).
Case 1 Case 2 Case 3 Case 4 Case 5 Case 6 Case 7
O S O S O S O S O S O S O S
age ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
eth ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ This is an empty cell This is an empty cell ✓ This is an empty cell ✓
edu ✓ ✓ ✓ ✓ ✓ This is an empty cell ✓ ✓ ✓ This is an empty cell ✓ ✓ ✓ This is an empty cell
age*eth ✓ This is an empty cell This is an empty cell ✓ ✓ ✓ This is an empty cell This is an empty cell This is an empty cell ✓ This is an empty cell This is an empty cell This is an empty cell ✓
age*edu ✓ This is an empty cell This is an empty cell ✓ ✓ This is an empty cell This is an empty cell This is an empty cell This is an empty cell This is an empty cell ✓ This is an empty cell ✓ This is an empty cell
eth*edu ✓ This is an empty cell This is an empty cell ✓ ✓ This is an empty cell This is an empty cell This is an empty cell This is an empty cell This is an empty cell This is an empty cell This is an empty cell This is an empty cell This is an empty cell
age*eth*edu ✓ This is an empty cell This is an empty cell ✓ ✓ This is an empty cell This is an empty cell This is an empty cell This is an empty cell This is an empty cell This is an empty cell This is an empty cell This is an empty cell This is an empty cell

Table B.2
Assumed regression coefficients in the outcome model for the simulation using a slightly unbalanced design
Table summary
This table displays the results of Assumed regression coefficients in the outcome model for the simulation using a slightly unbalanced design All , Main effects and Two variables (appearing as column headers).
All Main effects Two variables
age (0.5, 1.375, 2.25, 3.125, 4) (0.5, 1.375, 2.25,
3.125, 4)		 (0.5, 1.375, 2.25, 3.125, 4)
eth (-2, -1, 0, 1, 2) (2, -1, 0, 1, 2) 0 → MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaaceaIWaGbaSaaaaa@345A@
edu (3, 2, 1, 0) (3, 2, 1, 0) (3, 2, 1, 0)
age*eth (4, 2, 1, 1, 3, 3, 2, 1, 1, 1, 2, 3, 2, 2, 1, 4, 4, 3, 2, 3, 2, 4, 1, 4, 1) 0 → MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaaceaIWaGbaSaaaaa@345A@ 0 → MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaaceaIWaGbaSaaaaa@345A@
age*edu (-2, -1, 2, 2, 1, -2, 2, 1, 0, -2, 1, -2, -1, 2, 1, -1, -1, 2, 0, 2) 0 → MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaaceaIWaGbaSaaaaa@345A@ (2, 0, -2, -2, 1, 1, -1, -2, -2, -1, -1, 1,
0, -1, -1, 2, 2, 1, -1, 0)
eth*edu (1, -2, 0, -3, -1, 0, -1, -2, 0, -1, -3, -3, 0, -1, -1, 0, 0, -1, 0, -1) 0 → MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaaceaIWaGbaSaaaaa@345A@ 0 → MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaaceaIWaGbaSaaaaa@345A@
age*eth*edu (-1, -0.5, 0.5, -1, -1, -0.5, -1, 0, -1, 0, -1, 0, 1, 1, 0.5, 1, 1, -1, -1, 0, -1, -0.5, -0.5, -1, 1, -1, -0.5, -1, 1, 0, 0.5, 0.5, 1, 0.5, 1, 1, 1, 0.5, 1, 0, 0, -0.5, 0, 1, -1, -1, 0, -1, -1, -1, -0.5, -0.5, 0, 1, -1, 0, 0, -0.5, 1, -0.5, 0.5, -1, 1, 0, 1, 0, -1, 0, -0.5, 1, -0.5, -1, -0.5, 0, 0.5, -0.5, 1, 0.5, -0.5, 0.5, 0, 1, 0, 1, 0.5, 0.5, 0.5, 0, 0, -0.5, 1, -1, 0, 1, 1, 1, 1, -0.5, -1, -1) 0 → MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaaceaIWaGbaSaaaaa@345A@ 0 → MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaaceaIWaGbaSaaaaa@345A@

Table B.3
Assumed regression coefficients in the selection model for the simulation using a slightly unbalanced design
Table summary
This table displays the results of Assumed regression coefficients in the selection model for the simulation using a slightly unbalanced design All, Main effects and Two variables (appearing as column headers).
All Main effects Two variables
Intercept -2 -2 -2
age (-2, -1.75, -1.5, -1.25, -1) (0, 0.5, 1, 1.5, 2) (-2, -1.5, -1, -0.5, 0)
eth (-1, -0.25, 0.5, 1.25, 2) (-2, -1.5, -1, -0.5, 0) (-1, -0.5, 0, 0.5, 1)
edu (0, 0.67, 1.33, 2) (0, 1, 2, 3) 0 → MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaaceaIWaGbaSaaaaa@345A@
age × MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaacqGHxdaTaaa@35A5@ eth (1, 1, -1, 1, -1, 1, -1, 0, 0, -1, 0, 0, -1, 1, 0, 0, -1, 1,
1, -1, -1, 0, 1, -1, 1)		 0 → MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaaceaIWaGbaSaaaaa@345A@ (-1, 1, 1, 1, -1, -1, -1, 0, -1, -1, -1, -1, 1, -1, -1, 0, 1, 1, -1, 1, -1, -1, 1, 0, 0)
age × MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaacqGHxdaTaaa@35A5@ edu (0, 1, -1, -1, 0, 1, 1, 0, 1, 0, 1, -1, -1, 1, 1, -1, 0, -1, 1, 1) 0 → MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaaceaIWaGbaSaaaaa@345A@ 0 → MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaaceaIWaGbaSaaaaa@345A@
eth × MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaacqGHxdaTaaa@35A5@ edu (-1, -1, 0, -1, -1, 1, 1, 1, 1, 0, -1, 0, -1, 0, -1, 1, 0, -1, -1, -1) 0 → MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaaceaIWaGbaSaaaaa@345A@ 0 → MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaaceaIWaGbaSaaaaa@345A@
age × MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaacqGHxdaTaaa@35A5@ eth × MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaacqGHxdaTaaa@35A5@ edu (0.8, -0.4, 0.6, -0.2, 0.8, 0.2, 0.4, 0.8, 0.4, -0.6, -0.8, -0.4, -0.8, -0.4, 0.4, -1, 0.6, -0.8, -0.6, 0.6, -0.2, 0.2, 0.6, -0.6, 0, 0, -1, -0.2, 0.6, 0.8, -0.4, 0.2, -0.8, 0.4, 0.6, -0.6, 0.8, 0, 0.2, -1, 1, 0.4, 0, 0.8, -0.2, 0, 0, 0.6, -0.8, -0.8, -0.2, 0.4, -1, -0.8, 1, -0.2, 0, 0.8, 0.6, 0.8, -0.2, -0.2, -0.8, 1, 0.8, 0.8, -0.4, -0.8, 0.4, -0.4, 1, -0.6, -1, -0.6, -0.2, 1, 1, -0.2, 1, 0.6, 0.4, 0.8, 0.2, -0.2, -0.6, 0, 0.8, -0.4, 0.4, 0.4, 0.6, -1, -0.8, -0.8, 1, 1, 0.4, 0.6, 0.4, 0.8) 0 → MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaaceaIWaGbaSaaaaa@345A@ 0 → MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaaceaIWaGbaSaaaaa@345A@

Table B.4
Covariates in the outcome (O) and selection (S) models for a very unbalanced design
Table summary
This table displays the results of Covariates in the outcome (O) and selection (S) models for a very unbalanced design Case 1, Case 2, Case 3 and Case 4 (appearing as column headers).
Case 1 Case 2 Case 3 Case 4
O S O S O S O S
age ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
eth ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
edu ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
sex ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
pov ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
cld ✓ This is an empty cell ✓ This is an empty cell ✓ ✓ ✓
eld ✓ ✓ This is an empty cell ✓ ✓ ✓ ✓ ✓
fam ✓ ✓ This is an empty cell ✓ ✓ ✓ ✓ ✓
age*eth ✓ ✓ This is an empty cell This is an empty cell ✓ This is an empty cell This is an empty cell ✓
age*edu ✓ ✓ This is an empty cell This is an empty cell ✓ This is an empty cell This is an empty cell ✓
eth*edu ✓ ✓ This is an empty cell This is an empty cell ✓ This is an empty cell This is an empty cell ✓
eth*pov ✓ ✓ This is an empty cell This is an empty cell ✓ This is an empty cell This is an empty cell ✓
age*pov ✓ ✓ This is an empty cell This is an empty cell ✓ This is an empty cell This is an empty cell ✓
pov*fam ✓ ✓ This is an empty cell This is an empty cell ✓ This is an empty cell This is an empty cell ✓
pov*eld ✓ ✓ This is an empty cell This is an empty cell ✓ This is an empty cell This is an empty cell ✓
pov*cld This is an empty cell ✓ This is an empty cell This is an empty cell This is an empty cell This is an empty cell This is an empty cell ✓
age*eth*edu ✓ ✓ This is an empty cell This is an empty cell ✓ This is an empty cell This is an empty cell ✓
age*eth*pov ✓ ✓ This is an empty cell This is an empty cell ✓ This is an empty cell This is an empty cell ✓

Table B.5
Assumed regression coefficients in the outcome (O) and selection (S) models for a very unbalanced design
Table summary
This table displays the results of Assumed regression coefficients in the outcome (O) and selection (S) models for a very unbalanced design O and S (appearing as column headers).
O S
age (2, 0, -2, -2, 1) (0, 0.75, 1.5, 2.25, 3)
eth (1, -1, -2, -2, -1) (-1, -0.5, 0, 0.5, 1)
edu (-1, 1, 0, -1) (0, 0.67, 1.33, 2)
sex (-1, 2) (-1, 0)
pov (2, 1, -1, 0, -1) (0, 1, 2, 3, 4)
cld 0 → MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaaceaIWaGbaSaaaaa@345A@ (-1, -0.33, 0.33, 1)
eld 0 → MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaaceaIWaGbaSaaaaa@345A@ (-2, -1, 0)
fam 0 → MathType@MTEF@5@5@+= feaagKart1ev2aqatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn hiov2DGi1BTfMBaeXatLxBI9gBaebbnrfifHhDYfgasaacPqpw0le9 v8qqaqFD0xXdHaVhbbf9v8WrFr0dc9vqFj0db9qqvqFr0dXdHiVc=b YP0xH8peeu0xXdcrpe0db9Wqpepec9ar=xfr=xfr=tmeaabaqaciGa caGaaeqabaGaaeaadaaakeaaceaIWaGbaSaaaaa@345A@ (-1, -0.67, -0.33, 0)

Table B.6
Euclidean distances between the weighted distributions and the population distribution. Str-W: model-based weighting under structured prior; Rake-W: weighting via raking adjustment; and PS-W: poststratification weighting
Table summary
This table displays the results of Euclidean distances between the weighted distributions and the population distribution. Str-W: model-based weighting under structured prior; Rake-W: weighting via raking adjustment; and PS-W: poststratification weighting Str-W, PS-W and Rake-W (appearing as column headers).
Str-W PS-W Rake-W
age 0.04 0.02 0.00
eth 0.08 0.06 0.00
edu 0.08 0.03 0.00
inc 0.02 0.02 0.00
age * eth 0.05 0.03 0.05
age * edu 0.05 0.02 0.05
age * inc 0.03 0.01 0.03
eth * edu 0.06 0.04 0.05
eth * inc 0.04 0.04 0.03
edu * inc 0.06 0.03 0.04
age * eth * edu 0.03 0.02 0.05
age * eth * inc 0.03 0.02 0.04
age * edu * inc 0.03 0.01 0.04
eth * edu * inc 0.04 0.02 0.04
age * eth * edu * inc 0.02 0.01 0.04

References

ACS Weighting Method (2014). American Community Survey Design and Methodology, Chapter 11: Weighting and Estimation. United States Census Bureau.

Beaumont, J.-F. (2008). A new approach to weighting and inference in sample surveys. Biometrika, 95, 539-553.

Breidt, F.J. (2008). Endogenous post-stratification in surveys: Classifying with a sample-fitted model. Annals of Statistics, 36, 403-427.

Breidt, F., and Opsomer, J. (2017). Model-assisted survey estimation with modern prediction techniques. Statistical Science, 32, 190-205.

Carvalho, C.M., Polson, N.G. and Scott, J.G. (2010). The horseshoe estimator for sparse signals. Biometrika, 97, 465-480.

Chambers, R.L., Dorfman, A.H. and Wehrly, T.E. (1993). Bias robust estimation in finite populations using nonparametric calibration. Journal of the American Statistical Association, 88,260-269.

Chen, Q., Elliott, M.R., Haziza, D., Yang, Y., Ghosh, M., Little, R., Sedransk, J. and Thompson, M. (2017). Approaches to improving survey-weighted estimates. Statistical Science, 32(2), 227-248.

Dahlke, M., Breidt, F., Opsomer, J. and Keilegom, I.V. (2013). Nonparametric endogenous poststratification in surveys. Statistica Sinica, 23, 189-211.

Deville, J.-C., and Särndal, C.-E. (1992). Calibration estimators in survey sampling. Journal of the American Statistical Association, 87, 376-382.

Deville, J.-C., Särndal, C.-E. and Sautory, O. (1993). Generalized raking procedures in survey sampling. Journal of the American Statistical Association, 88(423), 1013-1020.

Elliott, M.R. (2007). Bayesian weight trimming for generalized linear regression models. Journal of Official Statistics, 33(1), 23-34.

Elliott, M.R., and Little, R.J. (2000). Model-based alternatives to trimming survey weights. Journal of Official Statistics, 16(3), 191-209.

Fay, R.E., and Herriot, R.A. (1979). Estimates of income for small places: An application of James-Stein procedures to census data. Journal of the American Statistical Association, 74(366a), 269-277.

Firth, D., and Bennett, K.E. (1998). Robust models in probability sampling. Journal of the Royal Statistical Society, Series B, 60, 3-21.

Fuller, W. (2009). Sampling Statistics. Hoboken: John Wiley & Sons, Inc.

Gelman, A. (2005). Analysis of variance: Why it is more important than ever (with discusion). Annals of Statistics, 33(1), 1-53.

Gelman, A. (2006). Prior distributions for variance parameters in hierarchical models. Bayesian Analysis, 3, 515-533.

Gelman, A. (2007). Struggles with survey weighting and regression modeling. Statistical Science, 22(2), 153-164.

Gelman, A., and Carlin, J.B. (2001). Poststratification and weighting adjustments. In Survey Nonresponse, (Eds., R. Groves, D. Dillman, J. Eltinge and R. Little).

Gelman, A., and Little, T.C. (1997). Poststratifcation into many cateogiries using hierarchical logistic regression. Survey Methodology, 23, 2, 127-135. Paper available at https://www150.statcan.gc.ca/n1/en/pub/12-001-x/1997002/article/3616-eng.pdf.

Gelman, A., and Little, T.C. (1998). Improving on probability weighting for household size. Public Opinion Quarterly, 62, 398-404.

Ghitza, Y., and Gelman, A. (2013). Deep interactions with MRP: Election turnout and voting patterns among small electoral subgroups. American Journal of Political Science, 57(3), 762-776.

Ghosh, M., and Meeden, G. (1997). Bayesian Methods for Finite Population Sampling. Chapman Hall/CRC Press.

Goodrich, B., and Gabry, J.S. (2017). rstanarm: Bayesian applied regression modeling via Stan. https://cran.r-project.org/web/packages/rstanarm/.

Groves, R., and Couper, M. (1995). Theoretical motivation for post-survey nonresponse adjustment in household surveys. Journal of Offcial Statistics, 11, 93-106.

Hájek, J. (1971). Comment on “An essay on the logical foundations of survey sampling” by D. Basu. In The Foundations of Survey Sampling, (Eds., V.P. Godambe and D.A. Sprott), 236. Holt, Rinehart and Winston.

Hastie, T., Tibshirani, R. and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2ndEdition. Springer.

Henry, K., and Valliant, R. (2012). Comparing alternative weight adjustment methods. Proceedings of the Section on Survey Research Methods, American Statistical Association.

Hoffman, M.D., and Gelman, A. (2014). The No-U-Turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15, 1351-1381.

Holt, D., and Smith, T.M.F. (1979). Post stratification. Journal of the Royal Statistical Society, Series A, 142(1), 33-46.

Kang, J.D.Y., and Schafer, J.L. (2007). Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data. Statistical Science, 22(4), 523-539.

Kim, J.K., and Skinner, C.J. (2013). Weighting in survey analysis under informative sampling. Biometrika, 100, 385-398.

Kott, P. (2009). Calibration weighting: Combining probability samples and linear prediction models. In Handbook of Statistics, Sample Surveys: Design, Methods and Application, (Eds., D. Pfeffermann and C.R. Rao), Volume 29B. Elsevier.

Little, R. (1983). Comment on “An evaluation of model-dependent and probability-sampling inferences in sample surveys”, by M.H. Hansen, W.G. Madow and B.J. Tepping. Journal of the American Statistical Association, 78, 797-799.

Little, R. (1991). Inference with survey weights. Journal of Official Statistics, 7, 405-424.

Little, R. (1993). Post-stratification: A modeler’s perspective. Journal of the American Statistical Association, 88, 1001-1012.

Little, R. (2004). To model or not to model? Competing modes of inference for finite population sampling inference for finite population sampling. Journal of the American Statistical Association, 99, 546-556.

Little, R. (2011). Calibrated Bayes, for statistics in general, and missing data in particular. Statistical Science, 26, 162-174.

Little, R., and Wu, M. (1991). Models for contingency tables with known margins when target and sampled populations differ. Journal of the American Statistical Association, 86, 87-95.

McConville, K.S., and Toth, D. (2019). Automated selection of post-strata using a model-assisted regression tree estimator. Scandinavian Journal of Statistics, 46(2), 389-413.

Park, D.K., Gelman, A. and Bafumi, J. (2005). State-level opinions from national surveys: Poststratification using multilevel logistic regression. In Public Opinion in State Politics, (Ed., J.E. Cohen), Standord University Press.

Pfeffermann, D. (1993). The role of sampling weights when modeling survey data. International Statistical Review, 61(2), 317-337.

Piironen, J., and Vehtari, A. (2016). On the hyperprior choice for the global shrinkage parameter in the horseshoe prior. https://arxiv.org/abs/1610.05559.

Potter, F.A. (1988). Survey of procedures to control extreme sample weights. Proceedings of the Section on Survey Research Methods, American Statistical Association, 453-458.

Potter, F.A. (1990). A study of procedures to identify and trim extreme sampling weights. Proceedings of the Section on Survey Research Methods, American Statistical Association, 225-230.

Rao, J.N.K. (1966a). Alternative estimators in PPS sampling for multiple characteristics. Sankhyā, Series A, 28(1), 47-60.

Rao, J.N.K. (1966b). On the relative effciency of some estimators in PPS sampling for multiple characteristics. Sankhyā, Series A, 28(1), 61-70.

Rao, J.N.K. (2011). Impact of frequentist and bayesian methods on survey sampling practice: A selective appraisal. Statistical Science, 26(2), 240-256.

Rao, J.N.K., and Molina, I. (2015). Small Area Estimation. New York: John Wiley & Sons, Inc.

Rasmussen, C.E., and Williams, C.K.I. (2006). Gaussian Processes for Machine Learning. MITPress, Cambridge, MA.

Reilly, C., Gelman, A. and Katz, J. (2001). Poststratication without population level information on the poststratifying variable, with application to political polling. Journal of the American Statistical Association, 96, 1-11.

Royall, R.M. (1968). An old approach to finite population sampling theory. Journal of the American Statistical Association, 63, 1269-1279.

Rubin, D.B. (1976). Inference and missing data (with discussion). Biometrika, 63, 581-592.

Rubin, D.B. (1983). Comment on “An evaluation of model-dependent and probability-sampling inferences in sample surveys”, by M.H. Hansen, W.G. Madow and B.J. Tepping. Journal of the American Statistical Association, 78, 803-805.

Särndal, C.-E., Swensson, B. and Wretman, J.H. (1992). Model Assisted Survey Sampling. New York: Springer.

Si, Y., and Gelman, A. (2014). Survey weighting for New York Longitudinal Survey on Poverty Measure. Technical report, Columbia University.

Si, Y., and Zhou, P. (2020). Bayes-raking: Bayesian finite population inference with known margins. Journal of Survey Statistics and Methodology, smaa008.

Si, Y., Pillai, N.S. and Gelman, A. (2015). Nonparametric Bayesian weighted sampling inference. Bayesian Analysis, 10, 605-625.

Si, Y., Trangucci, R. and Gabry, J.S. (2020). Computation codes for manuscript “Bayesian hierarchical weighting adjustment and survey inference”. https://github.com/yajuansisophie/weighting.

Stan Development Team (2017). Stan modeling language user’s guide and reference manual. http://mc-stan.org.

Stan Development Team (2018). Stan: A C++ library for probability and sampling. http://mcstan.org.

Tang, X., Ghosh, M., Ha, N.S. and Sedransk, J. (2018). Modeling random effects using global-local shrinkage priors in small area estimation. Journal of the American Statistical Association, 0(0),1-14.

Valliant, R., Dever, J.A. and Kreuter, F. (2018). Practical Tools for Designing and Weighting Survey Samples, 2ndEdition. New York: Springer.

Valliant, R., Dorfman, A. and Royall, R. (2000). Finite Population Sampling and Inference. New York: John Wiley & Sons, Inc.

Volfovsky, A., and Hoff, P. (2014). Hierarchical array priors for ANOVA decompositions of cross-classified data. Annals of Applied Statistics, 8(1), 19-47.

Wang, W., Rothschild, D., Goel, S. and Gelman, A. (2015). Forecasting elections with nonrepresentative polls. International Journal of Forecasting, 31(3), 980-991.

Wimer, C., Garfinkel, I., Gelblum, M., Lasala, N., Phillips, S., Si, Y., Teitler, J. and Waldfogel, J. (2014). Poverty tracker − Monitoring poverty and well-being in NYC. Columbia Population Research Center and Robin Hood Foundation.

Wu, C., and Sitter, R.R. (2001). A model-calibration approach to using complete auxiliary information from survey data. Journal of the American Statistical Association, 96(453), 185-193.

Xia, X., and Elliott, M.R. (2016). Weight smoothing for generalized linear models using a Laplace prior. Journal of Official Statistics, 32(2), 507-539.

Yougov (2017). Introducing the Yougov referendum model. https://yougov.co.uk.

Yuan, M., and Lin, Y. (2006). Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society, Series B, 68, 49-67.

Zhang, X., Holt, J.B., Yun, S., Lu, H., Greenlund, K.J. and Croft, J.B. (2015). Validation of multilevel regression and poststratification methodology for small area estimation of health indicators from the behavioral risk factor surveillance system. American Journal of Epidemiology, 182(2), 127-137.

More information

ISSN : 1492-0921

Editorial policy

Survey Methodology publishes articles dealing with various aspects of statistical development relevant to a statistical agency, such as design issues in the context of practical constraints, use of different data sources and collection techniques, total survey error, survey evaluation, research in survey methodology, time series analysis, seasonal adjustment, demographic studies, data integration, estimation and data analysis methods, and general survey systems development. The emphasis is placed on the development and evaluation of specific methodologies as applied to data collection or the data themselves. All papers will be refereed. However, the authors retain full responsibility for the contents of their papers and opinions expressed are not necessarily those of the Editorial Board or of Statistics Canada.

Submission of Manuscripts

Survey Methodology is published twice a year in electronic format. Authors are invited to submit their articles in English or French in electronic form, preferably in Word to the Editor, (statcan.smj-rte.statcan@canada.ca, Statistics Canada, 150 Tunney’s Pasture Driveway, Ottawa, Ontario, Canada, K1A 0T6). For formatting instructions, please see the guidelines provided in the journal and on the web site (www.statcan.gc.ca/SurveyMethodology).

Note of appreciation

Canada owes the success of its statistical system to a long-standing partnership between Statistics Canada, the citizens of Canada, its businesses, governments and other institutions. Accurate and timely statistical information could not be produced without their continued co-operation and goodwill.

Standards of service to the public

Statistics Canada is committed to serving its clients in a prompt, reliable and courteous manner. To this end, the Agency has developed standards of service which its employees observe in serving its clients.

Copyright

Published by authority of the Minister responsible for Statistics Canada.

© Her Majesty the Queen in Right of Canada as represented by the Minister of Industry, 2020

Use of this publication is governed by the Statistics Canada Open Licence Agreement.

Catalogue No. 12-001-X

Frequency: Semi-annual

Ottawa

close


Table of contents

Abstract - Article main page
Section 1. Introduction
Section 2. Motivating application
Section 3. Method
Section 4. Simulation studies
Section 5. Application to longitudinal study of wellbeing
Section 6. Discussion
Acknowledgements
Appendix
References

close
Date modified: