Bayes, buttressed by design-based ideas, is the best overarching paradigm for sample survey inference
Section 3. Design-based versus model-based inference

• Previous
• Next

The survey sampling literature features many lively controversies (e.g., Smith, 1976, 1994; Kish, 1995; Brewer, 2013; Little, 2014) between “design-based” inference, where inference is based on the sampling distribution (2.2) and “model-based” inference, where inference is based on model distribution $f_{Y|Z}\left(Y|Z,\theta \right)$ if selection is ignorable, or on the full model distribution (2.1) or (2.3) if selection is nonignorable or there is nonresponse. I seek inferences that are both design-based and model-based, in that they are based on Bayesian models but have good design-based properties.

Rubin (1984) distinguished between statistical inference for a particular data set, and the properties of that inference $–$ consistency, confidence coverage $–$ in repeated sampling. To be broadly credible, the inference should also have good repeated-sampling properties. This goal of the “design-based” approach should also be a goal of Bayesian models for surveys $–$ the model and prior should be chosen to yield inferences with good design-based properties. To achieve this, features of the complex probability sample design need to be part of the model $–$ stratification and weighting incorporated via covariates, multistage sampling incorporated via hierarchical models. The inclusion of a prior distribution in Bayesian modeling, decried by some as yet another assumption to be added to the model, for me provides an additional tool over superpopulation modeling. It provides more flexibility than superpopulation modeling, which effectively restricts the choice to uniform priors.

In addition to having good frequentist properties, the inference based on $q$ and $I$ needs to be appropriate $–$ Rubin (2019) uses the term “relevant” $–$ for the realized data set. Let $D$ denote the data that are the basis for the inference, and $\tilde{D}$ the particular realization of $D,$ the sample and respondent values actually obtained. Whether $q$ and $I$ are derived from a formal Bayesian model, an estimating equation, or some algorithmic procedure, they should provide good inference for the data $\tilde{D},$ not other data sets $D$ that may have been obtained. Bayesian methods tend to have this property, because the posterior distribution conditions on $\tilde{D};$ but a confidence interval should also be approximately valid when viewed as a credible interval that conditions on $\tilde{D},$ if only because this is how a non-statistician tends to interpret it. Design-based confidence intervals can be lacking from this perspective, as illustrated in examples 2 and 4 below.

To summarize, a common goal of design-based and model-based inference is to arrive at a value of $q$ that makes efficient use of the data, has some property like design consistency which implies that it is not too far from $Q,$ and an interval $I$ that is as narrow as the information in the data allows, while including $Q$ with a probability close to the nominal 95% value. Rubin (2019) associates these properties with a “sage” statistician in his Waksberg lecture.

Design-based methods are often rationalized as avoiding the need for a model, because properties like design consistency are not based on a model for the data. However, the performance of design-based methods often depends on an implicit model, and modifying the estimate based on a more realistic model can improve the inference, from a design-based or model-based perspective. This point is illustrated in examples 3-5 below.

The question of the appropriate reference set for repeated sampling properties like confidence intervals is fraught with difficulties, specifically on whether to condition on ancillary statistics or on statistics that are close to ancillary (e.g., Birnbaum, 1962; Berger and Wolpert, 1988; Ghosh, Reid and Fraser, 2010). These questions also arise when assessing the repeated-sampling properties of a Bayesian inference, but do not apply to the inference itself because the posterior distribution conditions on $\tilde{D}.$

The design-based approach to sample survey inference is too limited in scope, failing to address adequately many of the problems of sample surveys in practice. Limitations include the following:

1. Design-based inference is asymptotic, and does not provide valid inferences in small samples. Consider the following simple example.

Example 1. Inference about a population mean from a simple random sample. Consider inference about a population mean of a variable $Y$ from a simple random sample of size $n$ from a population of size $N.$ The standard design-based 95% confidence interval takes the form

$$\overline{y}\pm 1.96s\sqrt{\left(1/n-1/N\right)},(3.1)$$

where $\overline{y}$ is the sample mean and $s$ is the sample standard deviation. This interval is asymptotic and does not provide valid small sample inferences. In particular, if $Y$ is continuous, better inference is usually obtained by replacing 1.96, the 97.5^th percentile of the normal distribution, by the 97.5^th percentile of a distribution that reflects uncertainty about estimating the variance, such as the t distribution with $n-1$ degrees of freedom. However, that procedure assumes a normal distribution for $Y$ and hence is not design-based. If $Y$ is binary with values 0 and 1, then $\overline{y}$ is the sample proportion, $s=\sqrt{\overline{y}\left(1-\overline{y}\right)}$ and (3.1) is the asymptotic Wald interval, which performs very poorly in small samples, particularly when the true proportion is near to 0 or 1. The Bayesian credible interval for a Jeffreys or uniform prior has much better frequentist properties. See Dean and Pagano (2015) and Franco, Little, Louis and Slud (2019) for comparisons of Wald intervals with alternatives for complex designs. Design-based inference is often a poor option for small samples, and in particular for small area estimation, where a model for $Y$ is invoked to “borrow strength” across areas.

2. Design-based inference does not handle survey unit or item nonresponse or response errors, because these problems require models to yield generally satisfactory results.
3. Design-based inference is not prescriptive, in a sense of prescribing the appropriate choice of inference method for the data at hand. The appropriate choice of estimator effectively requires an implicit model, as in “model-assisted” estimation (e.g., Särndal, Swensson and Wretman, 1992). For example, the regression or ratio estimator for incorporating auxiliary information, or the Horvitz-Thompson or Hájek estimator for incorporating survey weights, are all based on implicit models, and if that model is far from realistic these methods may be severely suboptimal $–$ Basu’s (1971) elephants being an extreme and satirical example. Bayesian inference based on more flexible models tend to do better, as discussed in example 5 below.
4. Design-based inference does not address how to provide inferences for non-probability samples, which are increasingly prevalent given the expense and difficulty of obtaining true random samples.

Point 4 does not rule out the use of design-based methods for deriving $q$ and $I,$ because we can always pretend that we have a probability sample, by assuming a model for the selection indicators $S$ that describe inclusion in the sample, and estimating the unknown parameters in that model (e.g., Elliott and Valliant, 2017). Statisticians who use design-based methods for inference from random samples tend to favor this “quasi-randomization” approach. However, it shares the limitations of the design-based approach for probability samples, namely the inability to handle small samples, missing data or response errors; the Bayesian toolkit for these problems is much more extensive.

• Previous
• Article main page
• Next