Tests for evaluating nonresponse bias in surveys Section 2. PoststratificationTests for evaluating nonresponse bias in surveys Section 2. Poststratification

2.1 Parameter and linearization variance

Suppose the finite population $U$ has $H$ strata, with $N_{h}$ primary sampling units (PSUs) in stratum $h,$ $M_{h i}$ units in PSU $i$ of stratum $h,$ and $M = \sum_{h i} M_{h i}$ units in total. Let $y_{h i k}$ denote the quantity of interest for unit $k$ in PSU $(h i) .$ A probability sample $S$ is taken from the population, with $n_{h}$ PSUs selected from stratum $h$ and $n = \sum_{h =1}^{H} n_{h} .$ The sample of PSUs from stratum $h$ is denoted by $S_{h},$ and the sample of units from PSU $(h i)$ is denoted by $S_{h i} .$ Each unit has a design weight $w_{h i k} = 1 / P (unit h i k \in S),$ and the PSU-level design weight is $w_{h i} = 1 / P (PSU h i \in S_{h}) .$

Two frameworks are commonly used for the nonresponse mechanism. In a two-phase “forward” framework, the sample is selected at phase 1 and the nonresponse mechanism is a second phase of selection (Oh and Scheuren 1987; Särndal and Lundström 2005). Fay (1991) proposed a “reverse framework” which was studied further by Shao and Steel (1999) and Haziza, Thompson, and Yung (2010). In this framework, the nonresponse mechanism is applied to the finite population first, and then the sample is selected. The reverse framework, which we follow in this paper, specifies a nonresponse mechanism for nonsampled as well as sampled units. We assume that every unit in the population has a value of the response indicator $r_{h i k} .$ Let $R_{h i k} = E [r_{h i k}]$ under the response mechanism in the finite population, so that $R_{h i k}$ is the value of the true response propensity for unit $(h i k)$ in the population.

Suppose the characteristic $y$ is known for all units in the selected sample. We compare the estimated population total using everyone in the sample with the estimated total using the poststratification-weighted respondents. There are $C$ poststrata and poststratum $c$ has $M_{c}$ population units with $M = \sum_{c =1}^{C} M_{c} .$ The poststratum counts $M_{c}$ may be obtained from the sampling frame if the poststratification variables are known for every unit in the frame. Often, however, the poststratum counts come from an external source such as a census. Let $δ_{c h i k} =1$ if unit $(h i k)$ is in poststratum $c$ and 0 otherwise. The population response rate in poststratum $c$ is $p_{c} = \sum_{h i k \in U} R_{h i k} δ_{c h i k} / M_{c} .$ Yung and Rao (2000) assumed that the response rate $p_{c}$ was the same for each poststratum. In many applications, however, the poststrata are formed so that response propensities within each poststratum are homogeneous, but the poststrata themselves have different mean response propensities. We therefore allow $p_{c}$ to differ among the poststrata.

If $y$ is known for all members of the selected sample, then the estimator of the population total using the sample is

${\hat{Y}}_{S S} = \sum_{h i k \in S} w_{h i k} y_{h i k} = \sum_{h i k \in U} Z_{h i k} w_{h i k} y_{h i k}, (2.1)$

where $w_{h i k}$ is the design weight for unit $k$ of PSU $i$ in stratum $h$ and $Z_{h i k}$ is the indicator variable for sample inclusion. Using the respondents only, the poststratified estimator of the population total is

${\hat{Y}}_{P S} = \sum_{c =1}^{C} M_{c} \frac{\sum_{h i k \in S} w_{h i k} r_{h i k} δ_{c h i k} y_{h i k}}{\sum_{h i k \in S} w_{h i k} r_{h i k} δ_{c h i k}} = \sum_{c =1}^{C} M_{c} \frac{{\hat{Y}}_{c}^{R}}{{\hat{M}}_{c}^{R}} . (2.2)$

We define the finite population parameter of interest to be the difference between the expected value of ${\hat{Y}}_{P S}$ and the expected value of ${\hat{Y}}_{S S},$ which will be 0 if there is no nonresponse bias after poststratification. Define

$\begin{array}{l} M_{c}^{R} & = \sum_{h i k \in U} δ_{c h i k} R_{h i k} = p_{c} M_{c}, \\ Y_{c}^{R} & = \sum_{h i k \in U} δ_{c h i k} R_{h i k} y_{h i k}, \end{array}$

and

$θ = \sum_{c =1}^{C} M_{c} \frac{Y_{c}^{R}}{M_{c}^{R}} - Y = \sum_{c =1}^{C} \frac{Y_{c}^{R}}{p_{c}} - Y . (2.3)$

Using the relation $\sum_{h i k \in U} δ_{c h i k} (R_{h i k} - p_{c}) =0,$

$\begin{array}{l} θ & = \sum_{c =1}^{C} \sum_{h i k \in U} y_{h i k} δ_{c h i k} (\frac{R_{h i k}}{p_{c}} - 1) \\ = \sum_{c =1}^{C} \sum_{h i k \in U} δ_{c h i k} (\frac{R_{h i k}}{p_{c}} - 1) (y_{h i k} - \frac{Y_{c}^{R}}{M_{c}^{R}}) . \end{array}$

We are interested in testing the hypothesis $H_{0} : θ =0$ vs. $H_{A} : θ \neq 0,$ or alternatively in obtaining a confidence interval for $θ .$ If the response propensity in each poststratum $c$ is uniform with $R_{h i k} = p_{c}$ for all units having $δ_{c h i k} =1,$ then $θ$ will be zero. Alternatively, $θ =0$ if there is no variability in the response variable $y_{h i k}$ within each poststratum. If either of these conditions holds, poststratification corrects for bias from nonresponse. Note that if each of the poststrata has uniform response propensity $-$ that is, the poststratification variables completely explain the variability in underlying response propensities $-$ then the poststratification will in fact remove bias for every possible $y$ variable. If the variance of $y_{h i k}$ is 0 within each poststratum, poststratification removes bias for $y$ but it does not necessarily remove bias for other variables.

We estimate $θ$ by $\hat{θ} = {\hat{Y}}_{P S} - {\hat{Y}}_{S S},$ which may be rewritten as

$\hat{θ} = {\hat{Y}}_{P S} - {\hat{Y}}_{S S} = \sum_{c =1}^{C} \frac{1}{p_{c}} ({\hat{Y}}_{c}^{R} - {\bar{Y}}_{c}^{R} ({\hat{M}}_{c}^{R} - M_{c}^{R}) + {\hat{T}}_{c}) - {\hat{Y}}_{S S}, (2.4)$

where ${\bar{Y}}_{c}^{R} = Y_{c}^{R} / M_{c}^{R},$ ${\bar{y}}_{c}^{R} = {\hat{Y}}_{c}^{R} / {\hat{M}}_{c}^{R},$ and

${\hat{T}}_{c} = - ({\bar{y}}_{c}^{R} - {\bar{Y}}_{c}^{R}) ({\hat{M}}_{c}^{R} - M_{c}^{R}) . (2.5)$

Theorem 1 gives the variance of $\hat{θ} .$ Define

$e_{R h i k} = \sum_{c =1}^{C} δ_{c h i k} {\frac{R_{h i k}}{p_{c}} (y_{h i k} - {\bar{Y}}_{c}^{R}) - y_{h i k}} .$

We assume the following regularity conditions.

(A1) The number of poststrata, $C,$ is fixed and $M_{c} / M \to λ_{c} \in (0,1) .$
(A2) There exists a constant $K$ such that $| y_{h i k} | < K$ for all $(h i k) .$
(A3) $\max_{h i k} w_{h i k} = O (M / n)$ and $\max_{h i k} w_{h i k} / w_{h i}$ is bounded.
(A4) $R_{h i k} > ε$ for all $(h i k),$ for a fixed $ε >0.$ This guarantees that every unit has a positive response propensity that is bounded away from 0.
(A5) The vector of response indicators $r = [r_{h i k}]$ is independent of the vector of sample inclusion indicators $Z = [Z_{h i k}] .$ In addition, $r_{h i k}$ and $r_{l j p}$ are independent when $(h i) \neq (l j),$ so that the response indicators in different PSUs are uncorrelated.

Assumptions (A1) and (A4) ensure that the denominator in (2.3) is nonzero almost surely. Assumption (A2) could be replaced by weaker Liapunov-type conditions such as those in Theorem 1.3.2 of Fuller (2009) or Yung and Rao (2000) if more restrictive assumptions are placed on the covariance structure of the response indicators $r_{h i k};$ however, in practice it can be assumed that almost any characteristic measured in a finite population is bounded. Assumption (A5) is weaker than the assumption used in Kim and Kim (2007) that the response indicators are independent across units. With assumption (A5), individuals in the same PSU (for example, persons in the same household or same city) may exhibit dependence when choosing whether to respond to a survey, but the response indicators of individuals in different PSUs are independent.

Theorem 1. Under conditions (A1) $-$ (A5), the variance of $\hat{θ}$ is

$V (\hat{θ}) = V_{1} (\hat{θ}) + V_{2} (\hat{θ}),$

where

$V_{1} (\hat{θ}) = V (\sum_{h i k \in U} Z_{h i k} w_{h i k} e_{R h i k}) + E [V [\sum_{h i k \in U} Z_{h i k} w_{h i k} \sum_{c =1}^{C} δ_{c h i k} \frac{r_{h i k}}{p_{c}} (y_{h i k} - {\bar{Y}}_{c}^{R}) | Z]] (2.6)$

and

$V_{2} (\hat{θ}) = V [\sum_{c =1}^{C} \frac{{\hat{T}}_{c}}{p_{c}}] + 2 Cov [\sum_{c =1}^{C} \frac{{\hat{T}}_{c}}{p_{c}}, \sum_{c =1}^{C} \frac{({\bar{y}}_{c}^{R} - {\bar{Y}}_{c}^{R}) {\hat{M}}_{c}^{R}}{p_{c}} - {\hat{Y}}_{S S}] = o (M^{2} / n) .$

The proof is given in the appendix. Usually, only $V_{1} (\hat{θ})$ would be considered because for most applications it has higher order than $V_{2} (\hat{θ}) .$ Unlike situations typically studied in survey sampling, however, the first-order term of the linearization variance can be zero for some situations, and in those cases $V (\hat{θ}) = V_{2} (\hat{θ}) .$ If the first-order term is not exactly zero but has order $o (M^{2} / n),$ both terms of the variance are needed.

The second term in (2.6) equals 0 if $p_{c} =1$ for all poststrata $c$ (that is, there is full response), or if there is no variability among the $y$ values within poststratum $c$ for each poststratum with $p_{c} <1.$ If the response indicators $r_{h i k}$ are all independent, then

$E [V (\sum_{h i k \in U} Z_{h i k} w_{h i k} \sum_{c =1}^{C} δ_{c h i k} \frac{r_{h i k}}{p_{c}} (y_{h i k} - {\bar{Y}}_{c}^{R}) | Z)] = \sum_{h i k \in U} w_{h i k} \sum_{c =1}^{C} δ_{c h i k} \frac{1 - p_{c}}{p_{c}} {(y_{h i k} - {\bar{Y}}_{c}^{R})}^{2} .$

Under the hypothesized uniform response propensity mechanism that $R_{h i k} = p_{c}$ for all population units in poststratum $c,$ the first term in (2.6) is

$V (\sum_{h i k \in U} Z_{h i k} w_{h i k} e_{R h i k}) = V {\sum_{h i k \in U} Z_{h i k} w_{h i k} \sum_{c =1}^{C} δ_{c h i k} (- {\bar{Y}}_{c}^{R})} = V (\sum_{c =1}^{C} {\hat{M}}_{c} {\bar{Y}}_{c}^{R}) .$

If response propensities are uniform, this term equals zero if the population mean of ${\bar{Y}}_{c}^{R}$ is the same for all poststrata and the estimated poststratum sizes sum to $M .$

If $(n / M^{2}) V_{1} (\hat{θ})$ converges to a positive constant, a linearization variance estimator for $V (\hat{θ})$ is

${\hat{V}}_{L} (\hat{θ}) = \sum_{h =1}^{H} \frac{n_{h}}{n_{h} - 1} \sum_{i \in S_{h}} {(b_{h i} - b_{h})}^{2} (2.7)$

where

$b_{h i} = \sum_{k \in S_{h i}} w_{h i k} {\sum_{c =1}^{C} \frac{M_{c}}{{\hat{M}}_{c}^{R}} r_{h i k} δ_{c h i k} (y_{h i k} - {\bar{y}}_{c}^{R}) - y_{h i k}}$

and

$b_{h} = \frac{1}{n_{h}} \sum_{i \in S_{h}} b_{h i} .$

Theorem 2. Suppose conditions (A1) $-$ (A5) hold and that $(n / M^{2}) V_{1} (\hat{θ})$ converges to a positive constant. Then $(n / M^{2}) [{\hat{V}}_{L} (\hat{θ}) - V_{1} (\hat{θ})] \to 0$ in probability.

Theorem 2 is proven in the Appendix.

2.2 Higher-order terms of the variance

When $V_{1} (\hat{θ}) = o (M^{2} / n),$ the higher-order terms of the variance are needed. Theorem 3 gives these higher-order terms for the special case of simple random sampling. For simple random sampling, each unit is denoted by the subscript $i$ instead of $h i k .$

Theorem 3. Suppose conditions (A1) $-$ (A5) are met, and that a simple random sample of $n$ units is selected from the population of $M$ units, where $n / M \to 0.$ Let ${\hat{Y}}_{c}^{N R} = \sum_{i \in S} w_{i} δ_{c i} y_{i} (1 - r_{i})$ be the estimated total for the nonrespondents in poststratum $c .$ Assume that ${\bar{y}}_{c}^{R}$ is independent of ${\hat{M}}_{c}^{R}$ and ${\hat{Y}}_{c}^{N R},$ and that all $r_{i}$ are independent and are independent of $Z_{i} .$ Then

$V_{2} (\hat{θ}) = \sum_{c =1}^{C} \frac{2 p_{c} - 1}{p_{c}^{2}} V [{\bar{y}}_{c}^{R} - {\bar{Y}}_{c}^{R}] V [{\hat{M}}_{c}^{R} - M_{c}^{R}] + o (M^{2} / n^{2}) .$

We can estimate $V_{2} (\hat{θ})$ in a simple random sample by

$\sum_{c =1}^{C} \frac{2 {\hat{p}}_{c} - 1}{{\hat{p}}_{c}^{2}} \frac{s_{c}^{2}}{n_{c}^{R}} \frac{M_{c} {\hat{p}}_{c} (M - M_{c} {\hat{p}}_{c})}{n},$

where ${\hat{p}}_{c}$ is the empirical response rate in poststratum $c,$ $n_{c}^{R}$ is the number of respondents in poststratum $c,$ and $s_{c}^{2}$ is the sample variance of $y$ for the respondents in poststratum $c .$

In practice, the estimated first-order term of the variance using (2.7) will in general be nonzero even when $V_{1} (\hat{θ}) =0.$ Thus, the estimated first-order term cannot be used to diagnose whether the higher-order terms are needed. However, the variance expression in (2.6) implies that $V_{1} (\hat{θ})$ is sufficiently large for the first-order approximation to be valid when all poststrata have response rates bounded away from one and non-negligible within-poststratum variance.

2.3 Jackknife

The jackknife estimator of the variance is defined as follows:

${\hat{V}}_{J} (\hat{θ}) = \sum_{g =1}^{H} \frac{n_{g} - 1}{n_{g}} \sum_{j \in S_{g}} {({\hat{θ}}^{(g j)} - \hat{θ})}^{2}, (2.8)$

where

$\begin{array}{l} {\hat{θ}}^{(g j)} & = {\hat{Y}}_{P S}^{(g j)} - {\hat{Y}}_{S S}^{(g j)}, \\ {\hat{Y}}_{P S}^{(g j)} & = \sum_{c =1}^{C} M_{c} \frac{\sum_{h i k \in S} w_{h i k}^{(g j)} r_{h i k} δ_{c h i k} y_{h i k}}{\sum_{h i k \in S} w_{h i k}^{(g j)} r_{h i k} δ_{c h i k}}, \\ {\hat{Y}}_{S S}^{(g j)} & = \sum_{h i k \in S} w_{h i k}^{(g j)} y_{h i k}, \end{array}$

and the jackknife weights are:

$w_{h i k}^{(g j)} = {\begin{array}{l} 0 & if & (h i) = (g j) \\ \frac{n_{h}}{n_{h} - 1} w_{h i k} & if & h = g, i \neq j \\ w_{h i k} & if & h \neq g \end{array} . (2.9)$

If $(n / M^{2}) V_{1} (\hat{θ})$ converges to a positive constant and assumptions (A1) $-$ (A5) hold, then ${\hat{V}}_{J} (\hat{θ}) / V_{1} (\hat{θ})$ converges to 1 in probability. This follows by standard jackknife arguments (Theorem 6.1 of Shao and Tu 1995) because the population parameter is a continuously differentiable function of population totals. Under the conditions of Theorem 2, either $\hat{θ} / \sqrt{{\hat{V}}_{L} (\hat{θ})}$ or $\hat{θ} / \sqrt{{\hat{V}}_{J} (\hat{θ})}$ may be used as a test statistic. Each approximately follows a standard normal distribution when the null hypothesis $H_{0} : θ =0$ is true.

2.4 Remarks and extensions

In this section we derived the linearization variance estimator for comparing the estimated population total of a quantity known for everyone in the selected sample with the poststratified estimate calculated using the respondents only. Theorems 1 and 2 also give the variance and variance estimator for comparing the estimator calculated using the selected sample with that from the base-weighted respondents. In that case, ${\hat{Y}}_{P S}$ reduces to an estimator with one poststratum, ${\hat{Y}}_{P S} = (M / {\hat{M}}^{R}) {\hat{Y}}^{R},$ where ${\hat{M}}^{R} = \sum_{(h i k) \in S} w_{h i k} r_{h i k} .$

What happens if $y$ is one of the poststratification variables? In the framework used in this section, the population counts for the poststratification variables are obtained from the sampling frame or an external source. If $y$ is a linear combination of poststratification class indicators, then ${\hat{Y}}_{P S}$ is the same for all possible samples and thus has zero variance. Then $V (\hat{θ}) = V ({\hat{Y}}_{S S}),$ which is the first-order term of the variance in Theorem 1. If $y$ is also a stratification variable in the design, then $V (\hat{θ})$ will be zero. If $y$ is not a stratification variable, then typically ${\hat{Y}}_{S S}$ will vary from sample to sample and will have variance of order $O (M^{2} / n)$ so that the test of nonresponse bias can be performed. We would expect the rejection rate for the test to be the significance level $α$ in this case.

The parameter $θ$ in (2.3) was defined as the difference between the poststratified population total, calculated using the population response propensities under the poststratification scheme adopted, and the unadjusted population total. In (2.4), the unadjusted population total $Y$ was estimated by the Horvitz-Thompson estimator. The parameter $θ$ could alternatively be estimated by

${\hat{θ}}_{2} = {\hat{Y}}_{P S} - \sum_{c =1}^{C} M_{c} \frac{{\hat{Y}}_{c}}{{\hat{M}}_{c}},$

in which a poststratified estimator is used instead of ${\hat{Y}}_{S S} .$ The variance of ${\hat{θ}}_{2}$ is expected to be less than the variance of $\hat{θ}$ under the poststratification assumptions, resulting in a more powerful test. However, when $y$ is a linear combination of the poststratum indicators, the statistic ${\hat{θ}}_{2}$ cannot be used to test $H_{0} : θ =0$ because $V ({\hat{θ}}_{2}) =0.$ A similar problem can occur when $y$ is highly correlated with the poststratification variables. The estimator $\hat{θ},$ by contrast, typically has positive variance even when $y$ is one of the poststratification variables.

Sometimes poststratification is performed using less-than-perfect poststratification totals $-$ for example, the totals may come from a large survey such as the American Community Survey which has its own sampling and nonsampling errors, or they may be from a census of a slightly different population. In some cases, poststratification variables such as race or ethnicity may be measured differently in the survey than in the source of the external population totals. Using $\hat{θ}$ rather than ${\hat{θ}}_{2}$ may detect differences that might be caused by a flawed poststratification.

If desired, the tests may be performed using means rather than totals. In this case, the population parameter is

$θ_{M} = \sum_{c =1}^{C} \frac{M_{c}}{M} \frac{Y_{c}^{R}}{M_{c}^{R}} - \bar{Y}$

where $\bar{Y} = Y / M,$ and may be estimated by

${\hat{θ}}_{M} = \sum_{c =1}^{C} \frac{M_{c}}{M} \frac{{\hat{Y}}_{c}^{R}}{{\hat{M}}_{c}^{R}} - \frac{{\hat{Y}}_{S S}}{\sum_{h i k \in S} w_{h i k}} . (2.10)$

ISSN : 1492-0921

Editorial policy

Survey Methodology publishes articles dealing with various aspects of statistical development relevant to a statistical agency, such as design issues in the context of practical constraints, use of different data sources and collection techniques, total survey error, survey evaluation, research in survey methodology, time series analysis, seasonal adjustment, demographic studies, data integration, estimation and data analysis methods, and general survey systems development. The emphasis is placed on the development and evaluation of specific methodologies as applied to data collection or the data themselves. All papers will be refereed. However, the authors retain full responsibility for the contents of their papers and opinions expressed are not necessarily those of the Editorial Board or of Statistics Canada.

Submission of Manuscripts

Survey Methodology is published twice a year in electronic format. Authors are invited to submit their articles in English or French in electronic form, preferably in Word to the Editor, (statcan.smj-rte.statcan@canada.ca, Statistics Canada, 150 Tunney’s Pasture Driveway, Ottawa, Ontario, Canada, K1A 0T6). For formatting instructions, please see the guidelines provided in the journal and on the web site (www.statcan.gc.ca/SurveyMethodology).

Note of appreciation

Canada owes the success of its statistical system to a long-standing partnership between Statistics Canada, the citizens of Canada, its businesses, governments and other institutions. Accurate and timely statistical information could not be produced without their continued co-operation and goodwill.

Standards of service to the public

Statistics Canada is committed to serving its clients in a prompt, reliable and courteous manner. To this end, the Agency has developed standards of service which its employees observe in serving its clients.

Copyright

Published by authority of the Minister responsible for Statistics Canada.

Use of this publication is governed by the Statistics Canada Open Licence Agreement.

Catalogue No. 12-001-X

Frequency: semi-annual

Ottawa

Date modified:: 2016-12-20

Language selection

Search and menus

Search

Tests for evaluating nonresponse bias in surveys Section 2. PoststratificationTests for evaluating nonresponse bias in surveys Section 2. Poststratification

2.1 Parameter and linearization variance

2.2 Higher-order terms of the variance

2.3 Jackknife

2.4 Remarks and extensions