Variance estimation under monotone non-response for a panel survey
Section 2. Correction of non-response and attrition

• Previous
• Next

2.1  Notation and main assumptions

We are interested in a finite population $U.$ A sample $s_0$ is first selected according to some sampling design $p(\cdot ),$ and we assume that the first-order inclusion probabilities ${\pi }_i$ are strictly positive for any $i\in U.$ This first sampling phase corresponds to the original inclusion of units in the sample.

We consider the case of a panel survey in which the sole units in the original sample $s_0$ are followed over time, without reentry or late entry units at subsequent times to represent possible newborns. We are therefore interested in estimating some parameter defined over the population $U,$ for some study variable $y_t$ taking the value $y_{it}$ for the unit $i$ at time $t.$ The units in the sample $s_0$ are followed at subsequent times $\delta \mathrm{<mo>=</mo>\; 1,}\dots \mathrm{,}t,$ and the sample is prone to unit non-response at each time. We note $r_i^{\delta }$ for the response indicator for unit $i$ at time $\delta ,$ and $s_{\delta }$ for the subset of respondents at time $\delta .$

We assume monotone non-response resulting in the nested sequence $s_0\supset s_1\supset \mathrm{...}\supset s_t.$ For $\delta \mathrm{<mo>=</mo>\; 1,}\dots \mathrm{,}t,$ we note $p_i^{\delta }\mathrm{<mo>=</mo>}\text{Pr}\left(i\in s_{\delta }|s_{\delta -1}\right)$ for the response probability of some unit $i$ to be a respondent at time $\delta .$ We assume that the data are missing at random, i.e. the response probability $p_i^{\delta }$ at time $\delta$ can be explained by the variables observed at times $\mathrm{0,}\dots \mathrm{,}\delta -1,$ including the variables of interest, see for example Zhou and Kim (2012). Also, we assume that at any time $\delta$ the units answer independently of one another, and we note $p_{ij}^{\delta }\mathrm{<mo>=</mo>}{p}_i^{\delta }{p}_j^{\delta }$ for the probability that two distinct units $i$ and $j$ answer jointly at time $\delta .$

2.2  Reweighted estimator

We are interested in estimating the total $Y\left(t\right)\mathrm{<mo>=</mo>}{\displaystyle {\sum }_{i\in U}}{y}_{it}$ at time $t.$ In practice, the response probabilities at each time are unknown and need to be estimated. We assume that at each time $\delta$ the probability of response is parametrically modeled as

$$p_i^{\delta }=f^{\delta }\left(z_i^{\delta }\text{}\mathrm{,}{\alpha }^{\delta }\right)(2.1)$$

for some known function $f^{\delta }\left(\cdot \mathrm{,}\cdot \right),$ where $z_i^{\delta }$ is a vector of variables observed for all the units in $s_{\delta -1},$ and ${\alpha }^{\delta }$ denotes some unknown parameter. Here and elsewhere, the superscript $\delta$ will be used when we account for non-response at time $\delta ,$ like for the probability $p_i^{\delta }$ of unit $i$ to be a respondent at time $\delta .$ Following the approach in Kim and Kim (2007), we assume that the true parameter is estimated by ${\widehat{\alpha }}^{\delta }\text{},$ the solution of the estimating equation

$$\frac{\partial }{\partial \alpha }{\displaystyle \sum _{i\in s_{\delta -1}}}{k}_i^{\delta }\left\{r_i^{\delta }\ln \left(p_i^{\delta }\right)+\left(1-r_i^{\delta }\right)\ln \left(1-p_i^{\delta }\right)\right\}=\mathrm{0,}(2.2)$$

with $k_i^{\delta }$ some weight of unit $i$ in the estimating equation. Customary choices for these weights include $k_i^{\delta }\mathrm{<mo>=</mo>\; 1}$ and $k_i^{\delta }\mathrm{<mo>=</mo>}{\pi }_i^{-1}\text{},$ see Fuller and An (1998), Beaumont (2005) and Kim and Kim (2007).

The estimated response probability at time $\delta$ is ${\widehat{p}}_i^{\delta }\mathrm{<mo>=</mo>}{f}^{\delta }\left(z_i^{\delta }\text{}\mathrm{,}{\widehat{\alpha }}^{\delta }\right).$ The propensity score adjusted estimator at time $t,$ which will be simply called the reweighted estimator in what follows, is defined as

$${\widehat{Y}}_t\left(t\right)\mathrm{<mo>=</mo>}{\displaystyle \sum _{i\in s_t}}\frac{y_{it}}{{\pi }_i{\widehat{p}}_i^{1\to t}}\text{with}{\widehat{p}}_i^{1\to t}\mathrm{<mo>=</mo>}{\displaystyle \prod _{\delta \mathrm{<mo>=</mo>\; 1}}^t}{\widehat{p}}_i^{\delta }\mathrm{.}(2.3)$$

Here and elsewhere, the subscript $t$ will be used when the sample observed at time $t$ is used for estimation, like for ${\widehat{Y}}_t(\cdot )$ which makes use of the sample $s_t.$ We simplify the notation as ${\widehat{Y}}_t\left(t\right)\equiv {\widehat{Y}}_t$ when the total at time $t$ is estimated by using the sample observed at time $t.$

2.3  Variance computation

Under some regularity assumptions on the response mechanisms and some regularity conditions on the $p^{\delta }\left(\cdot \mathrm{,}\cdot \right)’\text{s,}$ we obtain from Theorem 1 in Kim and Kim (2007) that we can write

$${\widehat{Y}}_t\mathrm{<mo>=</mo>}{\widehat{Y}}_{\text{lin}\text{}\mathrm{,}t}\left(t\right)+O_p\left(N{n}^{-1}\right)\mathrm{,}(2.4)$$

where

$${\widehat{Y}}_{\text{lin}\text{}\mathrm{,}t}\left(t\right)\mathrm{<mo>=</mo>}{\displaystyle \sum _{i\in s_{t-1}}}\frac{1}{{\pi }_i{\widehat{p}}_i^{1\to t-1}}\left\{k_i^t{\pi }_i{\widehat{p}}_i^{1\to t-1}{p}_i^t{\left(h_i^t\right)}^{\top }{\gamma }^t+\frac{r_i^t}{p_i^t}\left(y_{it}-k_i^t{\pi }_i{\widehat{p}}_i^{1\to t-1}{p}_i^t{\left(h_i^t\right)}^{\top }{\gamma }^t\right)\right\},(2.5)$$

and where for any $\delta \mathrm{<mo>=</mo>\; 1,}\dots \mathrm{,}t$ we denote by $h_i^{\delta }$ the value of $h_i^{\delta }\left(\alpha \right)\mathrm{<mo>=</mo>}\partial \text{logit}\left(p_i^{\delta }\right)/\partial \alpha$ evaluated at $\alpha \mathrm{<mo>=</mo>}{\alpha }^{\delta },$ and

$${\gamma }^{\delta }={\left\{{\displaystyle \sum _{i\in s_{\delta -1}}}{k}_i^{\delta }{p}_i^{\delta }\left(1-p_i^{\delta }\right)h_i^{\delta }{\left(h_i^{\delta }\right)}^{\top }\right\}}^{-1}{\displaystyle \sum _{i\in s_{\delta -1}}}\frac{1-p_i^{\delta }}{{\widehat{p}}_i^{1\to \delta -1}}{h}_i^{\delta }\frac{y_{it}}{{\pi }_i}\mathrm{.}(2.6)$$

From (2.5), we obtain that

$$E\left\{{\widehat{Y}}_{\text{lin}\text{}\mathrm{,}t}\left(t\right)|s_{t-1}\right\}\mathrm{<mo>=</mo>}{\widehat{Y}}_{t-1}\left(t\right)\mathrm{,}(2.7)$$

with ${\widehat{Y}}_{t-1}\left(t\right)$ the estimator of $Y\left(t\right)$ computed on $s_{t-1}.$ Using a proof by induction, it follows from (2.4) and (2.7) that ${\widehat{Y}}_t$ is approximately unbiased for $Y\left(t\right).$ Also, the variance of ${\widehat{Y}}_t$ may be asymptotically approximated by

$$V_{\text{app}}\left({\widehat{Y}}_t\right)\mathrm{<mo>=</mo>}V\left({\displaystyle \sum _{i\in s_0}}\frac{y_{it}}{{\pi }_i}\right)+E\left[{\displaystyle \sum _{\delta \mathrm{<mo>=</mo>\; 1}}^t}V\left\{{\widehat{Y}}_{\text{lin}\text{}\mathrm{,}\delta }\left(t\right)|s_{\delta -1}\right\}\right]\mathrm{.}(2.8)$$

The first term in the right-hand side of (2.8) is the variance due to the sampling design, that we note as $V^p\left({\widehat{Y}}_t\right).$ The second term in the right-hand side of (2.8) is the variance due to non-response, that we note as $V^{\text{nr}}\left({\widehat{Y}}_t\right).$ From (2.5), this asymptotic variance is given by

$$V^{\text{nr}}\left({\widehat{Y}}_t\right)\mathrm{<mo>=</mo>}E\left({\displaystyle \sum _{\delta \mathrm{<mo>=</mo>\; 1}}^t}{V}^{\text{nr}\delta }\left({\widehat{Y}}_t\right)\right)\mathrm{,}(2.9)$$

where

$$V^{\text{nr}\delta }\left({\widehat{Y}}_t\right)\mathrm{<mo>=</mo>}{\displaystyle \sum _{i\in s_{\delta -1}}}{p}_i^{\delta }\left(1-p_i^{\delta }\right){\left(\frac{y_{it}}{{\pi }_i{\widehat{p}}_i^{1\to \delta -1}{p}_i^{\delta }}-k_i^{\delta }{\left(h_i^{\delta }\right)}^{\top }{\gamma }^{\delta }\right)}^2\text{}\mathrm{.}(2.10)$$

We note that for each of its component $\delta \mathrm{<mo>=</mo>\; 1,}\dots \mathrm{,}t,$ the term $V^{\text{nr}\delta }\left({\widehat{Y}}_t\right)$ in (2.10) includes a centering term $k_i^{\delta }{\left(h_i^{\delta }\right)}^{\top }{\gamma }^{\delta }\text{},$ which is essentially a prediction of ${\left({\pi }_i{\widehat{p}}_i^{1\to \delta -1}{p}_i^{\delta }\right)}^{-1}{y}_i$ by means of regressors $h_i^{\delta }\text{}.$ This centering is due to the estimation of the response probabilities. Suppressing these centering terms, equations (2.9) and (2.10) would lead to the variance of the estimator of $Y\left(t\right)$ we would obtain by replacing in (2.3) the estimated probabilities by their true values. The variance of this estimator is usually larger than that of the reweighted estimator in (2.3); see also Beaumont (2005), equation (5.7) and Kim and Kim (2007), equation (17), for the case $t\mathrm{<mo>=</mo>\; 1.}$

2.4  Variance estimation

At time $t,$ an approximately unbiased estimator for the variance due to the sampling design $V^p\left({\widehat{Y}}_t\right)$ is

$${\widehat{V}}_t^p\left({\widehat{Y}}_t\right)\mathrm{<mo>=</mo>}{\displaystyle \sum _{i\mathrm{,}j\in s_t}}\frac{{\Delta }_{ij}}{{\pi }_{ij}}\frac{1}{{\widehat{p}}_{ij}^{1\to t}}\frac{y_{it}}{{\pi }_i}\frac{y_{jt}}{{\pi }_j}\mathrm{,}(2.11)$$

where ${\widehat{p}}_{ij}^{1\to t}\equiv {\displaystyle {\prod }_{\delta \mathrm{<mo>=</mo>\; 1}}^t}{\widehat{p}}_{ij}^{\delta }\text{},$ and where ${\widehat{p}}_{ij}^{\delta }\mathrm{<mo>=</mo>}{\widehat{p}}_i^{\delta }$ if $i\mathrm{<mo>=</mo>}j,$ and ${\widehat{p}}_{ij}^{\delta }\mathrm{<mo>=</mo>}{\widehat{p}}_i^{\delta }{\widehat{p}}_j^{\delta }$ otherwise. Following equation (25) in Kim and Kim (2007), $V^{\text{nr}}\left({\widehat{Y}}_t\right)$ may be approximately unbiasedly estimated at time $t$ by

$${\widehat{V}}_t^{\text{nr}}\left({\widehat{Y}}_t\right)\mathrm{<mo>=</mo>}{\displaystyle \sum _{\delta \mathrm{<mo>=</mo>\; 1}}^t}{\widehat{V}}_t^{\text{nr}\delta }\left({\widehat{Y}}_t\right)(2.12)$$

where

$${\widehat{V}}_t^{\text{nr}\delta }\left({\widehat{Y}}_t\right)\mathrm{<mo>=</mo>}{\displaystyle \sum _{i\in s_t}}\frac{{\widehat{p}}_i^{\delta }\left(1-{\widehat{p}}_i^{\delta }\right)}{{\widehat{p}}_i^{\delta \to t}}{\left(\frac{y_{it}}{{\pi }_i{\widehat{p}}_i^{1\to \delta }}-k_i^{\delta }{\left({\widehat{h}}_i^{\delta }\right)}^{\top }{\widehat{\gamma }}_t^{\delta }\right)}^2\text{}\mathrm{,}(2.13)$$

$${\widehat{h}}_i^{\delta }\mathrm{<mo>=</mo>}h\left(z_i\text{}\mathrm{,}{\widehat{\alpha }}^{\delta }\right)\mathrm{,}(2.14)$$

$${\widehat{\gamma }}_t^{\delta }\mathrm{<mo>=</mo>}{\left\{{\displaystyle \sum _{i\in s_t}}{k}_i^{\delta }\frac{{\widehat{p}}_i^{\delta }\left(1-{\widehat{p}}_i^{\delta }\right)}{{\widehat{p}}_i^{\delta \to t}}{\widehat{h}}_i^{\delta }{\left({\widehat{h}}_i^{\delta }\right)}^{\top }\right\}}^{-1}{\displaystyle \sum _{i\in s_t}}\frac{1-{\widehat{p}}_i^{\delta }}{{\widehat{p}}_i^{1\to t}}{\widehat{h}}_i^{\delta }\frac{y_{it}}{{\pi }_i}\mathrm{.}(2.15)$$

This leads to the global variance estimator at time $t$

$${\widehat{V}}_t\left({\widehat{Y}}_t\right)\mathrm{<mo>=</mo>}{\widehat{V}}_t^p\left({\widehat{Y}}_t\right)+{\widehat{V}}_t^{\text{nr}}\left({\widehat{Y}}_t\right)\mathrm{.}(2.16)$$

A simplified estimator of the variance due to non-response is obtained by ignoring the prediction terms $k_i^{\delta }{\left({\widehat{h}}_i^{\delta }\right)}^{\top }{\widehat{\gamma }}_t^{\delta }$ for each of the $\delta \mathrm{<mo>=</mo>\; 1,}\dots \mathrm{,}t$ variance components. After some algebra, this leads to the simplified variance estimator

$${\widehat{V}}_{t\text{}\mathrm{,}\text{simp}}^{\text{nr}}\left\{{\widehat{Y}}_t\left(t\right)\right\}\mathrm{<mo>=</mo>}{\displaystyle \sum _{i\in s_t}}\frac{1-{\widehat{p}}_i^{1\to t}}{{\left({\widehat{p}}_i^{1\to t}\right)}^2}{\left(\frac{y_{it}}{{\pi }_i}\right)}^2\text{}\mathrm{.}(2.17)$$

The main advantage of this simplified variance estimator is that it only requires the knowledge of the estimated response probabilities. On the other hand, the computation of the variance estimator in (2.12) requires the knowledge of the response models used at all times. The simplified variance estimator is therefore of particular interest for secondary users of the survey data, for which the estimated response probabilities may be the only available information related to the response modeling. This simplified variance estimator will tend to overestimate the variance due to non-response of $\left({\widehat{Y}}_t\right)$ if the prediction term $k_i^{\delta }{\left(h_i^{\delta }\right)}^{\top }{\gamma }^{\delta }$ partly explains ${\left({\pi }_i{\widehat{p}}_i^{1\to \delta -1}{p}_i^{\delta }\right)}^{-1}{y}_{it}\text{}.$

2.5  Application to the logistic regression model

In the particular case when a logistic regression model is used at each time $\delta ,$ the model (2.1) may be rewritten as

$$\text{logit}\left(p_i^{\delta }\right)\mathrm{<mo>=</mo>}{\left(z_i^{\delta }\right)}^{\top }{\alpha }^{\delta }\mathrm{.}(2.18)$$

We obtain ${\widehat{h}}_i^{\delta }\mathrm{<mo>=</mo>}{z}_i^{\delta }\text{},$ and the estimator for the variance due to non-response is given by (2.12), with

$${\widehat{V}}_t^{\text{nr}\delta }\left({\widehat{Y}}_t\right)\mathrm{<mo>=</mo>}{\displaystyle \sum _{i\in s_t}}\frac{{\widehat{p}}_i^{\delta }\left(1-{\widehat{p}}_i^{\delta }\right)}{{\widehat{p}}_i^{\delta \to t}}{\left(\frac{y_{it}}{{\pi }_i{\widehat{p}}_i^{1\to \delta }}-k_i^{\delta }{\left(z_i^{\delta }\right)}^{\top }{\widehat{\gamma }}_t^{\delta }\right)}^2\text{}\mathrm{,}(2.19)$$

$${\widehat{\gamma }}_t^{\delta }\mathrm{<mo>=</mo>}{\left\{{\displaystyle \sum _{i\in s_t}}{k}_i^{\delta }\frac{{\widehat{p}}_i^{\delta }\left(1-{\widehat{p}}_i^{\delta }\right)}{{\widehat{p}}_i^{\delta \to t}}{z}_i^{\delta }{\left(z_i^{\delta }\right)}^{\top }\right\}}^{-1}{\displaystyle \sum _{i\in s_t}}\frac{1-{\widehat{p}}_i^{\delta }}{{\widehat{p}}_i^{1\to t}}{z}_i^{\delta }\frac{y_{it}}{{\pi }_i}\mathrm{.}(2.20)$$

If the reweighted estimator is computed at time $t\mathrm{<mo>=</mo>\; 1},$ the estimator in (2.12) for the variance due to non-response may be rewritten as

$${\widehat{V}}_1^{\text{nr}}\left({\widehat{Y}}_1\right)\mathrm{<mo>=</mo>}{\displaystyle \sum _{i\in s_1}}\left(1-{\widehat{p}}_i^1\right){\left(\frac{y_{i1}}{{\pi }_i{\widehat{p}}_i^1}-k_i^1{\left(z_i^1\right)}^{\top }{\widehat{\gamma }}_1^1\right)}^2\mathrm{.}(2.21)$$

If the reweighted estimator is computed at time $t\mathrm{<mo>=</mo>\; 2},$ the estimator in (2.12) for the variance due to non-response may be rewritten as

$$\begin{array}{ll}{\widehat{V}}_2^{\text{nr}}\left({\widehat{Y}}_2\right)\hfill & \mathrm{<mo>=</mo>}{\displaystyle \sum _{i\in s_2}}\frac{\left(1-{\widehat{p}}_i^1\right)}{{\widehat{p}}_i^2}{\left(\frac{y_{i2}}{{\pi }_i{\widehat{p}}_i^1}-k_i^1{\left(z_i^1\right)}^{\top }{\widehat{\gamma }}_2^1\right)}^2\hfill \\ \hfill & +{\displaystyle \sum _{i\in s_2}}\left(1-{\widehat{p}}_i^2\right){\left(\frac{y_{i2}}{{\pi }_i{\widehat{p}}_i^1{\widehat{p}}_i^2}-k_i^2{\left(z_i^2\right)}^{\top }{\widehat{\gamma }}_2^2\right)}^2\text{}\mathrm{.}(2.22)\hfill \end{array}$$

In practice, the model of Response Homogeneity Groups (RHG) is often assumed when correcting for unit non-response. Under this model, it is assumed that at each time $\delta \mathrm{<mo>=</mo>\; 1,}\dots \mathrm{,}t,$ the sub-sample $s_{\delta -1}$ may be partitioned into $C\left(\delta -1\right)$ groups $s_{\delta -1}^c\text{}\mathrm{,}c\mathrm{<mo>=</mo>\; 1,}\dots \mathrm{,}C\left(\delta -1\right),$ such that the response probability $p_i^{\delta }$ is constant inside a group. This model is a particular case of the logistic regression model in (2.18), obtained with

$$z_i^{\delta }\mathrm{<mo>=</mo>}{\left[1\left\{i\in s_{\delta -1}^1\right\}\mathrm{,}\dots \mathrm{,}1\left\{i\in s_{\delta -1}^{C\left(\delta -1\right)}\right\}\right]}^{\top }\text{}\mathrm{,}(2.23)$$

and the variance due to non-response is estimated accordingly. Explicit formulas are given in Appendix.

• Previous
• Article main page
• Next