Measuring uncertainty associated with model-based small area estimators
Section 4. Design MSE estimation

• Previous
• Next

In this section we first study design MSE estimation and then propose composite MSE estimation that provides a balance between the design bias and the coefficient of variation.

4.1  Area-level model

We now turn to estimating the design MSE of the EB estimator by treating the small area parameters ${\theta }_i$ as fixed unknown parameters. As noted in the introduction, survey statisticians are often interested in estimating the design MSE of EB estimators in line with the traditional design MSE estimators of direct estimators for large areas with adequate sample sizes. The design MSE is given by ${\text{MSE}}_d\left({\widehat{\theta }}_i^{\text{EB}}\right)=E\left[{\left({\widehat{\theta }}_i^{\text{EB}}-{\theta }_i\right)}^2|\theta \right],$ where $\theta ={\left({\theta }_1,\dots ,{\theta }_m\right)}^{\prime }$ is the vector of area means.

Expressing ${\widehat{\theta }}_i^{\text{EB}}$ as ${\widehat{\theta }}_i+h_i\left(\widehat{\theta }\right)$ with $h_i\left(\widehat{\theta }\right)=-\left(1-{\widehat{\gamma }}_i\right)\left({\widehat{\theta }}_i-z_i^{\prime }\widehat{\beta }\right),$ an exactly unbiased estimator of the design MSE is given by

$${\text{mse}}_d\left({\widehat{\theta }}_i^{\text{EB}}\right)={\psi }_i+2{\psi }_i\left[\partial h_i\left(\widehat{\theta }\right)/\partial {\widehat{\theta }}_i\right]+h_i^2\left(\widehat{\theta }\right).(4.1)$$

Datta, Kubokawa, Molina and Rao (2011) give an explicit expression for the derivative in the second term of (4.1) in the case of REML estimators of model parameters. The estimator (4.1) can take negative values and can be very unstable in terms of relative root mean squared error (RRMSE) as shown by Datta et al. (2011). It follows that (4.1) is not a reliable estimator of the design MSE, although it is design unbiased. Our simulation results in Section 5 study the conditional properties of the MSE estimators (3.1) and (4.1) in the design-based framework.

Some theoretical insights can be obtained by focusing on the case of known model parameters and considering the best estimator (2.2) of the area mean ${\theta }_i.$ In this case, Rivest and Belmonte (2000) obtained a design-unbiased estimator given by

$${\text{mse}}_d\left({\tilde{\theta }}_i^B\right)={\gamma }_i{\psi }_i+{\left(1-{\gamma }_i\right)}^2\left[{\left({\widehat{\theta }}_i-z_i^{\prime }\beta \right)}^2-\left({\psi }_i+{\sigma }_v^2\right)\right].(4.2)$$

Note that for a large sampling variance ${\psi }_i$ we have ${\gamma }_i\approx 0$ and (4.2) reduces to

$${\text{mse}}_d\left({\tilde{\theta }}_i^B\right)\approx {\left({\widehat{\theta }}_i-z_i^{\prime }\beta \right)}^2-{\psi }_i.(4.3)$$

It follows from (4.3) that the MSE estimator can take negative values and, in fact, the probability of getting a negative value is close to 0.5 when ${\gamma }_i$ is close to zero or sampling variance ${\psi }_i$ is large. In this special case of known model parameters, we can study the design bias of the model MSE estimator of (2.2), given by $\text{mse}\left({\tilde{\theta }}_i^B\right)={\gamma }_i{\psi }_i,$ when averaged over the areas. It can be shown that the average design bias converges in model probability to zero as $m\to \infty$ (Rao and Molina, 2015, page 287). This result suggests that the model MSE estimator should perform well in terms of average design bias, provided the assumed model is valid.

The design-unbiased estimator (4.1) is not usable in practice when it takes a negative value for the sample at hand. Therefore, we propose a modification of (4.1) that leads to a positive MSE estimator. We denote the modified MSE estimator by ${\text{mod-mse}}_d\left({\widehat{\theta }}_i^{\text{EB}}\right).$ It uses (4.1) when it takes a positive value for the sample at hand and replaces (4.1) by the model MSE estimate (3.1) when (4.1) takes a negative value. It is possible to use some other positive MSE estimate, for example a naïve positive design-based MSE estimator proposed by Pfeffermann and Gilboa (2017). We have not studied this modification in our simulation study.

We now propose composite estimators of the design MSE that attempt to provide a balance between design bias and RRMSE. One composite estimator is obtained by taking a weighted average of the design MSE estimator (4.1) and the unconditional model MSE estimator (3.1) with weights ${\widehat{\gamma }}_i$ and $\left(1-{\widehat{\gamma }}_i\right)$ respectively. This composite MSE estimator may be written as

$${\text{mse}}_{c1}\left({\widehat{\theta }}_i^{\text{EB}}\right)={\widehat{\gamma }}_i{\text{mse}}_d\left({\widehat{\theta }}_i^{\text{EB}}\right)+\left(1-{\widehat{\gamma }}_i\right)\text{mse}\left({\widehat{\theta }}_i^{\text{EB}}\right).(4.4)$$

It follows from (4.4) that less weight is given to the design MSE estimator when the sampling variance is large and this controls the RRMSE of the composite MSE estimator. Also, the composite MSE estimator has always a smaller design bias than the model MSE estimator. When ${\widehat{\gamma }}_i$ (or the area sample size) is very small, another choice of the compositing weights is to replace ${\widehat{\gamma }}_i$ by $\sqrt{{\widehat{\gamma }}_i}$ and $1-{\widehat{\gamma }}_i$ by $1-\sqrt{{\widehat{\gamma }}_i}$ in (4.4). The resulting composite MSE estimator

$${\text{mse}}_{c2}\left({\widehat{\theta }}_i^{\text{EB}}\right)=\sqrt{{\widehat{\gamma }}_i}{\text{mse}}_d\left({\widehat{\theta }}_i^{\text{EB}}\right)+\left(1-\sqrt{{\widehat{\gamma }}_i}\right)\text{mse}\left({\widehat{\theta }}_i^{\text{EB}}\right)(4.5)$$

gives more weight to ${\text{mse}}_d\left({\widehat{\theta }}_i^{\text{EB}}\right)$ than (4.4) and thus performs better in terms of design bias at the expense of increased MSE. Similar to (4.4), the alternative composite MSE estimator (4.5) has always a smaller design bias than the model MSE estimator. Both (4.4) and (4.5) can also take on negative values but likely not as often due to their construction. To ensure positive composite MSE estimators, we make a modification similar to ${\text{mod-mse}}_d\left({\widehat{\theta }}_i^{\text{EB}}\right)$ and replace (4.4) and (4.5) by the model MSE estimate (3.1) when they take negative values for the sample at hand. We denote the modified estimators by ${\text{mod-mse}}_{c1}\left({\widehat{\theta }}_i^{\text{EB}}\right)$ and ${\text{mod-mse}}_{c2}\left({\widehat{\theta }}_i^{\text{EB}}\right)$ respectively. In Section 5, we look at the performance of the two modified composite MSE estimators relative to the model MSE estimator (3.1) and the modified design MSE estimator in terms of ARB, RRMSE and coverage rate of confidence intervals.

4.2  Unit-level model

We focus on simple random sampling (SRS) without replacement in each area. Even for this special design, no closed form expressions for the design MSE of the EB estimator ${\widehat{\overline{Y}}}_i^{\text{EB}}$ and its estimator are available in the literature, unlike in the case of the area-level model. Therefore, we propose a heuristic method by evaluating the design MSE of the best estimator ${\widehat{\overline{Y}}}_i^B$ given by (2.6), under SRS assuming all the model parameters are known and then estimating the design MSE. The resulting design-unbiased MSE estimator of the best estimator depends on the model parameters and we replace the model parameters by their REML estimators. The resulting MSE estimator is not design-unbiased for the design MSE of the EB estimator and it is likely to underestimate the true design MSE because the variability associated with the estimated model parameters is not taken into account. We study its design performance in a simulation study.

Under SRS without replacement within area $i,$ we have

$${\widehat{\overline{Y}}}_i^B-{\overline{Y}}_i=a_i\left({\overline{u}}_i-{\overline{U}}_i\right)-\left(1-a_i\right){\overline{U}}_i,(4.6)$$

where ${\overline{u}}_i$ is the area sample mean and ${\overline{U}}_i$ is the area population mean of the values $u_{ij}=y_{ij}-x_{ij}^{\prime }\beta .$ It follows from (4.6) that the design MSE of the best estimator is given by

$${\text{MSE}}_d\left({\widehat{\overline{Y}}}_i^B\right)=E_d{\left({\widehat{\overline{Y}}}_i^B-{\overline{Y}}_i\right)}^2=a_i^2{V}_d\left({\overline{u}}_i\right)+{\left(1-a_i\right)}^2{\overline{U}}_i^2,(4.7)$$

where

$$V_d\left({\overline{u}}_i\right)=n_i^{-1}\left(1-f_i\right)S_{ui}^2,\text{and}{S}_{ui}^2={(N_i-1)}^{-1}{\displaystyle \sum _{j=1}^{N_i}{(u_{ij}-{\overline{U}}_i)}^2},(4.8)$$

noting that the cross-product term is zero under SRS.

It now follows from (4.7) and (4.8) that a design unbiased MSE estimator of the best estimator is given by

$${\text{mse}}_d\left({\widehat{\overline{Y}}}_i^B\right)=a_i^2{n}_i^{-1}\left(1-f_i\right)s_{ui}^2+{\left(1-a_i\right)}^2{\widehat{\overline{U}}}_i^{2D},(4.9)$$

where ${\widehat{\overline{U}}}_i^{2D}=n_i^{-1}{\displaystyle {\sum }_{j=1}^{n_i}{u}_{ij}^2}-N_i^{-1}\left(N_i-1\right)s_{ui}^2$ and $s_{ui}^2={\left(n_i-1\right)}^{-1}{{\displaystyle {\sum }_{j=1}^{n_i}\left(u_{ij}-{\overline{u}}_i\right)}}^2.$ By replacing the model parameters in (4.9) by their REML estimators, a design-based MSE estimator of the EB estimator is obtained, denoted by ${\text{mse}}_d^{*}\left({\widehat{\overline{Y}}}_i^{\text{EB}}\right).$ This MSE estimator is likely to underestimate the design MSE of the EB estimator because the best estimator (2.6) does not account for the variability in the estimators of model parameters.

A composite MSE estimator, ${\text{mse}}_c^{*}\left({\widehat{\overline{Y}}}_i^{\text{EB}}\right),$ is now obtained by taking a weighted combination of ${\text{mse}}_d^{*}\left({\widehat{\overline{Y}}}_i^{\text{EB}}\right)$ and the model-based MSE estimator $\text{mse}\left({\widehat{\overline{Y}}}_i^{\text{EB}}\right)$ with weights ${\widehat{\gamma }}_i$ and $1-{\widehat{\gamma }}_i$ respectively. It is given by

$${\text{mse}}_c^{*}\left({\widehat{\overline{Y}}}_i^{\text{EB}}\right)={\widehat{\gamma }}_i{\text{mse}}_d^{*}\left({\widehat{\overline{Y}}}_i^{\text{EB}}\right)+\left(1-{\widehat{\gamma }}_i\right)\text{mse}\left({\widehat{\overline{Y}}}_i^{\text{EB}}\right).(4.10)$$

Molina and Kominiak (2017) proposed parametric and non-parametric bootstrap estimators of the design MSE of ${\widehat{\overline{Y}}}_i^{\text{EB}}.$ They also obtained a composite MSE estimator, similar to (4.10), by using the non-parametric bootstrap (NPB) MSE estimator and the parametric bootstrap (PB) MSE estimator as the two components of the composite MSE estimator associated with ${\widehat{\gamma }}_i$ and $\left(1-{\widehat{\gamma }}_i\right)$ respectively. As noted by the authors, a drawback with this composite MSE estimator is “that it requires to run both PB and NPB procedures for each area, which makes it computationally slower.” Molina and Kominiak (2017) also proposed a parametric design bootstrap (PDB) composite MSE estimator. The PDB estimator avoids running both PB and NPB procedures for each area. Both bootstrap composite MSE estimators performed well in a design-based simulation study.

• Previous
• Article main page
• Next