2. Simulation studies: OBP vs EBLUP

Jiming Jiang, Thuan Nguyen and J. Sunil Rao

Previous | Next

2.1  A demonstration

We first use a simple simulated example to demonstrate the potential impact of model misspecification in terms of the design-based predictive performance of the OBP and the EBLUP. Consider a case where a single covariate, $x_{ij},$ is thought to be linearly associated with the response $y_{ij}$ through the following NER model:

$$y_{ij}=\beta x_{ij}+v_i+e_{ij}\mathrm{,}i=\mathrm{1,}\dots \mathrm{,}m\mathrm{,}j=\mathrm{1,}\dots \mathrm{,5}(2.1)$$

(so we have $n_i=\mathrm{5,1}\le i\le m$ in this case), where $\beta$ is an unknown coefficient, and $v_i\mathrm{,}{e}_{ij}$ are the same as in (1.1). Thus, in particular, there is a belief that the mean response should be zero when the value of the covariate is zero.

We consider three different sample sizes: $m=50, 100$ or $400$ in conjunction with two different true values of $b:b=0.5$ or $1.0$, where $b$ is defined below. Thus, there are six cases, each being a combination of the sample size and $b$ value. In each case, an $x$ subpopulation is generated from the normal distribution with mean equal to 1 and standard deviation equal to $\sqrt{0.1}\approx \mathrm{0.32.}$ The $y$ subpopulation is then generated from the following super-population heteroscedastic NER model:

$$Y_{ik}=b+v_i+e_{ik}\mathrm{,}i=\mathrm{1,}\dots \mathrm{,}m\mathrm{,}k=\mathrm{1,}\dots \mathrm{,}\text{1,000}(2.2)$$

(so the subpopulation size is $N_i=\text{1,000},1\le i\le m),$ where $v_i$ is generated from the normal distribution with mean 0 and standard deviation $\sqrt{0.1}\approx 0.32;e_{ij}$ is generated from the normal distribution with mean 0 and standard deviation ${\sigma }_i,$ where ${\sigma }_i^2$ are generated independently from the Uniform $\left[\mathrm{0.05,0.15}\right]$ distribution (so that range for ${\sigma }_i$ is approximately from 0.22 to 0.39); and the $v_i’\text{s}$ and $e_{ik}’\text{s}$ are generated independently. It is seen that the assumed NER model is misspecified in terms of both the mean and the variance functions. Once the $x$ and $y$ subpopulations are generated, they are fixed throughout the simulations.

In each simulation, we draw a simple random sample of size 5 from $\left\{\mathrm{1,}\dots \mathrm{,}\text{1,000}\right\}$ that determines the samples $x_{ij}$ and $y_{ij},j=\mathrm{1,}\dots \mathrm{,5},$ for each $i.$ This is repeated for $K=\text{1,000}$ simulation runs. We make same-data comparisons of the OBP and EBLUP, with the ML estimator of $\gamma$ for the latter, in terms of both the overall and area-specific MSPEs. The overall MSPE is defined as $\text{MSPE}\left(\widehat{\theta }\right)=$ $\text{E}\left({\left|\widehat{\theta }-\theta \right|}^2\right)={\displaystyle {\sum }_{i=1}^m\text{E}{\left({\widehat{\theta }}_i-{\theta }_i\right)}^2},$ where $\theta ={\left({\theta }_i\right)}_{1\le i\le m}$ is the vector of true small area means with ${\theta }_i={\overline{Y}}_i,$ and $\widehat{\theta }={\left({\widehat{\theta }}_i\right)}_{1\le i\le m}$ is the vector of predicted values (either by OBP or by EBLUP). Note that the same measure has been used in Jiang et al. (2011). Table 2.1 reports the overall MSPE results, where the MSPE is evaluated empirically by $K^{-1}{\displaystyle {\sum }_{k=1}^K}{\left|{\widehat{\theta }}^{\left(k\right)}-{\theta }^{\left(k\right)}\right|}^2=K^{-1}{\displaystyle {\sum }_{k=1}^K}{\displaystyle {\sum }_{i=1}^m}{\left\{{\widehat{\theta }}_i^{\left(k\right)}-{\theta }_i^{\left(k\right)}\right\}}^2,$ and ${\theta }^{\left(k\right)}=$ ${\left[{\theta }_i^{\left(k\right)}\right]}_{1\le i\le m}$ and ${\widehat{\theta }}^{\left(k\right)}={\left[{\widehat{\theta }}_i^{\left(k\right)}\right]}_{1\le i\le m}$ are the $\theta$ and $\widehat{\theta }$ in the $k^{\text{th}}$ simulation run, respectively. It is seen that the percentage increase in the overall MSPE of the EBLUP over the OBP ranges between around 20% to almost 1,000%, depending on the sample size and value of $b.$ The patterns shown here are consistent with those in Jiang et al. (2011) under the Fay-Herriot model, where model-based predictive performances are evaluated. However, the gain by the OBP is much more significant, for $m=100$ and $m=400,$ than those reported in Jiang et al. (2011).

Table 2.1
Overall empirical MSPE (% Increase is EBLUP over OBP)
Table summary
This table displays the results of Overall empirical MSPE (% Increase is EBLUP over OBP). The information is grouped by $m$ (appearing as row headers), $b$, OBP, EBLUP and % Increase (appearing as column headers).

$m$	$b$	OBP	EBLUP	% Increase
50	0.5	0.130	0.161	24
50	1.0	0.503	0.598	19
100	0.5	0.076	0.277	264
100	1.0	0.396	1.077	172
400	0.5	0.096	0.965	905
400	1.0	0.393	4.046	930

As for the area-specific MSPEs, following Jiang et al. (2011), we use boxplots to exhibit the distributions of the area-specific MSPEs associated with both methods. See Figure 2.1. The plots reveal details not shown by the overall MSPEs. For example, it might be wondered whether the percentage increase by the EBLUP in the overall MSPE is simply due to the increased number of areas adding together. A simple calculation suggests that this may not be true, for example, $\left(400/50\right)\times 19\%$ is only $152\%\left(\text{not }930\%\right).$ A more explicit explanation is given in Figure 2.1. For example, comparing the case of $m=\mathrm{50,}b=1$ with that of $m=\mathrm{400,}b=1,$ it is seen that while there is a considerable overlap between the boxplots of OBP and EBLUP in the former case, the boxplots are completely separated in the latter case; in other words, the largest area-specific MSPE of the OBP is smaller than the smallest area-specific MSPE of the EBLUP. This pattern cannot be simply credited to adding or duplicating the areas. In fact, in the latter case, the OBP is doing much better than the EBLUP not just overall, but also for every one of the 400 small areas. This is clearly something never reported before. For example, in the first simulated example of Jiang et al. (2011), the authors found that the OBP has smaller MSPE compared to the EBLUP for half of the small areas while the EBLUP has smaller MSPE compared to the OBP for the other half; similar patterns were found in the second simulated examples in Jiang et al. (2011).

The estimation of the area-specific MSPEs of the OBP is considered in Section 3.

Figure 2.1 Area-specific Empirical MSPEs (Boxplots). Upper Left: $m=50,b=0.5;$ Upper Right: $m=50,b=1.0;$ Middle Left: $m=\mathrm{100,}b=0.5;$ Lower Left: $m=\mathrm{400,}b=0.5;$ Lower Right: $m=\mathrm{400,}b=\mathrm{1.0.}$

Description for Figure 2.1

2.2 Further considerations

The situation considered in Subseciton 2.1 might be a little extreme (and this is why we call it a "theoretical demonstration�). In practice, the assumed model may not be completely wrong, or may be close to be correct. In this subsection we first consider a case where the assumed model is "partially correct�. Namely, the slope in (2.1) is nonzero (so the assumed model is correct in this regard); the intercept is nonzero, but its value is much smaller compared to those considered in Subsection 2.1 (so the assumed model is wrong, but not "terribly wrong�). More specifically, the true underlying model is

$$Y_{ij}=b_0+b_1{X}_{ik}+v_i+e_{ik}\mathrm{,}i=\mathrm{1,}\dots \mathrm{,}m\mathrm{,}k=\mathrm{1,}\dots \mathrm{,}\text{1,000}\mathrm{,}(2.3)$$

as opposed to (2.2), where $b_0=0.2,b_1=0.1;$ the $v_i$ are generated independently from the normal distribution with mean 0 and standard deviation 0.1; and $e_{ik}$ are generated from the heteroscedastic normal distribution as in Subseciton 2.1. In addition to the overall MSPE, we also report contribution to the MSPE due to "bias� and "variance�. Let $d_i={\widehat{\theta }}_i-{\theta }_i,$ and $d_i^{\left(k\right)}$ be $d_i$ based on the $k^{\text{th}}$ simulated data set, $1\le k\le K.$ We define the empirical bias and variance for the $i^{\text{th}}$ small area as ${\overline{d}}_i=K^{-1}{\displaystyle {\sum }_{k=1}^K{d}_i^{\left(k\right)}}$ and $v_i^2={\left(K-1\right)}^{-1}{\displaystyle {\sum }_{k=1}^K}{\left\{d_i^{\left(k\right)}-{\overline{d}}_i\right\}}^2,$ respectively. Let ${\text{MSPE}}_i$ denote the empirical MSPE for the $i^{\text{th}}$ small area. It is easy to show that the overall empirical MSPE is

$${\displaystyle \sum _{i=1}^m{\text{MSPE}}_i}=\frac{K-1}{K}{\displaystyle \sum _{i=1}^m{v}_i^2}+{\displaystyle \sum _{i=1}^m}{\left({\overline{d}}_i\right)}^2\mathrm{.}$$

Thus, the bias and variance contribution to the overall MSPE are defined as ${\displaystyle {\sum }_{i=1}^m}{\left({\overline{d}}_i\right)}^2$ and ${\displaystyle {\sum }_{i=1}^m{v}_i^2},$ respectively. Results based on $K=1\text{,}000$ simulation runs are presented in Table 2.2. As we can see, for the smaller $m,m=50,$ OBP performs (slightly) worse than the EBLUP, but for the larger $m,m=100$ and $m=400,$ OBP performs (slightly) better, and its advantage increases with $m.$ As for the bias, variance contribution, OBP seems to have smaller bias, and smaller variance for larger $m\left(m=\mathrm{100,400}\right).$

Table 2.2
Overall Empirical MSPE (bias, variance contribution): Assumed model is partially correct; % Increase is MSPE of EBLUP over MSPE of OBP (negative number indicates decrease)
Table summary
This table displays the results of Overall Empirical MSPE (bias. The information is grouped by $m$ (appearing as row headers), OBP, EBLUP and % Increase (appearing as column headers).

$m$	OBP	EBLUP	% Increase
50	0.421 (0.224, 0.197)	0.405 (0.238, 0.167)	-4.0
100	0.733 (0.448, 0.285)	0.748 (0.457, 0.291)	2.1
400	2.745 (1.847, 0.899)	2.848 (1.878, 0.971)	3.8

Next, we consider a case where the assumed model is actually correct. Namely, the true underlying model is (2.3) with $b_0=0;$ the errors $e_{ik}$ are homoscedastic with variance equal to 0.1, and everything else is the same as the case considered above. Results based on $K=1\text{,}000$ simulation runs are presented in Table 2.3. This time, we see that the EBLUP performs slightly better than OBP under different $m,$ but the difference is diminishing as the sample size increases. As for the bias, variance contribution, EBLUP seems to have smaller variance, and smaller bias for larger $m\left(m=\mathrm{100,400}\right),$ but its advantages in both bias and variance shrink as $m$ increases.

Table 2.3
Overall Empirical MSPE (bias, variance contribution): Assumed model is correct; % Increase is MSPE of EBLUP over MSPE of OBP (negative number indicates decrease)
Table summary
This table displays the results of Overall Empirical MSPE (bias. The information is grouped by $m$ (appearing as row headers), OBP, EBLUP and % Increase (appearing as column headers).

$m$	OBP	EBLUP	% Increase
50	0.335 (0.204, 0.131)	0.330 (0.205, 0.125)	-1.4
100	0.749 (0.457, 0.292)	0.746 (0.456, 0.290)	-0.4
400	2.796 (1.800, 0.997)	2.794 (1.799, 0.996)	-0.1

In summary, the simulation results suggest that, when the assumed model is slightly misspecified, OBP may not outperform EBLUP when $m,$ the number of small areas, is relatively small; however, OBP is expected to outperform EBLUP when $m$ is relatively large, and the advantage of OBP over EBLUP increases with $m$ (recall the definition of the overall MSPE). On the other hand, when the assumed model is correct, EBLUP is expected to perform better than OBP, although the difference may be ignorable; and the advantage of EBLUP over OBP is disappearing as $m$ increases. These findings, along with those in Subsection 2.1, are very much in line with those of Jiang et al. (2011; Section 4) under the Fay-Herriot model.

Previous | Next