5 Simulation study

J.N.K. Rao, F. Verret and M.A. Hidiroglou

Previous | Next

We conducted a small simulation study on the performance of the proposed WEE estimators under the simple nested error mean model, using $\mu =0.5$, ${\sigma }_v^2=0.5$ and ${\sigma }_e^2=\mathrm{2.0.}$ The population consists of $N=$ 1,000 clusters, each containing $M_i=M=100$ elements. A two-stage sampling design with $n=$ 50 sample clusters and $m_i=m=5$ sample elements from each sample cluster is used. Clusters are selected by simple random sampling, and the elements within clusters by the Rao-Sampford probability proportional to size (PPS) sampling method (Rao 1965 and Sampford 1967) with specified size measures $z_{ij}.$ The size measures are chosen to reflect different levels of informativeness.

Following Asparouhov (2006), we considered both invariant and non-invariant selections. For invariant selection, the size measure $z_{ij}$ depends only on the level 1 errors and is invariant across clusters. In particular, we let

$$z_{ij}={\left(1+\exp \left\{-0.5\left[e_{ij}/\alpha +e_{ij}^{*}{\left(1-{\alpha }^{-2}\right)}^{1/2}\right]\right\}\right)}^{-1},\left(5.1\right)$$

where $e_{ij}^{*}$ is independent of $e_{ij}$ but with the same distribution, $N\left(0,{\sigma }_e^2=2.0\right)$. For non-invariant selection, the size measure $z_{ij}$ depends on both level 1 and level 2 errors and hence non-invariant across clusters. In particular, we replace $e_{ij}$ and $e_{ij}^{*}$ in (3.7) by $v_i+e_{ij}$ and $v_i^{*}+e_{ij}^{*}$ respectively, where $v_i^{*}$ is independent of $v_i$ but with the same distribution $N\left(0,{\sigma }_v^2=0.5\right)$. We considered four values of $\alpha$ in (5.1): $\alpha =1,2,3,\infty ,$ where $\alpha =\infty$ corresponds to non-informative sampling within each cluster, $\alpha =1$ corresponds to the most informative sampling and informativeness decreases as $\alpha$ increases.

We used the design-model ( $pm$ ) approach to simulate $R=1,000$ samples for each specified $\alpha$ and separately for invariant and non-invariant selections. Under this approach, we generated a population with $N=1,000$ and $M_i=M=100$ from the model and then selected a two-stage sample of elements as specified above. The two-step process was repeated $R=1,000$ times to simulate $1,000$ samples.

5.1 Performance of estimators

From each sample, we computed the estimates of $\mu ,{\sigma }_v^2$ and ${\sigma }_e^2$ using REML, weighted scaling methods A and A1, the proposed WEE method and the alternative method of Korn and Graubard (abbreviated KG). Biases and variances of the estimators were computed from the $1,000$ estimates. Performance of alternative estimators is judged using two performance measures: Bias ratio = BR = (Bias)/ (square root of variance) and relative root mean squared error = RRMSE = (square root of MSE)/ (true parameter value). Tables 5.1, 5.2 and 5.3 respectively report the BR values of the estimators of $\mu ,{\sigma }_v^2$ and ${\sigma }_e^2$. RRMSE values of the estimators of $\mu ,{\sigma }_v^2$ and ${\sigma }_e^2$ are reported in Tables 5.4, 5.5 and 5.6 respectively.

Table 5.1
Bias ratio (%) of estimators of $\mu$
Table summary
This table displays bias ratio (%) of estimators of $\mu$. The information is grouped by $\alpha$ (appearing as row headers), invariant and non-invariant (appearing as column headers).

$\alpha$	Invariant			Non-invariant
	REML	A	A1/WEE/KG	REML	A	A1/WEE/KG
1	346.5	80.2	2.2	370.9	83.9	3.0
2	167.7	40.1	0.3	172.3	45.3	6.1
3	114.3	30.7	4.5	114.9	30.8	4.8
$\infty$	2.0	2.5	2.1	-1.5	-2.4	-2.2

Table 5.1 reports bias ratio (%) of the estimators of $\mu$ based on REML, weight-scaling methods A and A1, KG and WEE. Note that in the case of $\mu$, estimators A1, KG and WEE (WCL) are identical. Results in Table 5.1 show that BR is similar for invariant and non-invariant selections and that BR of REML and A decrease as $\alpha$ increases. Further, REML leads to large bias under informative sampling, even for $\alpha =3$; for example, BR for REML ranges from 114% to 346% under invariant selection. Method A also leads to significant BR under informative sampling; for example BR for A ranges from 30.8% to 83.9% under non-invariant selection. On the other hand, BR of WEE, A1 and KG does not depend on $\alpha$ and it is small ( $\left|BR\right|<6\%$ ). Under non-informative sampling, REML performs well as expected ( $\left|BR\right|<3\%$ ).

Table 5.2
Bias ratio (%) of estimators of ${\sigma }_v^2$
Table summary
This table displays Bias ratio (%) of estimators of ${\sigma }_v^2$. The information is grouped by $\alpha$ (appearing as row headers), REML, A, A1, WEE, KG (appearing as column headers).

$\alpha$	REML	A	A1	WEE	KG
Invariant Selection
1	0.6	59.5	59.3	-8.5	33.2
2	0.5	24.5	26.3	-10.0	8.0
3	-3.4	16.1	18.2	-13.6	0.4
$\infty$	-0.1	14.8	17.1	-8.9	0.6
Non-invariant Selection
1	-49.0	50.1	58.9	-4.4	24.0
2	-10.9	24.6	28.7	-7.0	7.1
3	-4.0	20.0	22.7	-7.8	4.6
$\infty$	-1.3	12.8	13.9	-13.3	-1.6

Turning to the estimation of ${\sigma }_v^2$, we first note that the proportion of times the estimate of ${\sigma }_v^2$ is negative is zero in the simulations for all four values of $\alpha$ and for all the estimation methods (REML, A, A1, WEE and KG). Table 5.2 reports BR values of the estimators of ${\sigma }_v^2$. It shows that the BR of REML is not affected by $\alpha$ under invariant selection, but is affected under non-invariant selection. In the latter case, REML leads to serious underestimation for $\alpha =1$ (BR= -49%) but |BR| decreases as $\alpha$ increases. Table 5.2 also shows that methods A and A1 do not perform well under informative sampling (BR ranging from 16% to 60%). KG did not perform well for $\alpha =1$ (BR=33% under invariant selection and BR=24% under non-invariant selection). On the other hand, WEE performs well for all values of $\alpha$ (BR ranging from -4% to -13%) although underestimation is consistent across values of $\alpha$.

Table 5.3
Bias ratio (%) of estimators of ${\sigma }_e^2$
Table summary
This table displays Bias ratio (%) of estimators of ${\sigma }_e^2$. The information is grouped by $\alpha$ (appearing as row headers), REML, A, A1, WEE, KG (appearing as column headers).

$\alpha$	REML	A	A1	WEE	KG
Invariant Selection
1	-106.9	-118.4	-66.9	2.4	-71.2
2	-22.7	-43.6	-34.3	2.1	-16.5
3	-9.4	-31.7	-28.4	2.9	-6.5
$\infty$	-0.4	-21.8	-23.8	0.3	0.4
Non-invariant Selection
1	-115.3	-131.3	-79.6	-6.9	-82.6
2	-30.4	-51.1	-43.3	-7.6	-23.9
3	-12.5	-34.9	-32.2	-2.3	-10.3
$\infty$	1.1	-20.2	-21.8	2.6	1.6

Table 5.3 reports BR values of the estimators of ${\sigma }_e^2$. It shows that BR values are similar for invariant and non-invariant selections, as in the case of $\mu$. REML and KG lead to serious underestimation when $\alpha =1$ (BR= -107% for REML and BR= -71% for KG under invariant selection), but |BR| decreases as $\alpha$ increases and becomes negligible for $\alpha =\infty$. Estimators A and A1 perform poorly for all values of $\alpha$ including $\alpha =\infty$. On the other hand, WEE performs well for all values of $\alpha$ with |BR|<8%. It appears that the instability introduced by the scale factor (2.9) might have contributed to the large |BR| for methods A and A1 even for the case of non-informative sampling $(\alpha =\infty )$.

Table 5.4
Relative root mean squared error (%) of estimators of $\mu$
Table summary
This table displays relative root mean squared error (%) of estimators of $\mu$. The information is grouped by $\alpha$ (appearing as row headers), invariant, non-invariant (appearing as column headers).

$\alpha$	Invariant			Non-invariant
	REML	A	A1/WEE/KG	REML	A	A1/WEE/KG
1	93.3	35.9	29.4	92.5	35.4	29.2
2	51.6	29.3	27.8	52.8	30.4	28.9
3	40.5	28.2	27.5	40.8	28.7	28.1
$\infty$	25.8	26.1	26.5	26.6	27.3	27.7

Relative root mean squared error

Table 5.4 shows that the RRMSE (%) values for estimators of $\mu$ are similar for invariant and non-invariant selections and that RRMSE of REML and A decrease as $\alpha$ increases. For informative sampling with $\alpha =1$, RRMSE for REML is large relative to RRMSE for WEE (A1 and KG) due to large BR. For example, RRMSE=93% for REML compared to RRMSE=29% for WEE. As expected, REML has the smallest RRMSE under non-informative sampling, but the increase in RRMSE for the other methods is quite small. Also, RRMSE of WEE (A1 and KG) depends on $\alpha$.

Table 5.5
Relative root mean squared error (%) of estimators of ${\sigma }_v^2$
Table summary
This table displays Relative root mean squared error (%) of estimators of ${\sigma }_v^2$. The information is grouped by $\alpha$ (appearing as row headers), REML, A, A1, WEE, KG (appearing as column headers).

$\alpha$	REML	A	A1	WEE	KG
Invariant Selection
1	36.5	47.3	51.1	43.6	43.8
2	37.1	39.7	41.1	40.5	39.5
3	36.3	37.3	38.7	39.5	37.8
$\infty$	35.8	36.9	38.1	38.7	37.2
Non-invariant Selection
1	36.7	44.6	52.6	43.4	41.5
2	35.6	37.9	40.4	39.3	37.7
3	37.0	38.7	40.4	40.2	38.8
$\infty$	36.6	37.2	38.0	39.0	37.8

Turning to RRMSE of estimators of ${\sigma }_v^2$, Table 5.5 shows that REML performs well for all $\alpha$ under invariant selection due to small BR in this case. We also note that KG and WEE are comparable in terms of RRMSE for all values of $\alpha$. Table 5.5 also shows that A and A1 lead to somewhat larger RRMSE for $\alpha =1$: 51% for A1 and 47% for A under invariant selection compared to 44% for WEE.

Table 5.6
Relative root mean squared error (%) of estimators of ${\sigma }_e^2$
Table summary
This table displays relative root mean squared error (%) of estimators of ${\sigma }_e^2$. The information is grouped by $\alpha$ (appearing as row headers), REML, A, A1, WEE, KG (appearing as column headers).

$\alpha$	REML	A	A1	WEE	KG
Invariant Selection
1	13.5	14.5	12.8	13.9	12.9
2	9.7	10.4	10.4	11.0	10.0
3	9.5	10.0	10.1	10.7	9.8
$\infty$	10.1	10.3	10.5	11.1	10.3
Non-invariant Selection
1	13.7	14.8	12.9	13.2	13.0
2	10.0	10.9	10.9	11.3	10.3
3	9.7	10.4	10.7	11.2	10.2
$\infty$	10.3	10.6	10.8	11.4	10.7

Table 5.6 gives RRMSE values of the estimators of ${\sigma }_e^2$ and we note that the values are similar for invariant and non- invariant selections. It also shows that RRMSE values are comparable for methods WEE, A, Al and KG even though in terms of bias ratio A, Al and KG performed poorly relative to WEE. This is due to larger variance for WEE compared to other methods. For example, in the case of invariant selection and $\alpha =1$ we have the following variances for WEE, KG and REML: 0.0771, 0.0438 and 0.0339 with corresponding bias ratios (%) from Table 5.3: 2.4, -71.2, and -106.9.

5.2 Performance of variance estimator

We now report some simulation results on the relative bias of the linearization variance estimator (3.12) of the WEE (WCL) estimator ${\widehat{\theta }}_w$. We first repeated the two-step process $R_1=2,000$ times and computed $v_L^{(r)}\left({\widehat{\theta }}_w\right)$ from each two-stage sample $r=1,\mathrm{...},2,000.$ The averages of the diagonal elements of $E\left\{v_L\left({\widehat{\theta }}_w\right)\right\}\approx v_L\left({\widehat{\theta }}_w\right)=R_1^{-1}{\displaystyle {\sum }_{r=1}^{R_1}{v}_L^{(r)}}\left({\widehat{\theta }}_w\right)$ are denoted by ${\overline{v}}_L\left({\widehat{\mu }}_w\right),{\overline{v}}_L\left({\widehat{\sigma }}_{vw}^2\right)$ and ${\overline{v}}_L\left({\widehat{\sigma }}_{ew}^2\right)$ respectively. We then generated $R_2=10,000$ independent samples and computed the empirical mean squared error (MSE) of the three estimators ${\widehat{\mu }}_w,{\widehat{\sigma }}_{vw}^2$ and ${\widehat{\sigma }}_{ew}^2.$ We have $\text{MSE}\left({\widehat{\mu }}_w\right)\approx R_2^{-1}{{\displaystyle {\sum }_{r=1}^{R_2}\left({\widehat{\mu }}_w^{(r)}-\mu \right)}}^2$ where ${\widehat{\mu }}_w^{(r)}$ is the estimate of $\mu$ from the r-th simulated sample, and similar expressions for $\text{MSE}\left({\widehat{\sigma }}_{vw}^2\right)$ and $\text{MSE}\left({\widehat{\sigma }}_{ew}^2\right).$ 

The relative bias of $v_L\left({\widehat{\mu }}_w\right)$ is calculated as

$\text{RB}\left\{v_L\left({\widehat{\mu }}_w\right)\right\}=\left[{\overline{v}}_L\left({\widehat{\mu }}_w\right)/\text{MSE}\left({\widehat{\mu }}_w\right)\right]-1$

and similarly $\text{RB}\left\{v_L\left({\widehat{\sigma }}_{vw}^2\right)\right\}$ and $\text{RB}\left\{v_L\left({\widehat{\sigma }}_{ew}^2\right)\right\}$ were calculated. Table 5.7 reports the RB values of the three variance estimators for invariant and non-invariant selections and $\alpha =1,2,3,\infty .$ It is clear from Table 5.7 that the linearization variance estimator performs well over all combinations with $\left|\text{RB}\right|<10\%.$

Table 5.7
Relative bias (%) of variance estimators
Table summary
This table displays relative bias (%) of variance estimators. The information is grouped by $\alpha$ (appearing as row headers), $v_L({\widehat{\mu }}_w)$, $v_L({\widehat{\sigma }}_{vw}^2)$, $v_L({\widehat{\sigma }}_{ew}^2)$(appearing as column headers).

$\alpha$	$v_L({\widehat{\mu }}_w)$	$v_L({\widehat{\sigma }}_{vw}^2)$	$v_L({\widehat{\sigma }}_{ew}^2)$
Invariant Selection
1	-3.0	-6.2	-7.5
2	-5.2	-4.5	-3.1
3	-1.3	-3.8	-1.8
$\infty$	-0.9	-2.5	-2.0
Non-invariant Selection
1	-3.8	-8.3	-4.2
2	-4.5	-5.8	-7.3
3	-4.3	-4.6	-5.7
$\infty$	-2.4	-2.7	-2.9

Previous | Next