3. Empirical examples

Stephen Ash

Previous | Next

The questions of interest for the empirical examples are:

Q1. How well does SDR perform with a subset of all the replicates needed for SDR to be equivalent to SD?

Q2. Which row assignment is better, RA1 or RA2?

Q3. Should we use more or fewer connected loops?

To address these questions, we applied the SDR variance estimator to several populations. With each population, we selected a $sys$ sample of size $n=$ 64. Table 3.1 outlines the three SDR estimators we applied.

Table 3.1
SDR estimators for the empirical examples

Estimator	$k_A$	$H_A$	$k_B$	$H_B$
1	4	$H_{4a}$	16	$H_{4a}\otimes H_{4a}$
2	16	$H_{4a}\otimes H_{4a}$	4	$H_{4a}$
3	64	$H_{4a}\otimes H_{4a}\otimes H_{4a}$	1	1

With this construction, the SDR estimators had $k_B=1,4,$ or 16 cycles, but all used the same $\tilde{H}=H_{4a}\otimes H_{4a}\otimes H_{4a},$ which is the normal Hadamard matrix of order $\tilde{k}=$ 64. For the three estimators of Table 3.1, we also varied the row assignment (RA1 and RA2) and the number of replicates used by each estimator is either 16, 32, 48, or 64. With both RA1 and RA2, there is only one connected loop within each cycle, so estimators 1, 2, and 3 had $k_B=16,4,$ and $1$ connected loops, respectively. In the Appendix Section, the results for the SDR estimators are summarized in Table A1 and Table A2 includes the SD1, SD2, and the $srswor$ variance estimators applied for comparison purposes.

Data sets used. The "A� populations are borrowed from the empirical example in Wolter (1984). For populations A1-A7, we generated 400 finite populations of size $N=$ 64,000. From each population, there were $b=$ 100 possible samples of size $n=$ 64. The samples are indexed as $i=1,2,\dots ,b=100$ and the units within each sample are indexed as $j=1,2,\dots ,n=64.$ Table 3.2 summaries how the variable of interest ${\mu }_{ij}$ is generated for each of the "A� populations.

Table 3.2
Description of Wolter's artificial populations

Population	Description	$n$	$b$	${\mu }_{ij}$	$e_{ij}$
A1	Random	20	50	0	$e_{ij}\text{iid}N\left(0,100\right)$
A2	Linear Trend	20	50	$i+\left(j-1\right)k$	$e_{ij}\text{iid}N\left(0,100\right)$
A3	Stratification Effects	20	50	j	$e_{ij}\text{iid}N\left(0,100\right)$
A4	Stratification Effects	20	50	$j+10$	$e_{ij}=\{\begin{array}{ll}{\epsilon }_{ij,}\hfill & \text{if}{\epsilon }_{ij}\ge -\left(j+10\right)\hfill \\ -\left(j+10\right),\hfill & \text{otherwise}\hfill \end{array}$ ${\epsilon }_{ij}\text{iid}N\left(0,100\right),�=0.8$
A5	Autocorrelated	20	50	0	$e_{ij}=�e_{i-1,j}+{\epsilon }_{ij}$ $e_{i1}~N\left(0,100/\left(1-{�}^2\right)\right)$${\epsilon }_{ij}\text{iid}N\left(0,100\right),�=0.8$
A6	Autocorrelated	20	50	0	same as A5 with $�=0.4$
A7	Periodic	20	50	$20\sin \left\{2\pi /50\left[i+\left(j-1\right)k\right]\right\}$	$e_{ij}\text{iid}N\left(0,100\right)$

Evaluation measures. We evaluated the different variance estimators with the three measures used by Wolter: expected relative bias (ERB), relative mean squared error (RMSE), and coverage ratios. The first measure, ERB, was used to examine the accuracy of the estimators and is defined for a specific estimator $\theta$ as $\text{ERB}\left({\widehat{v}}_{\theta }\right)=E_m\left(E_p\left({\widehat{v}}_{\theta }-v\right)\right)/E_m\left(v\right).$ In our notation, $E_p$ and $E_m$ refer to the design and model expectations, respectively. To examine the variance of the estimators, we also measured the RMSE, which is defined as $\text{RMSE}\left({\widehat{v}}_{\theta }\right)=E_m\left(E_p{\left({\widehat{v}}_{\theta }-v\right)}^2\right)/E_m\left(v\right).$ Coverage ratios were calculated as the percent of times the true population total fell within the confidence interval using the estimate, i.e., $\left(\widehat{Y}-z_{\alpha }\sqrt{{\widehat{v}}_{\alpha }},\widehat{Y}+z_{\alpha }\sqrt{{\widehat{v}}_{\alpha }}\right)$ . Here $z_{\alpha }$ is the value from a normal distribution and was chosen to make 95% confidence intervals.

Results. With respect to Q1, columns 4-7 of Table A1 show that increasing the number of replicates had minimum impact on the bias. Only with the linear trend population (A2) did the SDR estimator with four connected loops show a consistent trend in reduced bias as the number of replicates increased. The other population and estimator combinations showed no significant decreasing or increasing trend as the number of replicates increased. This finding is a positive result because it indicates that reducing the set of replicates does not increase the bias. As expected, the RMSEs in columns 8-11 in Table A1 did increase as the number of replicates decreased, but surprisingly the increase was relatively minor. Similarly, the confidence intervals in columns 12-15 improved with increased replicates, except with populations A2 and A7.

When comparing RA1 and RA2 of Q2, the SDR estimator with four connected loops usually had smaller biases (columns 4-7 in Table A1) and variances (columns 8-11 in Table A1) with RA1 as compared to RA2. With 16 connected loops, both the biases and variances were similar for both RA1 and RA2. This evidence suggests that both the bias and variance are improved, but the impact reduces as the size of the connected loops decreases.

Addressing Q3, the biases diminished in columns 4-7 with an increasing number of connected loops. The exception was the periodic population (A7). When the RMSEs of SD1 and SD2 were not similar as in linear trend population (A2), increasing the number of connected loops also reduced the RMSEs. This result is not surprising. The estimator with one large connected loop is equivalent to SD2, so it can have the largest biases and RMSEs due to the term ${\left({\widehat{y}}_1-{\widehat{y}}_{64}\right)}^2.$ In the other direction, more connected loops effectively reduces the impact of the term ${\left({\widehat{y}}_1-{\widehat{y}}_{64}\right)}^2,$ so the estimator acts more like SD1, which generally has less bias and variance than SD2.

Previous | Next