6. Simulation studies

Qi Dong, Michael R. Elliott and Trivellore E. Raghunathan

Previous | Next

In this section, we conduct two simulation studies to evaluate the repeated sampling properties of the population estimators constructed using the nonparametric method that generates synthetic populations while adjusting for the complex sampling design features. The first of these considers a one-stage, unequal probability of selection design where we vary the number of weighted FPBB draws for each synthetic population and the number of synthetic populations to assess the impact on inference. The second compares inferential properties from observed data and from the posterior distribution obtained from synthetic population in a stratified, multistage, unequal probability of selection sample, this time fixing the posterior sample size while considering both population means and population regression parameters as targets of inferences.

6.1 Single stage, unequal probability of selection sample design

We generated outcome data $Y$ in a population of $N$ subjects from a moderately skewed gamma distribution, conditional on uniformly distributed covariate $X:$

$$\begin{array}{c}{X}_i~\text{UNI}\left(0.05;0.65\right),i=1,\dots ,N\\ Y_i|X_i=x_i~\text{GAMMA}\left(10*x_i,1\right)\end{array}$$

We assume $X$ is fully observed for the population, and that the probability of selection $\pi$ is proportional to $X,$ so that ${\pi }_i\doteq n{x}_i/{\displaystyle {\sum }_i{x}_i}$  in a without-replacement sample design as long as $n<<N.$ The estimand of interest is the population mean $\overline{Y}=N^{-1}{\displaystyle {\sum }_{i=1}^N{y}_i}=$ 3.564. Note that $\text{corr}\left(Y_i,X_i\right)=$ 0.6794, so that unweighted sample means will be positively biased, and use of design weights $w_i=1/{\pi }_i$ are required to obtained unbiased estimates of $\overline{Y}.$ We generated a population of size $N=$ 1,000 from which we sampled $n=100;$ bias, empirical and estimated variance, 95% interval length, and nominal 95% coverage are then estimated from 200 independent samples from the population. We varied the total number of simulated populations $L$ as 5, 20, 100, and 1,000, and the number of FPBB draws $F$ of size $N-n$ (so that $K=9$ ) as 1, 20, and 100, in full factorial design. Variance, interval length, and interval coverage are obtained via the normal approximation; for $L=$ 100 and 1,000, we also obtained variance, interval length, and interval coverage using the direct draws from the posterior predictive distribution, since a sufficient number of draws from the posterior were available to make such estimates.

Table 6.1 shows the results of the simulation study. In all cases the point estimate ${\overline{Q}}_L$ of the population mean was approximately unbiased, reflecting the ability of the weighted FPBB to "undo� the sampling weights in the generation of the synthetic population. Under the normal approximation, larger numbers of the synthetic population were associated with smaller variances and narrower interval lengths, as expected with larger numbers of degrees of freedom, although the difference between 20 and 100 was minimal, just as the $t_{20}$ distribution begins to approximate a standard normal. Finally, using only a single FPBB draw of size $N-n$ appeared to overestimate the variance and lead to overcoverage, especially for small values of $L$. Values of $L$ and $F$ of 20 or greater appeared to yield reasonable results. Use of the direct draws for $L=$  100 and 1,000 yielded to variance and credible interval estimates that were very similar to that of the normal approximation, with slightly narrower interval lengths and somewhat less conservative coverage.

6.2  Stratified, multistage, unequal probability of selection sample design

We generated a population with strata and clusters within each stratum from the following bivariate normal distribution:

$$\left(\begin{array}{c}{X}_{1ijk}\\ X_{2ijk}\end{array}\right)~N\left(\left(\begin{array}{c}500+4.5\text{*}i+u_{ij}\\ 500+4.5\text{*}i+u_{ij}\end{array}\right),\left(\begin{array}{cc}100& 50\\ 50& 100\end{array}\right)\right),$$

where

$i=1:150$ denotes the stratum effect,

$u_{ij}~N\left(0,10\right)$ denotes the random cluster effect,

$a_i~\text{uniform}\left(2,52\right)$ is the number of clusters within stratum $i,$

$b_{ij}~\text{uniform}\left(10,20\right)$ is the number of units within cluster $j$ of stratum $i.$

The population for the simulation study has 61,324 subjects. We draw a stratified clustering sampling with unequal probabilities of selection. Specifically, we select two clusters from each stratum with probabilities proportional to cluster size (PPS) given by $b_{i·}={\displaystyle {\sum }_{j=1}^{a_i}{b}_{ij}}.$ Within each selected cluster, we select approximately $1/5$ of the population. Thus, the probability that unit $ij$ is selected is given by

$${\pi }_{ij}=\frac{2b_{i·}}{{\displaystyle {\sum }_{j=1}^{a_i}{b}_{ij}}}\times \frac{\lfloor b_{ij}/5\rfloor }{b_{ij}}$$

for all $j$ elements in cluster $i$ with corresponding weight

$$w_{ij}=\frac{b_{ij}{\displaystyle {\sum }_{j=1}^{a_i}{b}_{ij}}}{2b_{i·}\lfloor b_{ij}/5\rfloor }\text{.}$$

Since the number of clusters and units are random, the complex sample size is slightly different across replications, averaging approximately 770.

Because of the large sample and population size, we focus on inference using $t$ approximations. We generate $L=$ 100 synthetic populations using $F$ weighted FPBB samples of size $K=100n.$ The estimands of interest are the population marginal mean for $x_1$

$${\overline{X}}_1=N^{-1}{\displaystyle \sum _{i=1}^N{X}_{1i}}$$

and similarly for $x_2,$ and the population regression coefficients of $x_1$ on $x_2$ given by

$$B_0={\overline{X}}_1-B_1{\overline{X}}_2,B_1=\frac{{\displaystyle {\sum }_{i=1}^N\left(X_{1i}-{\overline{X}}_1\right)\left(X_{2i}-{\overline{X}}_2\right)}}{{\displaystyle {\sum }_{i=1}^N{\left(X_{2i}-{\overline{X}}_2\right)}^2}}.$$

We drew 200 independent samples from the population and used the sample data directly to compute weighted sample means and linear regression coefficients along with associated variance estimates and 95% nominal confidence intervals using Taylor Series approximations, and compared these with the equivalent estimates obtained using the nonparametric synthetic data. Results are given in Table 6.2. (Since the marginal means have the same superpopulation value, we combine the results in Table 6.2.) Figure 6.1 displays the scatter plot of the pairs of estimated mean, intercept and slope from the actual samples and the corresponding synthetic populations along with a 45-degee line. The sampling distributions of the actual sample and synthetic population estimates closely correspond. The point estimates and standard errors for both the means and regression parameters closely correspond. The 95% confidence interval coverage rates for all three statistics also closely correspond, and are close to nominal values.

Figure 6.1 Scatter plot of the descriptive and analytic statistics from the actual and synthetic populations

Description for figure 6.1

Previous | Next