Suggestion of confidence interval methods for the Cronbach alpha in application to complex survey data
Section 3. Simulation

• Previous
• Next

We investigate the performance of the proposed methods in two scenarios; stratified two-stage cluster sampling and single-stage unequal probability sampling.

For stratified two-stage cluster sampling, the finite population is generated using three strata where each stratum includes 200 PSUs and 50 secondary sampling units (SSUs) totaling 30,000 SSUs. The underlying distributions that are used include the multi-normal distribution, multi-lognormal distribution and correlated ordinal data categorized from multi-lognormal distribution variables. The cases of $p=5$ and $p=10$ are considered. Different means are used for the different strata. The observations are correlated within a PSU. See the footnote of Table 3.1 for the detailed parameter information. Simple random sampling is carried out at the first-stage and second-stage, respectively, within each stratum. Thus, the appropriate weights are calculated per stratum as $\left(N_h{M}_h\right)/\left(n_h{m}_h\right)$ for each individual (SSU), where $N_h,$ $M_h,$ $n_h$ and $m_h$ are the number of PSUs per stratum, the number of SSUs per PSU, first-stage sample size per stratum, and second-stage sample size, respectively. Since the population is finite, the true value of $\alpha$ is known from the generated population.

For unequal probability sampling (Table 3.2), we generate a population of 30,000, where the underlying distributions of the data are the multi-normal distribution, multi-lognormal distribution and correlated ordinal data categorized from the multi-lognormal distribution variables similar to the cases found in Table 3.1. See the footnote of Table 3.2 for the detailed parameter information. Each individual $i$ is assigned a random number $x_i$ from the Binomial (20, 0.5) distribution, achieving the semblance of SSU sizes per PSUs. For sampling, the first-order inclusion probability is proportional to size $x_i$ (probability proportional to size sampling). Thus, the weight for an individual $i$ is obtained as $n^{-1}{\displaystyle {\sum }_k{x}_k}/x_i,$ where $n$ is the sample size. The sample selection procedure uses the systematic sampling technique that considers first-order inclusion probabilities. For the linearization method, the variance is estimated using the usual estimator for with-replacement sampling (Mach et al., 2007) as a conservative approximation of the methods for without-replacement sampling (Wolter, 1985). Since the sampling fraction is negligible in the simulation, the finite population correction is not incorporated. The 95% confidence interval is obtained based on the normal approximation.           

Table 3.1 (stratified two-stage cluster sampling) and Table 3.2 (single-stage unequal probability sampling) show the coverage rates and average widths of the confidence intervals based on the proposed linearization method and the coverage-corrected bootstrap method (1,000 simulations per scenario). The linearization method and the coverage-corrected methods are evaluated using same simulated data sets. For the coverage-corrected method, we use $B=200$ for the first bootstrap, $B=200$ for the second bootstrap. The linearized method shows the coverage rates as being close to the target confidence level for the multi-normal distributions and correlated ordinal data in most scenarios. We note that, in the random variable settings, the confidence intervals based on a normal approximation work well with various ordinal data once the variance is correctly obtained (Maydeu-Olivares, Coffman and Hartmann, 2007). Our simulation results show that the normal approximation works well with the ordinal data in finite population settings as well. When the underlying distribution is the multi-lognormal distribution, the coverage rates of the confidence intervals based on the normal approximation may be somewhat lower than the target coverage rate, but they improve with increasing sample sizes. For the multi-lognormal distribution, the coverage-corrected bootstrap method using the weight adjustment by Rao and Wu (1988) shows substantially improved coverage rates comparing to the linearized method. In comparison to the linearization method, the coverage-corrected bootstrap method has slightly increased widths, and the coverage rates are reasonably close to the target confidence level for most cases in Tables 3.1 and 3.2.

We also note that for the stratified sampling cases with relatively low $\alpha$ values, we can identify cases that the coverage-corrected method provides less-than-desirable coverage rates, with the multi-normal or ordinal data indicating that the coverage-corrected method is not a panacea for interval estimation. Here, the linearization method is a reasonable choice over the coverage-corrected method if the underlying distribution is ordinal or normal.

Thus, we conclude that, for general ordinal data, which are typical responses for most assessment instruments, the linearization method will be satisfactory to obtain the confidence intervals. When the instruments consist of continuous data and some skewed distributions are observed, the coverage-corrected bootstrap method will generally provide more accurate confidence intervals than the normal approximation.

It may be of interest to compare the performance of the proposed confidence interval methods to other existing confidence interval methods in a random variable setting since the proposed methods can be applied to these settings, as shown in (2.10). Table 3.3 presents the comparisons of the coverage rates and widths of various confidence interval methods based on the data generated from a random variable. The existing confidence interval methods can be categorized to either using an analytical distribution based on the multi-normal distribution, or using a large sample approximation for the normal distribution of $\widehat{\alpha }$ or a transformation of $\widehat{\alpha }.$ For the existing methods, we consider three normal-based confidence intervals and a bootstrap method, i.e., confidence intervals based on the exact F distribution using the normal data (van Zyl, Neudecker and Nel, 2000; Kistner and Muller, 2004), a large sample approximation of $\log \left(1-\widehat{\alpha }\right)/2$ (van Zyl et al., 2000), a large sample approximation of $\widehat{\alpha }$ based on the “distribution-free” standard error estimate (Yuan et al., 2003; Maydeu-Olivares et al., 2007), and the percentile bootstrap confidence interval with a single bootstrap (DiCiccio and Romano, 1988). These techniques are compared to the confidence intervals based on the linearization method and the coverage-corrected bootstrap method. The data are generated from the multi-normal distribution, multi-lognormal distribution, and the correlated ordinal data similar to the simulations in the previous tables. The values of $\alpha$ in Table 3.3 are for the random variables. In general, the results seem similar to those of finite population cases. The existing confidence interval methods, as well as the linearization method, perform unsatisfactorily with the lognormal data, yet their coverage rates are close to the target confidence levels using the ordinal data and normal distributions when the sample sizes increase. The coverage-corrected bootstrap method shows a coverage rate close to the confidence level with a lognormal distribution while providing wider confidence interval widths than the other methods. In the case of the multi-normal distribution, the coverage-corrected bootstrap method seems to have higher coverage rates than the target confidence level. In comparison with the single bootstrap method, the coverage-corrected method increases the coverage rates by 1 to 3% overall for the multi-lognormal distribution cases.

• Previous
• Article main page
• Next