Suggestion of confidence interval methods for the Cronbach alpha in application to complex survey data
Section 4. Application

• Previous
• Next

In this section, we provide detailed information regarding the NCS-R survey and subgroup analysis using the data sets. The relevance of the instruments may vary based on the different demographic groups studied, and thus a relatively low reliability in a certain group would be an indication that the instrument items may need some adjustments for that group. Using the data from the NCS-R, we investigate the changes of $\alpha$ using the Kessler 10 (K10, Kessler, Andrews, Colpe, Hiripi, Mroczek, Normand, Walters and Zaslavsky, 2002), the Kessler 6 (K6, Kessler et al., 2002) and the Sheehan Disability Scale (SDS, Sheehan, Harnett-Sheehan and Raj, 1996). More details about these scales are explained in Section 4.1.

4.1  The data

The NCS-R is a mental health survey for a nationally representative sample of English-speaking noninstitutionalized household residents in the United States (Kessler et al., 2004) and it uses the fully structured World Health Organization’s (WHO) World Mental Health Survey version of the Composite International Diagnostic Interview (WMH-CIDI) (Byers, Yaffe, Covinsky, Friedman and Bruce, 2010). Using computer-assisted personal interviews, the NCS-R was carried out to obtain further information not fully covered in the previous baseline National Comorbidity Survey (NCS). A total of 9,282 participants 18 years and older completed the Part I interview, and a subsample of 5,692 participants completed the Part II instruments. The data sets are publicly accessible and downloadable on the ICPSR (Inter-university Consortium for Political and Social Research) website (https://www.icpsr.umich.edu/icpsrweb). The NCS-R is based on a stratified multi-stage probability sample design (42 strata where each stratum has two PSUs, totaling 84 PSUs), and the sample weights are provided in the data to reflect the survey design. Each PSU consists of metropolitan statistical areas or counties (Kessler et al., 2004). The final weights in the NCS-R data are adjusted for nonresponses to the survey instruments. Weights accounting for the designs of the different parts of the surveys (i.e., Parts I and II) are provided, respectively, in the NCS-R data. The weights are normalized to have a sum equal to 9,282 for Part I and 5,692 for Part II (mean weight = 1), respectively. In this case, the weights do not represent the inverse of the selection probabilities. Due to this and the fact that the sample size is quite small compared to the total population of interest, the finite population correction is not considered in the data analysis. Incorporating these weights corrects the overrepresentation of “racial minorities, females, residents of the Midwest, people with 13+ years of education, and residents of metropolitan areas” (Kessler et al., 2004).

The 10-item Kessler psychological distress scale or the K10 is an instrument used to assess the distress level of people (Kessler et al., 2002), and the K6 is an abbreviated set of six items from the K10. Both the K10 and K6 are considered effective scales for screening mental disorders (Brouwer, Cornelius, van der Klink and Groothoff, 2013). The K10 for 30-day symptoms is included in the Part II instruments. It is composed of 10 questions of a self-reported assessment of psychological distresses in the worst month of the past year for each interviewee. The questions ask feelings such as tiredness, nervousness, hopelessness, and so forth. All 10 questions produce an ordinal data scoring of 1 (all of the time) to 5 (none of the time). The final total score ranges from 10 to 50 with the higher scores showing more distress. The K10 values in the NCS-R have missing data, and the weights given by the NCS-R adjust for survey nonresponses, but they do not adjust for items with missing data. Although these missing data may compromise the unbiasedness of the weighted estimation (Alegria, Jackson, Kessler and Takeuchi, 2007), we use only completed data and remedial approaches such as weighting class adjustment or imputation of the data are not considered in our analysis.

The SDS assesses functional impairment associated with mental disorders (Sheehan et al., 1996). The SDS in the NCS-R assesses disorder-specific role impairments (Sheehan et al., 1996; Druss, Hwang, Petukhova, Sampson, Wang and Kessler, 2009). It consists of four questions evaluating the disruption of activities associated with home, work, social and close relationship using 0 to 10 scales, with higher scores showing more severe impairment. In this paper, among the SDS scales of various mental disorders, we use the SDS for the participants with chronic conditions as a Part II instrument. Since the SDS is disorder-specific, it has missing data. For the data analysis, we use only complete data.

4.2  Subsample analysis

For the subgroups, a domain analysis may be applied. Suppose that a domain indicator function $I_k^d$ $\left(d=1,\dots ,D\right)$ has a value of 1 if the unit $k$ is in a domain $d$ (i.e., $k\in s_d)$ and 0 otherwise. Then, the statistics of the domain are estimated by modifying the weight as $w_k^{\left(d\right)}=w_k{I}_k^d.$ The procedures used to obtain the estimates and the corresponding variance or covariance are carried out with the modified weights. Since the sample size is not fixed but is rather treated as an estimate, an estimator such as the sample mean and sample variance can be considered as the ratio estimator, i.e., both the numerator and the denominator are estimated, and the variance of the estimator is obtained accordingly. However, when the sample size is large, thus the ratio between the domain sample size and the whole sample size is close to the true population ratio, it is known that the variance of the ratio estimator is approximately the same as that of the estimator with the fixed sample size using only the subgroup of interest, making “little difference in practice” regarding those estimators (Lohr, 1999, page 79). The negligible difference between the domain estimator and the estimator using only the subsample can be easily shown using the variance estimator in an unequal probability sampling with replacement setting. Let ${\widehat{\overline{Y}}}_d$ indicate the domain estimator of the mean (Lohr, 1999) for single-stage sampling, i.e., ${\widehat{\overline{Y}}}_d={\displaystyle {\sum }_{k=1}^n{w}_k{I}_k^d{y}_k}/{\displaystyle {\sum }_{k=1}^n{w}_k{I}_k^d}={\displaystyle {\sum }_{k\in s_d}{w}_k{y}_k}/{\displaystyle {\sum }_{k\in s_d}{w}_k},$ where the last term uses only the subsample. Now, for the variance estimator of ${\widehat{\overline{Y}}}_d$ (Paben, 1999; SAS/STAT user’s guide, 2010), we can show

$$\widehat{V}\left({\widehat{\overline{Y}}}_d\right)={\displaystyle \sum _{k=1}^n{\left\{\frac{w_k{I}_d\left(y_k-{\widehat{\overline{Y}}}_d\right)}{{\displaystyle {\sum }_{l=1}^n{w}_l{I}_d}}\right\}}^2}\frac{n}{n-1}\simeq {\displaystyle \sum _{k\in s_d}{\left\{\frac{w_k\left(y_k-{\widehat{\overline{Y}}}_d\right)}{{\widehat{N}}_d}\right\}}^2}\frac{n_d}{n_d-1},(4.1)$$

where $n_d$ is the sample size of $s_d.$ Here, the right-hand side of equation (4.1) uses the observation only in domain $s_d.$ Based on this fact, the variance for a subgroup is obtained based only on the data from the subgroup of interest in this paper.

When implementing the bootstrap method, we use $n_h^{*}=2,$ which produces all the positive weights in (2.12). In the subsample analysis, the bootstrap sample may contain only one PSU per stratum. In this case, the variance cannot be estimated. If we have multiple strata with one PSU, we combine those strata. If we have only one stratum with one PSU, we merge that stratum with another stratum arbitrarily. The rationale of this practice is that the variance incorporating strata is usually smaller than that without strata, thus such a practice may produce a wider (more conservative) confidence interval.

4.3  Results

The estimates of $\alpha$ and their confidence intervals for the whole participants are shown in Table 4.1. The table presents the confidence intervals using the coverage-corrected percentile method and the confidence interval using the linearization method for each instrument. Between the K10 and K6, it appears that the K10 has a higher $\alpha$ estimate. This may be explained by the fact that the removed items from the K10 are highly correlated with the remaining items in the K6, thus removing these items results in a reduced $\widehat{\alpha }$ value. The coverage-corrected percentile method shows confidence intervals that are close to the linearization method, while slightly wider. Considering the ease of calculation, when an analysis deals with instruments with ordinal data, the results of the similar confidence intervals in Table 4.1 may indicate that a normal approximation using the proper variance estimation may be satisfactory for the investigated instruments, which do not include the skewed continuous data that we examined in Tables 3.1 and 3.2.

The subgroup analysis is shown in Table 4.2, where $\widehat{\alpha }$ and the confidence intervals are presented for different groups by age, gender and marriage status. The age groups are defined as young (34 years and under), middle aged (35-64 years), and old aged (65 years and over) per the available literature (e.g., Sunderland, Hobbs, Anderson and Andrews, 2012), where the cut-off points for the age groups are decided by epidemiological studies and the traditional definition of old age. The marriage status is defined by grouping married and unmarried (including divorced, separated, widowed and never married). Both the coverage-corrected bootstrap method and the linearization method provide comparable confidence intervals while the coverage-corrected bootstrap produces a slightly wider confidence interval. Considering that the coverage-corrected method is computationally intensive, the linearization method may be preferred when the instruments consist of ordinal scales.

To this end, we conclude this section with a discussion of the results of the subgroups. Sizable differences in $\widehat{\alpha }$ between the groups are found in the age groups with the K10 and K6 and marital status in the SDS. There are no overlaps of the confidence intervals between the middle and old-age groups in the K10 and K6. This indicates that the questions in the K10 and K6 may be relatively less consistent among the old-age group than the middle-age group. For the SDS, there is also no overlap of the confidence intervals between the married and the unmarried groups. That is, the consistency of the questions is substantially lower for the unmarried group than for the married group. We speculate that the SDS items include the impairment of a certain area that may be more relevant to the married group than the unmarried group (e.g., a disruption of activities associated with home, work, social and close relationship).

• Previous
• Article main page
• Next