Investigating alternative estimators for the prevalence of serious mental illness based on a two-phase sample
Section 3. The MHSS subsample

• Previous
• Next

3.1  About the MHSS subsample

The NSDUH is a stratified multi-stage probability survey. In 2008 through 2012, the MHSS subsample was drawn annually from adults responding to the corresponding NSDUH using Poisson sampling. Subsample selection probabilities were determined each year using an algorithm that tended to oversample adults with higher levels of psychological distress. The algorithm varied across the years. See Center for Behavioral Health Statistics and Quality (2014, Chapter 3) for more details.

A respondent subsample size of roughly 750 was targeted for 2008 while respondent subsamples of 500 each for the 2009 and 2010, and 1,500 each for the 2011 and 2012 were likewise targeted. A data set combining all the respondent from 2008 to 2012 was created for modeling SMI. Weights for modeling were developed assuming that the same model held across all the years. As a result, more weight was given to the samples from 2011 and 2012 than to earlier years (Center for Behavioral Health Statistics and Quality, 2014; Chapter 5).

For our purposes, we treat those subsample weights and associated NSDUH weights as given and based on survey-sampling theory. We also treat the strata and two variance primary sampling units (PSUs) per each of the 50 variance strata developed for the MHSS subsample variance estimator as if they were the NSDUH variance strata and variance PSUs. Finally, we treat the NSDUH PSUs as if they were selected with replacement.

3.2  Variance estimation under survey-sampling theory

Since the bias-corrected estimated domain totals in equations (2.9) and (2.10) are nearly unbiased under survey-sampling theory, one can use linearization to estimate their variances. In what follows, we use variants of the bias-corrected estimators in equation (2.9) and (2.10) to simplify the variance estimation.

Recalling that ${\omega }_k=0$ when $k\notin S^{\prime }\text{},$ a variance estimator for the sample mean

$${\overline{y}}_{z\left(d\right)}=\frac{{\displaystyle {\sum }_S{w}_k{z}_k{d}_k}}{{\displaystyle {\sum }_S{w}_k{d}_k}},(3.1)$$

under a stratified, multistage sample, where $z_k=p_k+\left({\omega }_k/w_k\right)\left(y_k-p_k\right)$ is

$$v\left({\overline{y}}_{z\left(d\right)}\right)=\frac{{\displaystyle {\sum }_{h=1}^{50}{\left[{\displaystyle {\sum }_{k\in S_{h1}}{w}_k{d}_k\left(z_k-{\overline{y}}_{z\left(d\right)}\right)-{\displaystyle {\sum }_{k\in S_{h2}}{w}_k{d}_k\left(z_k-{\overline{y}}_{z\left(d\right)}\right)}}\right]}^2}}{{\left({\displaystyle {\sum }_S{w}_k{d}_k}\right)}^2},(3.2)$$

where $S_{hj}$ are the respondents in the $j^{\text{th}}$ variance PSU and variance stratum $h.$ It is also a variance estimator for the following asymptotically-identical variant of the bias-corrected probability estimator:

$${\overline{y}}_{P-\text{BC}2\left(d\right)}=\frac{{\displaystyle {\sum }_S{w}_k{p}_k}{d}_k}{{\displaystyle {\sum }_S{w}_k{d}_k}}+\frac{{\displaystyle {\sum }_S{\omega }_k\left(y_k-p_k\right)d_k}}{{\displaystyle {\sum }_S{w}_k{d}_k}}.(3.3)$$

This is because the MHSS subsample is Poisson (and thus independent across adults as well as PSUs) and the first stage of the NSDUH sample is treated as if it were drawn with replacement.

Similarly, by redefining $z_k=c_k+\left({\omega }_k/w_k\right)\left(y_k-c_k\right),$ a variance estimator for the sample mean in equation (3.1) is also an estimator for that variance of this variant of the bias-corrected estimator:

$${\overline{y}}_{C-\text{BC}2\left(d\right)}=\frac{{\displaystyle {\sum }_S{w}_k{c}_k}{d}_k}{{\displaystyle {\sum }_S{w}_k{d}_k}}+\frac{{\displaystyle {\sum }_S{\omega }_k\left(y_k-c_k\right)d_k}}{{\displaystyle {\sum }_S{w}_k{d}_k}}.(3.4)$$

The variance estimation approach taken above assumes that the domain respondent subsample sizes are such that $p_k/P_k$ and $c_k/C_k$ can be treated as unity, where $P_k$ and $C_k$ are the limits of $p_k$ and $c_k\text{},$ respectively, as the subsample (along with the NSDUH sample and population) grows arbitrarily large. In fact, all these ratios are assumed to be $1+{\text{O}}_P\left(1/\sqrt{n}\right),$ where $n$ is the MHSS subsample size.

Consider now a computed bias-correction term, say ${\displaystyle {\sum }_S{\omega }_k\left(y_k-p_k\right)d_k}/{\displaystyle {\sum }_S{w}_k{d}_k}$ or ${\displaystyle {\sum }_S{\omega }_k\left(y_k-c_k\right)d_k}/{\displaystyle {\sum }_S{w}_k{d}_k}\text{}.$ To assess whether the term is significantly different from zero, one can create an asymptotic $t-$ statistic in the usual fashion, dividing the term by its standard error.

When evaluating the estimators in Section 3.3, we will instead use the asymptotically equivalent:

$$BiasMeasure\left({\overline{y}}_{P\left(d\right)}\right)={\displaystyle {\sum }_{S^{{}^{\prime }}}{\omega }_k\left(y_k-p_k\right)d_k}/{\displaystyle {\sum }_{S^{{}^{\prime }}}{\omega }_k{d}_k}\text{},(3.5)$$

and

$$BiasMeasure\left({\overline{y}}_{C\left(d\right)}\right)={\displaystyle {\sum }_{S^{{}^{\prime }}}{\omega }_k\left(y_k-c_k\right)d_k}/{\displaystyle {\sum }_{S^{{}^{\prime }}}{\omega }_k{d}_k}(3.6)$$

to create asymptotic $t-$ statistics for evaluating domain-level biases so that the DESCRIPT procedure in SUDAAN (RTI International, 2012) can be employed treating the $p_k$ and $c_k$ as fixed (similarly, that variance estimator for (3.3) in equation (3.2) can be computed using DESCRIPT). Moreover, since virtually all the sampling error in the bias-correction terms comes from the MHSS subsampling phase (even in 2011 and 2012, subsample was only 3% of the NSDUH adult sample), we treat the standard errors of the bias measures as if they were computed for a Poisson sample with ignorably small sampling fractions, which is equivalent to a with-replacement element sample for variance-estimation purposes. For example, for the variance estimator of $BiasMeasure\left({\overline{y}}_{P\left(d\right)}\right),$ we compute (using SUDAAN’s DESCRIPT): ${v\left[BiasMeasure\left({\overline{y}}_{P(d)}\right)\right]=\frac{n}{n-1}{\displaystyle {\sum }_{S^{{}^{\prime }}}{\left[{\omega }_k\left\{\left[y_k-p_k\right]-BiasMeasure\left({\overline{y}}_{P\left(d\right)}\right)\right\}{d}_k\right]}^2}/\left({\displaystyle {\sum }_{S^{{}^{\prime }}}{\omega }_k{d}_k}\right)}^2\text{},$ where $n$ is the sample size of $S^{\prime }\text{}.$

3.3  Evaluating the estimators

The model used by SAMHSA to predict SMI from adult NSDUH respondents was a logistic model with five variables (Center for Behavioral Health Statistics and Quality, 2014; Chapter 7). Two of the variables were rescaled total scores from short forms that measure psychological distress and functional impairment due to distress. The third was a dichotomous $\left(0/1\right)$ variable created from the answers to a series of questions assessing whether the respondent had a major depressive episode in the previous year. The fourth was also $0/1$ and indicated whether the respondent seriously contemplated suicide in the past year, and the fifth was a linear function of age from 18 to 30 that stayed constant after 30. Details on how this model was selected can be found in Center for Behavioral Health Statistics and Quality (2015, Chapter 4).

We used that model to create a set of domain-level cut point and probability estimates from the combined 2008-2012 data sets and to evaluate their potential biases. Some of the results are displayed in Tables 3.1 and 3.2. These tables reviewed domain estimates based on personal characteristics rather than state of residence because it seemed more likely that significant biases would be found for the characteristics like these rather than for states. Moreover, sample sizes for characteristics tended to be larger than those for states.

Table 3.1 show that using the bias-corrected probability in equation (2.9) is usually slightly more efficient (has a smaller standard error) than the direct estimator ${\overline{y}}_{U\left(d\right)}={\displaystyle {\sum }_{S^{{}^{\prime }}}{\omega }_k{d}_k{y}_k}/{\displaystyle {\sum }_{S^{{}^{\prime }}}{\omega }_k{d}_k}\text{}.$ The bias-corrected cut point estimator in equation (2.10) is sometimes more efficient than the direct estimator, sometimes not. The standard errors in Table 3.1 are the square roots of linearization variance estimators for the direct estimator ${\overline{y}}_{U\left(d\right)}$ above or the bias-corrected estimator in equation (3.1) with the appropriated defined nonrandom $z_k\text{},$ each computed as a stratified with-replacement sample of primary sampling units and a probability subsample of individuals within each PSU; that is, with equation (3.2). For $v\left({\overline{y}}_{U\left(d\right)}\right),$ $z_k-{\overline{y}}_{z\left(d\right)}$ is replaced by $y_k-{\overline{y}}_{U\left(d\right)}\text{}.$

The table suggests that there is little gain to be had from model correction, and leads us back to the model-driven probability and cut point estimators in equations (2.7) and (2.8) unless they exhibit systematic biases. Table 3.2 (with bias measures and their standard errors computed as described in Section 3.2) strongly suggests that the probability estimator, although unbiased when estimating SMI prevalence among all adults, can be very biased at the domain level. The cut point estimator, by contrast, is significantly biased at the 0.1 level in only two domains and never at the 0.05 level. Since we computed two-sided $p-$ values for 32 domains, finding two domains with $p-$ values below 0.1 is about what one should expect under the null hypothesis that the cut point estimator is not biased at the domain level.

One curious result bears a brief mention. The cut point estimator among all adults had very little bias (-0.01), so its estimated root mean squared error equaled the standard error of the bias-corrected cut point after rounding (0.26). Oddly, this value was less than the standard error of its bias measure (0.27). One possible reason for the difference between the two standard errors was that we used ${\displaystyle {\sum }_S{\omega }_k\left(y_k-c_k\right)d_k}/{\displaystyle {\sum }_S{w}_k{d}_k}$ as the bias-correction term and ${\displaystyle {\sum }_S{\omega }_k\left(y_k-c_k\right)d_k}/{\displaystyle {\sum }_S{\omega }_k{d}_k}$ as the bias measure within a domain; all adults being the special case where $d_k\equiv 1.$ Our analysis (not shown) was that the difference in the denominators had very little impact.

What has a greater impact was ignoring the stratification and clustering in the NSDUH sample when computing the standard errors of the bias measures. Unexpectedly, ignoring the clustering actually tended to increase standard errors. This may be because the clustering in the NSDUH has virtually no measurable impact on variance so that any difference between standard error estimates computed with and without clustering is attributable to random noise or to asymptotic biases that are not actually ignorable in finite estimates.

3.4  A hybrid cut point

Consider the following hybrid of the probability and standard cut point estimators. Suppose we sorted the NSDUH sample rather than just the MHSS subsample by the fitted $p_k$ values, and established a cut point $p_H$ such that

$${\displaystyle \sum _{\begin{array}{c}k\in S\\ p_k\ge p_H\end{array}}{w}_k}={\displaystyle \sum _{k\in S}{w}_k{p}_k}(3.7)$$

holds as closely as possible. Setting $h_k=1$ when $p_k>p_H$ and 0 otherwise, the hybrid cut point estimator for SMI prevalence in a domain is

$${\overline{y}}_{H\left(d\right)}=\frac{{\displaystyle {\sum }_S{w}_k{h}_k{d}_k}}{{\displaystyle {\sum }_S{w}_k{d}_k}}.(3.8)$$

It is not hard to see that for all adults, if a $p_H$ could be found that satisfied equation (3.7), then the hybrid cut point estimator would equal the probability estimator exactly. Failing that the hybrid cut point estimator for all adults would have a slight bias, which could be measured, squared, and then added to the standard error of the probability estimator to equal its root mean squared error. In this case, the hybrid SMI prevalence estimate for all adults rounded to 3.89. Its root mean squared error rounded to the same value as the standard error of the probability estimator (0.23).

Table 3.3 repeats much of Table 3.2 for the standard cut point but also displays analogous results for the hybrid

$$\left(BiasMeasure\left({\overline{y}}_{H\left(d\right)}\right)\right)={\displaystyle {\sum }_S{\omega }_k\left(y_k-h_k\right)d_k}/{\displaystyle {\sum }_S{\omega }_k{d}_k}.(3.9)$$

Its standard error is computed analogously to those of ${\overline{y}}_{C\left(d\right)}$ and ${\overline{y}}_{P\left(d\right)}\text{}.$ The two sets of cut point outcomes are similar, but the bias measure for the hybrid estimator was significantly different from zero at the 0.05 level in two domains (both with $p-$ values of 0.043). Since there are 32 domains analyzed, this remains consistent with the null hypothesis of no bias at the domain level.

• Previous
• Article main page
• Next