View the most recent version.

Findings

The National Longitudinal Survey of Children and Youth (NLSCY) is a federally sponsored, national prospective study designed to measure the well-being, health and development of Canadian children from birth through young adulthood. The survey began in 1994/1995, and data collection has occurred at two-year intervals since then. As part of the interview, the parent (usually the biological mother) was asked to describe the child’s behaviour using the Children’s Behaviour Scale.

The entire scale is composed of several subscales, one of which, the Hyperactivity/Inattention Subscale (H/I Scale), is designed to identify hyperactive, inattentive and impulsive behaviours in children aged 4 to 11 in large, population-based studies. The items were taken from the Ontario Child Health Study¹ and the Montreal Longitudinal Study.²

Several researchers have used high scores on this scale as a proxy for clinically significant symptoms often identified with Attention Deficit Hyperactivity Disorder (ADHD).^3-5 However, comparisons across studies are hampered by the lack of consistency in classifying children likely to have a clinically significant disorder such as ADHD. Two studies where the scale has been dichotomized to distinguish children with significant difficulties used thresholds of 1.5 standard deviations above the population mean,^3,5 and another used the top 10%.⁴

This article evaluates the parent-reported NLSCY H/I Scale with data from cycle 1 (1994/1995) of the survey. The NLSCY H/I Scale is based on the Ontario Child Health Study Survey Diagnostic Instrument (OCHS SDI)¹ hyperactivity scale, which was validated against Diagnostic and Statistical Manual of Mental Disorders, Third Edition (DSM-III) diagnosis of ADHD, and used a combination of parent- and teacher-reports for case identification. However, DSM criteria for ADHD have been substantially revised since DSM-III was published.⁶ Nonetheless, since DSM-II,⁷ the underlying conceptualization has remained a long-standing childhood disorder characterized by detrimental levels of overactivity, impulsiveness and distractibility, and a short attention span.

In addition to changes in diagnostic criteria over time, another reason for evaluating how well the parent-reported NLSCY H/I Scale identifies children at risk of ADHD is that 52% of the teacher-reported information was missing in cycle 1.⁸ To determine whether the child showed symptomatic behaviours in more than one context (at home and at school), the OCHS SDI required that both parent and teacher rate the child. However, the lack of teacher responses for slightly more than half the NLSCY participants substantially undermines this method of identifying cases in the NLSCY data.

The method in the present study is based on the work of Boyle et al.,⁹ who recommended that survey instruments designed for population studies incorporate “elements of distress, impairment and therapeutic concern” in defining a case, rather than simply applying a threshold number of symptoms.⁹ In 1999, Goodman demonstrated that a measure of child “impact” that combined “distress” and “social impairment” improved case identification, compared with parent and teacher ratings alone.¹⁰

The NLSCY database contains two variables that represent these elements: current use of methylphenidate (Ritalin), which is used almost exclusively to treat childhood ADHD,^11,12 and previous diagnosis of emotional, psychological or nervous disorder. In each instance, the child’s parent would have sought professional assistance. In the first, before prescribing methylphenidate, a physician concurred that the child required treatment, and in the second, for the child to have a diagnosed emotional disorder, a health professional perceived enough impairment to warrant diagnosis and treatment

To some extent, the groups of children that these two variables identify may overlap. However, one represents a narrow, and the other, a broad, conceptualization of disorder. In 1994/1995, methylphenidate accounted for the vast majority of stimulant medications prescribed to children,^11,12 and was likely to be a marker for ADHD, but not for other disorders. In addition, the NLSCY question specified current medication use.¹³ By contrast, a diagnosis of emotional, mental or nervous disorder could apply to several conditions, only one of which might be ADHD. As well, the diagnosis was not specific to the time of the interview.

The NLSCY includes items about functional impairment (academic performance, getting along with peers, and getting along with parents), the third element identified by Boyle et al. As was done by Boyle et al., this study combined these items to indicate impaired functioning in at least one domain.⁹

Another consideration in developing a model to evaluate the H/I Scale is the sensitivity and specificity of the threshold for case identification. Sensitivity is the percentage of cases that the threshold identifies as positive that truly have the disorder (true positives/true positives + false negatives). Specificity is the percentage of cases that the threshold identifies as negative that do not have the disorder (true negatives/true negatives + false positives).¹⁴ For diagnostic screening, the ideal threshold maximizes both sensitivity and specificity, although false positives and false negatives may be common. But when a scale is used to determine the prevalence of a relatively rare disorder in a large non-clinical population, priority should go to minimizing the error rate.¹⁵

Like other measures designed for use in population samples, the NLSCY H/I Scale is abbreviated. Such scales are highly sensitive, but not very specific. A threshold chosen to balance sensitivity and specificity would yield excessively high rates of false positives in population samples. The overall error rate is lowest when the threshold is set where the numbers of false positives and false negatives are closest to equivalent. This results in greater specificity and less sensitivity, the strategy chosen for this study because of the relatively low prevalence of ADHD in the population.

The goal is to develop a model for evaluating the NLSCY H/I Scale so that it can be used to identify children with clinically significant ADHD symptoms in large population-based studies. The model is tested with 1994/1995 data, the cycle with the most complete information and that has not been subject to attrition over time. The objectives are to: 1) evaluate the strength of the association between scores on the NLSCY H/I Scale and each of three potential criteria for ADHD: current methylphenidate use, diagnosis of an emotional disorder, and functional impairment; 2) identify the criterion with the strongest association, adjusting for age, sex and socio-economic status; and 3) identify the threshold with the most nearly equal false negatives and false positives. The point prevalence of clinically significant ADHD symptoms among Canadian children is also estimated.

Methods

Sample

The NLSCY used a random sampling frame of households with clusters within age groups and large geographic areas to be representative of children in the 10 provinces. Children in highly mobile, transient or homeless families were under-represented. Children living in institutions and on Aboriginal reserves were excluded. A full description of the NLSCY is available elsewhere.¹⁶

In each household, Statistics Canada interviewers administered a standardized questionnaire to the person most knowledgeable about the child (the biological mother in 89.9% of cases). (In this study, the term “mother” or “parent” is used rather than person most knowledgeable because the NLSCY H/I Scale was designed to be parent-reported.) The overall response rate was 87%. The population analysed is the subset of the NLSCY sample consisting of children aged 6 to 11 in 1994/1995 whose parent responded to the interview—a total of 10,498, representing 2.36 million children across Canada.

The statistical models were developed based on a training sample derived using a replicate sampling with replacement strategy,^17,18 followed by testing the statistical model in the NLSCY sample. A random half-sample for model development was not feasible because of confidentiality constraints imposed by the small number of respondents scoring positive for the clinical indicators. The training sample was produced by aggregating 10 replicate random samples, each equivalent to 10% of the NLSCY sample, a strategy similar to a simplified bootstrap^17,18 procedure. The resulting sample is comparable to the NLSCY sample (Table 1). The final models reported here are those evaluated in the NLSCY sample, specifically the NLSCY cycle 1 subsample of children whose parents answered the interview questions.

Table 1
Selected characteristics of training sample and National Longitudinal Survey of Children and Youth (NLSCY) sample, household population aged 6 to 11, Canada excluding territories, 1994/1995

Measures

The Hyperactivity/Inattention Subscale of the parent-reported NLSCY Children’s Behaviour Scale consists of 8 items (can’t sit still, distractible, fidgets, impulsive, difficulty sitting still, cannot settle for long, can’t concentrate, inattentive) scored as 0 (not true), 1 (sometimes true) or 2 (often true), resulting in a continuous scale with scores from 0 to 16. Internal consistency on factor analysis is good (Cronbach’s α =0.86).⁸

The covariates are the child’s age and sex, low maternal education (did not complete secondary school) and low household income, based on Statistics Canada’s derived variable of household size and income (below the 1995 low income cut-off).¹⁹

Three clinical indicators reported by the parent were evaluated as potential criteria for validity:

Current methylphenidate use: “Does … (your child) … take any of the following medications on a regular basis … Ritalin?” (Yes/No)

Previous diagnosis of emotional disorder: “ Does … (your child) … have any of the following long-term conditions that have been diagnosed by a health professional? … Emotional, psychological or nervous disorder?” (Yes/No)

Impairment in academic, social or family functioning:

• “Based on your knowledge of his/her school work, including report cards,… how is he/she doing overall?”
• “During the past six months, how well has … (your child) … gotten along with other kids such as friends or classmates (excluding brothers or sisters)?”
• “How well has … (your child) … gotten along with his/her parents?”

Parents evaluated each of the three areas on a 5-point scale; scores of 4 or more (poor or very poor functioning) on any of the three scales indicated functional impairment in one or more areas.⁹

Data analysis

The research design is a retrospective cross-sectional analysis. Logistic regression analysis using backward selection was applied to the training sample to measure the association of the H/I Scale against each of the potential criterion variables, adjusted for age, sex, low maternal education and low household income. The regression models included cross-sectional population weights.^8,20,21 Only independent variables with a statistical significance of p < 0.01 were retained in the final models. Best-fit statistical models were chosen using the –2 Log Likelihood statistic and the Hosmer-Lemeshow goodness-of-fit test. Receiver Operating Characteristic (ROC) curves were examined to identify the model with the greatest area under the curve. Sensitivity analyses throughout the full range of scores were used to identify the threshold scores with the most nearly equivalent false negatives and false positives. Frequency estimates were normalized to adjust for missing values, and reported with cross-sectional population weights and variance estimates.8,20 The criterion variable of choice was the one with the greatest area under the ROC curve, with the largest beta, and whose threshold has the highest specificity.

After the preferred criterion variable(s) were determined, the statistical models were tested in the NLSCY sample. Using the threshold score identified during model development, the association of the binary NLSCY H/I Scale was measured against the chosen criteria in the NLSCY sample (including cross-sectional population weights) to understand the properties of case identification. In addition, an estimate of the population prevalence of clinically significant ADHD symptoms was generated. All analyses were performed using SAS version 8.2.²²

Results

Scores on the parent-reported NLSCY H/I Scale was associated with each of the three clinical indicators: current methylphenidate use; previous diagnosis of emotional disorder; and impaired functioning in academic, social or family domains among children aged 6 to 11 (Table 2). Statistical overlap among the three models was substantial, with beta values ranging narrowly from 0.29 (SE = 0.01) for impaired functioning to 0.31 (SE = 0.02) for emotional disorder.

Table 2
Logistic regression models of current methylphenidate use, diagnosed emotional disorder and functional impairment, household population aged 6 to 11, Canada excluding territories, 1994/1995

The sensitivity analyses and resulting ROC curves show that methylphenidate use and emotional disorder produced essentially the same statistical model—that is, a model with a greater area under the ROC curve and higher specificity at the identified threshold compared with impaired functioning (Table 3, Figure 1). With either methylphenidate use or emotional disorder as the criterion, a H/I Scale threshold of 14 or more out of 16 distinguished cases from non-cases. With methylphenidate use as the criterion, sensitivity = 0.25 and specificity = 0.98; with emotional disorder, sensitivity = 0.23 and specificity = 0.98. This threshold resulted in highly specific, but not very sensitive, case identification.

Table 3
Sensitivity and specificity values for threshold on National Longitudinal Survey of Children and Youth Hyperactivity/Inattention Subscale, by current methylphenidate use, diagnosed emotional disorder and functional impairment, household population aged 6 to 11, Canada excluding territories, 1994/1995

Figure 1
Receiver Operating Characteristic (ROC) curves from sensitivity analyses for National Longitudinal Survey of Children and Youth Hyperactivity/Inattention Subscale plotted against current methylphenidate use, diagnosed emotional disorder and functional impairment, household population aged 6 to 11, Canada excluding territories, 1994/1995

The logistic regression models for the continuous scale highlight similarities and differences between the models (Table 2). If methylphenidate use is the criterion, the best-fit model for the continuous H/I Scale includes age and sex as modifiers, with older children and boys more likely to be taking methylphenidate. If emotional disorder is the criterion, the best-fit model for the continuous H/I Scale includes age and low household income, with older children and those from low-income households more likely to have been diagnosed.

To better understand the properties of case identification using the H/I Scale, the association of the binary variable with methylphenidate use and with emotional disorder was examined (data not shown). Children using methylphenidate were more likely to: show high rather than low levels of parent-reported hyperactivity and inattention (OR = 16.2; 99% CI = 11.2 to 23.5); be boys rather than girls (OR = 3.8; 99% CI = 2.6 to 5.4); and be near the older end of the 6 to 11 age range (OR = 1.1; CI = 1.0 to 1.2). Children with a previous diagnosis of emotional, psychological or nervous disorder were more likely to: show high levels of parent-reported hyperactivity and inattention (OR = 16.9; CI = 11.3 to 25.2); come from households with a low income (OR = 2.2; CI = 1.6 to 3.1), and be near the older end of the age range (OR = 1. 3; CI = 1.2 to 1.4).

Based on a threshold of 14 or more on the NLSCY H/I Scale, an estimated 2.1% (99% CI = 1.5 to 2.7) of Canadian children aged 6 to 11 had clinically significant ADHD symptoms.

Discussion

The current study demonstrates that the NLSCY Hyperactivity/Inattention Subscale was associated with two clinical indicators of ADHD in Canadian children aged 6 to 11 in 1994/1995: methylphenidate use, adjusted for age and sex, and previous diagnosis of emotional disorder, adjusted for age and household income. Earlier studies based on the NLSCY have shown an association between high levels of parent-reported hyperactivity and methylphenidate use among school-aged boys,^3,5 but this is the first to examine the association of hyperactivity with emotional disorder, and to develop a model to determine a threshold for use as a marker for identifying children with ADHD.

Although there is no clear statistical advantage to choosing either current methylphenidate use or previous diagnosis of emotional disorder as the criterion for evaluating the NLSCY H/I Scale, there may be broad conceptual value in choosing the latter. While it is no surprise that methylphenidate use can be a criterion for ADHD, it is somewhat more novel that a history of emotional disorder can be used as a criterion as well. The ROC curves for methylphenidate use and for emotional disorder appear to be essentially the same statistical model. This is consistent with the likelihood that children taking methylphenidate were diagnosed before they began taking it. That is, the basic construct of “caseness” is met —a child came to the attention of a health professional because of parental concern, and that professional agreed that therapeutic attention was warranted. However, diagnosis of emotional disorder represents a wide array of potential disorders. As such, it could be considered for use as a criterion to evaluate other subscales of the NLSCY Children’s Behaviour Scale as potential measures of mental health disorders.

The similarity in the statistical models raises the question of whether methylphenidate use and diagnosed emotional disorder represent the same children. As noted earlier, the two groups may overlap, but only partially. For example, boys were more likely than girls to use methylphenidate, but sex was not significantly associated with diagnosed emotional disorder. And while children in low-income households were more likely to have been diagnosed with an emotional disorder, household income was not associated with methylphenidate use. The lack of overlap may be attributable to several factors. The NLSCY question about emotional disorder asked if the child had ever received a diagnosis. Therefore, a child may have been diagnosed with ADHD in the past, but was not taking medication at the time of the interview. As well, parent reports that a professional had diagnosed the child with an emotional disorder could refer to a wide range of conditions, including cognitive and learning problems identified by educators, and other behaviour disorders for which medication is not the treatment of choice.

The prevalence estimate of 2.1% for ADHD among 6- to 11-year-olds in Canada is low compared with other estimates. The Ontario Child Health Study reported 6.1% among children aged 4 to 16,²³ and the Quebec Child Mental Health Survey, 5.4% among children aged 6 to 14. ²⁴ These two estimates were based on combined parent and teacher information about symptoms and a measure of impairment. In a systematic review of studies using this combination of case identification methods, Waddell et al. generated a summary prevalence estimate of childhood ADHD of 4.8% (95% CI: 2.7 to 7.3).²⁵

The low estimate from the NLSCY H/I Scale may reflect the use of only parent information. However, it may also reflect the input of clinical professionals. In 2000, British researchers, Goodman et al., estimated that 2.4 % of children aged 5 to 15 had ADHD according to DSM-IV criteria.²⁶ Their method for identifying cases included parent and teacher reports and measures of impairment, but in addition, a clinician reviewed all material to decide if the child met diagnostic criteria. Health professionals examine children with behavioural problems to find explanations other than ADHD, a judgment not available from surveys, and one that could influence rates of case identification.²⁷ Therefore, an alternative explanation for the low NLSCY estimate is that it may reflect the practice of Canadian health professionals in 1994/1995. For instance, in 1995/1996, administrative data from Manitoba identified 2.9% of children aged 7 to 9 and 2.2% of children aged 10 to 13 with ADHD, a rate similar to that of the NLSCY.²⁷

Limitations

An important question raised by this study is whether parent-reported clinical case markers are the “gold standard” for identification of childhood ADHD. A strong argument can be made that parent reports introduce multiple sources of potential error. Waddell et al.²⁵ recommended independent professional input for population-based studies. The original design of the NLSCY would have offered the opportunity to use this method, but missing teacher data preclude this option.

The issue is how best to take advantage of the strengths of NLSCY data to examine predictors and consequences of severe childhood hyperactivity and inattention. The clinical markers are useful target criteria for severe behaviour problems. Specifically, parent-reported history of emotional disorder can be used to set thresholds for the children’s behaviour questionnaire subscales in the NLSCY.

The replicate sampling strategy with replacement rather than a random half sample to create the development set could be a limitation, because individual participants might appear in the dataset more than once. Although this may seem to interfere with the independence of observations (and potentially bias derived estimates), the strategy is a simplified version of the bootstrapping procedure used to provide reliable variance estimates and confidence intervals around values derived in population samples.^17,18 In fact, estimates from the development sample were highly comparable to estimates in the NLSCY parent sample (Table 1) .

Some researchers have suggested that NLSCY data can be used without population weights for studies where population estimates are not the primary focus. While the methodological gap addressed here may appear to be such a study, it is important to consider the likelihood of geographic variability. Differences in rates of ADHD diagnosis and psychostimulant prescriptions in adminstrative data suggest that Canadian children experience differential access to specialists and differences in clinical practice.^27,28 For the current study, the prevalence of clinical markers is too low to generate provincial estimates. Statistics Canada’s cross-sectional population weighting strategies were used to resolve the population distribution issues and provide a national estimate.

An additional limitation is that the scale was developed to elicit information about children aged 4 to 11, but previous diagnosis of emotional disorder and impairment in school functioning were asked only for children aged 6 to 11. Also, the sample size was not large enough to perform separate sensitivity analyses by sex and age, which are both known modifiers of hyperactivity and attention span. It is plausible that separate threshold values should be set by sex or by age. Finally, the small number of children reported as using methylphenidate or having been diagnosed with an emotional disorder introduces uncertainty. However, a conservative approach was used in the regression analyses, retaining only variables with statistical significance <0.01 and thereby increasing confidence in the results.

Future directions

With the method described in this paper, the parent-reported NLSCY H/I Scale can be used to identify clinically significant ADHD symptoms in Canadian children aged 6 to 11, either as an outcome measure for investigating developmental antecedents of such symptoms, or as an independent variable predicting adolescent and adult outcomes in the NLSCY sample. Even without teacher information, a score of 14 or more on the scale identifies children likely to have clinically significant difficulties. As well, previous diagnosis of emotional disorder can be used as a target criterion to evaluate other subscales of the NLSCY children’s behaviour questionnaire. With a common method of using the behaviour subscales as clinical markers, NLSCY data can be applied to the study of childhood mental health disorders.

Acknowledgements

Access to the National Longitudinal Survey of Children and Youth was made possible through the Toronto Region Research Data Center of Statistics Canada at the University of Toronto.