A New Survey Measure of Disability: the Disability Screening Questions (DSQ)
3. Developing the DSQ
Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please "contact us" to request a format other than those available.
- Main page
- 1. Context
- 2. Measuring disability at Statistics Canada
- 3. Developing the DSQ
- 4. The new DSQ
- 5. Severity score
- 6. The short DSQ
- 7. Current and future uses of the DSQ
- 8. Comparing the DSQ with other health measures
- 9. Recent developments
- 10. Conclusion
- More Information
- PDF version
The first version of the DSQ was developed over two years, beginning in the spring of 2010 with a review of previous disability modules used in Canada (such as HUI3 or WG) and other instruments that might be considered or had been used in other countries, such as the Budapest Initiative (BI) or WHODAS.Note 1 The DSQ development team included subject matter experts and analysts from ESDC (HRSDC at that time) and subject matter experts and methodologists from Statistics Canada (STC), under the guidance of a Technical Advisory Group (TAG) made up of community and academic experts. Based on that review, options were proposed for approaches and questions that might be used. Efforts were made to employ wording used by instruments such as the HUI3, WG, and BI in order to promote comparability. Qualitative testing was conducted by Statistics Canada's Questionnaire Design Resource Centre (QDRC) in September 2010, March 2011, and May 2011.Note 2
A goal in developing the DSQ was to follow the social model of disability and take into account activity limitations to identify disability. Also, because the DSQ were meant for inclusion on general population surveys, it could take no more than 2 minutes, on average, to administer. At the same time, it was hoped that the DSQ should distinguish as many disability types as possible and treat all disability types consistently.
3.1. The social model of disability
The social model of disability recognizes that while disability is linked to an individual and to the degree of difficulty with certain tasks, the level of impairment, or the existence of a health condition, it also considers the individual’s environment as a key factor in determining disability.Start of text box
The medical model of disability focuses on deficiencies, ailments or inabilities. It considers that disability is caused by a physiological disease or injury resulting in a “damaged” body or mind that does not function in a manner considered normal for a human being. It does not explicitly recognize that society is organized based on certain assumptions, one of which is that everyone is able-bodied.
The social model views disability as a social disadvantage that an environment imposes on top of an individual’s impairment. It looks at ways of removing barriers that restrict life choices for disabled people.
With the social model of disability, it was important to ask not only about difficulties that people may have doing certain tasks, but also about how often these difficulties limit their daily activities. According to the social model, individuals can have a functional impairment without having a disability if their physical and social environments are adapted so that their daily activities are not limited.
The definition of disability used prior to the DSQ did not always take activity limitations into account. For instance, the PALS followed a mix of the functional and social models to identify persons with a disability, depending on the disability type. The PALS asked questions about activity limitations for some disability types, but they were not always considered in the determination of the presence of a disability.
3.2. Content validity of the DSQ
Content validity is the degree to which an instrument measures what it is supposed to measure. With the DSQ, it was assessed with qualitative testing of each question.
Qualitative testing is used to ensure that the questions and answers actually capture the concepts that they were meant to measure. It can uncover problems, even with very simple questions.
The original long version of the DSQ underwent three major rounds of qualitative testing in six locations across Canada. Approximately 125 individual interviews and four focus group discussions were conducted. Recruitment of respondents for the in-depth interviews took several factors into consideration:
- Mix of English and French;
- Mix of males and females;
- Mix of ages (20 or older);Note 5
- Mix of educational attainment, household size, and household income;
- Mix of disability types (based on questions asked during recruitment of participants).
The results of each round of qualitative testing were used to improve the questions for the next round. As well, the results from the first location of a given round were often used to improve the questions for the following locations in that round.
The remainder of this section is an overview of the main results of each round of testing. The wording of each question tested is not given; only the general idea and main decisions made in designing the questionnaire are mentioned. The final wording of each question retained for the DSQ is presented in Section 4.3.
When qualitative testing started, the DSQ were called the Disability Identification Questions. Hence, the QDRC reportsNote 6 written for the first two rounds of testing refer to that name, while the last one, to the Disability Screening Questions.
First round of qualitative testing - September 2010
The first round of testing started with 15 English interviews in Vancouver, followed by 15 English interviews in Ottawa, and 16 French interviews in Montréal.
The questions for the first round covered all the disability types that had been included in the 2006 PALS. However, the questions were split into two sections.
The first section asked about difficulty with certain tasks for some of the disability types, and for others, about difficulties that reduced the amount or kind of activity that the person can do (for example: “Do you have any learning difficulties that reduce the amount or kind of activities you can do? Examples of learning difficulties include attention problems, hyperactivity and dyslexia.”). It was hoped that the latter questions would save time by not requiring two separate questions—one to ask about the presence of difficulty and another to ask about activity limitation resulting from it. These questions did not work well because respondents were confused by asking about the presence of difficulty, and at the same time, if the difficulty reduced activities.
The second section in the test questionnaire asked if the limitations reported in the first section created difficulties in specific domains: finding or maintaining employment, attending school, transportation, household tasks, self-care, activities outside the home, communicating, getting along with others, and participating in other activities. The aim was to cover domains in the United Nations Convention on the Rights of Persons with Disabilities. The 2006 PALS did make reference to specific domains for selected disability types, but this was done immediately after each question on difficulty for a given disability type, rather than in a separate section after all the questions on difficulty for all disability types were asked. As well, a limited set of domains (home, school or work, and other) was considered in the PALS.
This second section was problematic because respondents were often not sure how to take limitations for several disability types into account when answering the domain-specific questions. For example, if in the first section of the questionnaire, an individual reported difficulties due to a mental health-related condition and difficulties with flexibility, by the time he/she got to the second section, it was difficult to report how much difficulty he/she had in each domain (finding employment, maintaining employment, attending school, etc.). The mental health-related condition of that person might give them some difficulty in certain domains, but a lot in others, while the flexibility issues might affect the same domains or different ones.
The first round of testing yielded several general findings/decisions:
- Certain questions were complex and difficult to answer; interviewers typically found the same questions to be verbose and problematic to administer.
- Participants did not always interpret questions as intended.
- Issues arose with the question framing, such as participants’ inability to retain the desired memory cue (6-month/long-term). Also, some participants were uncertain whether to consider factors such as assistive devices (for example, walkers and canes) or medication when answering certain questions.
- Participants did not consistently map their internally generated answers to the response categories given by the survey questions. This was due both to the choice of scale as well as the number of points offered on the scale.
- Inconsistent use and understanding of terminology may cause interpretation problems, for example “difficulty,” “condition,” “limitations,” “disability or disorder.”
- To keep the module as short as possible, many of the questions in the first section (and response options) had been selected from other survey instruments and then combined. Subtle question development and design considerations may have been overlooked.
- An attempt was made to use questions from other instruments. For example, the question initially proposed to detect a mobility disability was based largely on the WG Short set, which inquired about “difficulty walking or climbing up and down steps.” Qualitative testing revealed that the double-barrelled question was difficult to answer in a final screening question. It was recommended that this be split into two questions.
- Particularly for mental health-related disabilities, respondents who experienced episodic difficulty were reluctant to report a condition if it was not experienced continually for six months or more. The question initially tested was directly from the PALS. However, some individuals had histories of being limited by mental health conditions that had lasted longer than six months, which was no longer the case. The wording had to be changed to reflect that the condition had lasted six months or longer, although symptoms had not necessarily lasted that long.
- Some PALS questions gave problematic examples. For instance, the reference to “phobias” for the mental health-related disability question was not interpreted as intended. Respondents tended to interpret “phobia” as something that is disliked intensely rather than a morbid fear or aversion.
- Questions in the second section were frequently misunderstood, particularly by respondents with several mild disabilities.
A major revamping of the questions took place for the second round of testing. The most notable changes were:
- Elimination of the second section, which asked about difficulties in several contexts
- To save interview time, the addition of filter questions at the beginning of the questionnaire to immediately screen out people who are not likely to have a disability.
The wording of questions was revised based on the results of the first round of testing; for instance, “Rarely” was added to the answer categories for the activity limitation questions, which initially had been “No,” “Sometimes,” “Often,” and “Always.”
Second round of qualitative testing - March 2011
The second round of testing of the DSQ took place in March, 2011. This round included 17 English interviews in Toronto, 16 French interviews in Montréal, and 16 English interviews in Ottawa. It also involved two focus groups in Montréal and two in Ottawa; one group in each location was for persons with a mental health-related condition, and the other, for persons with a learning disability.
The addition of filter questions at the beginning of the questionnaire distinguished people who are most likely to have a disability from those who are not. Those identified as likely were asked a more detailed set of screening questions to determine if they did, in fact, have a disability and what type. This strategy is necessary to reduce average interview time, especially given the relatively low prevalence of disability in the population and the large number of questions required to identify it.
These filter questions had to fulfill several conditions:
- minimal total time, since the questions must be asked of everybody;
- broad enough to include people with all types and levels of disability, but not so broad that the filtered-in rate would be too high;
- properly filter-in people with non-physical disabilities; and
- minimize the number of false negatives to avoid excluding persons with a disability.
These filter questions, like the Census filter questions, were not meant to measure disability. They are deliberately somewhat vague in order to include people who may have a disability. It is the role of the follow-up questions (or screening questions) to properly identify persons with a disability.
The questionnaire for this test started with five filter questions; screener questions were then asked of people who said “yes” to at least one filter question.
Two versions of screener questions for vision were tested. The most effective asked about the ability to see rather than difficulty seeing. This wording was retained in the final DSQ. To keep the question structure similar, the question on hearing also asked about ability to hear (rather than difficulty hearing). These are the only two disability types in the final DSQ that ask about ability instead of difficulty.
A decision that emerged from the second round of testing was to replace the filter question that was meant to identify people with a mental health-related condition by the first screener question for this disability type. The tested filter question was too general; it did not identify people with a mental health-related condition, but they were identified by the screener question which offered more precise examples. As a result, respondents are asked the filter question first, and some of them are asked the same question again later as a screener (Section 4.3.4).
After the second round of testing, the “Communication” disability type was dropped from the DSQ. A few variations of question wording were explored, but each elicited diverse interpretations. When interviewers explored the questions with respondents, cultural overtones were apparent in how they interpreted the questions and determined their responses.
The fifth filter question (“Do you have any other health problem or condition that has lasted or you expect to last for six months or more?”) tended to filter in too many false positives. To reduce the number of false positives, a sixth filter question was added: “How often does this health problem or condition limit your daily activities?” Those who reported being “Rarely,” “Sometimes,” “Often” or “Always” limited were filtered in and proceeded to the screeners.
Third round of qualitative testing - May 2011
The third round of qualitative testing of the DSQ was conducted in May, 2011. This round consisted of 22 interviews: 8 English and 6 French in Moncton; 8 English in Vancouver.
According to the QDRC observation report:
- In general, participants were able to answer the questions without difficulty.
- This is by far the most efficient version of the DSQ that has been tested since September 2010.
- In a few rare cases, respondents were unsure whether to report conditions that they had in the past (for example, someone with vision problems three years ago who had surgery and now sees without problems).
- Recommendation: If respondents are unsure, interviewers should record respondents’ present-day conditions and difficulties.
- The change to “activités” (from “activités quotidiennes”) from the March phase of testing worked well for French respondents. When probed, they gave a broad range of examples from housework, work, shopping, etc., which is close to the English “daily activities.”
The recommendations made by QDRC after this round of testing had to do with a better use of dynamic textNote 7 in question wording, slight modification of answer categories, emphasis on certain words in the questions, and better instructions for interviewers.