Canadian Survey on Disability, 2017: Concepts and Methods Guide
8. Differences between the 2017 and 2012 cycles of the Canadian Survey on Disability

Skip to text

Text begins

Improvements were made to the methodology and content of the 2017 Canadian Survey on Disability (CSD) which affected time series comparability with 2012 CSD data. Differences between the 2017 and 2012 cycles are described below. Because of these changes, it is neither possible nor recommended to compare data over time between these two surveys. Any attempts at comparison will be considered invalid.

8.1 New census filter questions

As was the case in 2012, the 2017 CSD used the Disability Screening Questions (DSQ) to identify Canadians with disabilities. However, the population targeted to participate in the 2017 survey was much more inclusive than in 2012 due to changes to the filter questions (i.e., the new question on Activities of Daily Living in the 2016 Census). Responses to the census filter questions targeted people most likely to have a disability in order to create the CSD survey frame. The new questions introduced in 2016 greatly improved coverage of people with a disability compared with the questions asked in previous censuses. The old filter questions were not as effective in targeting people with a cognitive or mental health-related disability.

When the DSQ were being developed, new filter questions^Note had been developed and were shown to provide good coverage of all people with a disability, and, more specifically, those who were less well targeted by the filter questions used in previous censuses. However, given that it was too late to change the filter questions for the 2011 National Household Survey (NHS) (which, at the time, replaced the long-form census questionnaire), the 2012 CSD survey frame had to be created based on answers to the old filter questions. In the summer of 2014, the new DSQ filter questions were tested and compared with the old filter questions as part of the Census Content Test. The analysis clearly showed that the new questions were much more inclusive and the results of these tests aligned with the results of the previous tests conducted during development of the DSQ (Statistics Canada, 2016). Therefore, the new filter questions from the DSQ were used in the 2016 Census long-form questionnaire to replace the old questions. As a result, the 2017 CSD survey frame was created to include all respondents aged 15 and over who, in response to the new census filter questions, reported sometimes, often or always having difficulty doing certain activities of daily living.

Certain data users may seek to compare the prevalence of disability between the 2012 and 2017 surveys. However, given the improvements made to the coverage of people with a disability (through the use of the new filter questions in the 2016 Census), a comparison would not be valid. In other words, the increased prevalence of disability in the 2017 CSD is attributable in large part to the improved coverage of people with a disability. It is not possible to quantify the proportion of the increase due to changes made to the filter questions.

This methodological change was the last improvement required to complete the full implementation of the DSQ in the CSD survey frame.

8.2 Changes to the CSD collection methods

As discussed in Section 4.3, data collection for the 2017 CSD was conducted using an electronic questionnaire that could either be self-administered (rEQ mode) online or completed over the phone with the assistance of an interviewer. Approximately two in five respondents (40%) opted for the self-administered online collection. In 2012, data collection consisted primarily of telephone interviews, as well as a few pen-and-paper interviews in the Northwest Territories. This change in collection mode may have affected responses. For example, the fact that those who answered the questions in rEQ mode could see all the possible response choices, as opposed to those who have the choices read to them over the phone by an interviewer, can have an impact on the responses. This is called the “mode effect.”

Thus, changes to the collection methods between the 2012 and 2017 CSD is another factor that could affect the comparability of the data between these two surveys.

8.3 Lag time between the 2016 Census and the 2017 CSD

As mentioned above, the CSD survey frame was established based on the responses to questions on activities of daily living in the 2016 Census long-form questionnaire and the CSD was conducted 10 to 16 months later. While the time lag in this instance was shorter than the time between the 2011 NHS and the 2012 CSD (16 to 20 months), it is not inconsequential. The possibility that a respondent who reported a difficulty in the census no longer experienced it at the time of the CSD or, conversely, was institutionalized, deceased or no longer living in the country is significant. As a result, to prevent underestimating the prevalence of disability due to these losses in the YES population,^Note the weights of the units in the NO population (for whom no collection was done) were adjusted downward to account for people who may have left the country or died between the census and CSD data collection. Given that this adjustment is related to the length of the time lag, it will have an effect on data comparisons. For more information on the weighting used for the CSD, please see Section 6.1.

In addition, some of the information collected in the census and annexed to the CSD file (e.g., data on labour force status, income, household composition) may have changed in the time between the two surveys. Even though the time lag between the 2016 Census and the 2017 CSD was shorter than the one between the 2011 NHS and the 2012 CSD, it still has an impact on the data.

8.4 Other methodological change

The 2012 CSD survey frame was established based on responses to the 2011 NHS, whereas the 2017 CSD survey frame was based on responses to the 2016 Census long-form questionnaire. Given the voluntary nature of the 2011 NHS, the response rate was lower than that of the mandatory 2016 Census. Although Statistics Canada adapted its collection and estimation procedures in order to mitigate, to the extent possible, the effect of non-response for the 2011 NHS, the difference in response rates between the 2011 NHS (77.2%) and the 2016 Census (97.8% for the long-form) may still have an impact on the comparison of results between the 2012 and 2017 CSD.  

8.5 Changes to CSD content

Compared to the 2012 CSD, the content of the 2017 CSD has been expanded to include many new indicators, has been revised and updated in many respects and has also been modestly cut back in a few subject areas.

Several new survey modules were added for 2017, covering a wide range of new data priorities, as described more fully in Chapter 2. In addition, almost 300 census variables were linked to the 2017 CSD compared to the 200 variables linked in 2012. Thus, for many 2017 content areas, there are no comparative indicators from 2012.

Many survey questions were also updated in 2017 to better reflect current realities and to correct known areas for improvement in the 2012 CSD. For example, the section on aids and assistive devices underwent many updates to capture the increasing usage of certain electronic technologies, such as devices with voice recognition software, smartwatches, tablets and e-books. Other revisions included questions about the age of onset of disabilities, which were greatly strengthened for 2017. In 2012, respondents were asked to specify up to two main conditions underlying their disability and asked about the age of onset for each of these conditions. For 2017, separate age of onset and age of limitation questions were asked for each of the 10 disability types in order to obtain more accurate data for persons with multiple disabilities. Also, where disability was the result of an accident or injury, new indicators were added to differentiate between workplace, home, motor vehicle and sports related causes. Finally, labour modules were also substantially reorganized in 2017. Questions were asked in a new order that better reflected standard employment indicators found on other labour surveys by Statistics Canada. In addition, modifications were made to the classification of retirement for 2017 to better capture the changing complexities in the patterns of work among older workers. Due to these many improvements, indicators across a whole spectrum of 2017 survey modules are not historically comparable with 2012 data.

For the many survey questions which were carried forward from 2012 to 2017 essentially unchanged in content, substantial formatting changes to these questions were required in order to adapt the survey for the Internet, self-reporting EQ screen environment. For example, instructions that were once directed at interviewers had to be tailored as on-screen help text. Also, the “Refusal” option once possible for interviewers to record, was no longer available for EQ questions and the “Don’t know” option was only possible for a small minority of EQ questions (where it was available as a response category).

It should also be noted that some of the content from the 2012 CSD was removed in 2017 in order to balance the respondent burden created by extensive new content additions.

These many changes to questionnaire content, wording, ordering and formatting mean that comparisons should not be made between the 2012 and the 2017 CSD data.

8.6 Summary and recommendation

As discussed above, the main differences between the 2017 and 2012 CSD can be summarized as follows:

• Changes to the 2016 Census filter questions provided increased coverage of persons with a disability for the 2017 CSD, especially persons with a cognitive or mental health-related disability.
• Data collection for the 2017 CSD was conducted using an electronic questionnaire, which was self-administered online by some respondents and completed by phone with the assistance of an interviewer for the others. This change could affect the collected data relative to the 2012 CSD, which was entirely conducted by telephone interview.^Note
• The time lag between the 2016 Census and the 2017 CSD was shorter than the one between the 2011 NHS and the 2012 CSD. Nonetheless, in both cases, survey weights of the NO sample had to be adjusted to avoid underestimating the prevalence of disability due to losses caused by deaths, institutionalization and people who left the country. The adjustment, which took into account the duration of the time lag, may have a slight impact on comparisons between the two cycles.
• The 2017 CSD survey frame was established based on the mandatory 2016 Census, which had a response rate of 97.8% (for the long-form), while the 2012 CSD frame was based on the voluntary 2011 NHS, which had a response rate of 77.2%. This change may have had an impact on the results of the 2017 CSD.
• Finally, changes to questionnaire content, wording, ordering and formatting were implemented in order to focus on new data priorities and adapt to the new EQ environment, which could potentially affect historical comparisons of indicators.

These changes, particularly the ones to the census filter questions, have a strong impact on data comparability. As a result, it is neither possible nor recommended to compare the 2017 and 2012 CSD data. Any attempts at comparison will be considered invalid.

• Previous
• Next