Analytical Studies: Methods and References
How Survey Mode and Survey Context Affect the Measurement of Self-Perceived Racial Discrimination across Cycles of the General Social Survey

Acknowledgements

The authors thank Rubab Arim, Tina Chui, Leila Golparvar, Amanda Halladay, Isabelle Marchant, René Morissette, Grant Schellenberg, Stacey Wan and Li Xue for their advice and comments on an earlier version of this article.

Authors

Feng Hou and Christoph Schimmele are with the Social Analysis and Modelling Division, Analytical Studies and Modelling Branch, at Statistics Canada.

Abstract

This article compares how survey mode, survey thematic context and sample design contribute to variation in responses to similar questions on self-perceived racial discrimination across the 2013, 2014, 2019 and 2020 cycles of the General Social Survey (GSS). Among individuals who were aged 15 to 74 years and designated as a visible minority, the shares who reported discrimination because of their race or colour in the previous five years were 31%, 13%, 22% and 38% in 2013, 2014, 2019 and 2020, respectively. The analysis shows that this variation across GSS cycles was primarily attributable to differences in survey mode and survey context.

Introduction

Discrimination is often measured by self-perceived experiences collected from household surveys.^Note  Previous studies have shown a strong association between self-perceived discrimination and outcomes such as life satisfaction (Houle & Schellenberg, 2010; Safi, 2010; Vang et al., 2019), sense of belonging (Berry & Hou, 2017; Painter, 2013; Wu & Finnsdottir, 2021), and physical and mental health (Du Mont & Forte, 2016; Pascoe & Smart Richman, 2009; Schmitt et al., 2014).

Statistics Canada’s General Social Survey (GSS) is a primary source for data on self-perceived discrimination. The GSS is a cross-sectional, national household survey that is conducted on an annual basis. The objectives of the GSS program are to monitor changes in the well-being of Canadians over time and to provide relevant information on social policy issues (Statistics Canada, 2019). Each cycle of the GSS focuses on a thematic topic, which is repeated every five to seven years. Recent GSS cycles on social identity (2013, 2020) and victimization (2014, 2019) have included questions on self-perceived discrimination.^Note  The 2013, 2019 and 2020 cycles were administered with computer-assisted telephone interviews (CATIs) and self-completed electronic questionnaires (EQs), while the 2014 cycle offered the CATI option only. The EQ option was introduced for some cycles in 2013, and the percentage of respondents using this option has increased significantly since then.

Both survey mode and survey context have been identified as sources of variation on subjective measures, affecting differences in responses to questions on life satisfaction, happiness, job satisfaction, self-reported financial difficulty and self-assessed health (Bonikowska et al., 2014; Breunig & McKibbin, 2011; Davillas, de Oliveira & Jones, 2022; Dolan & Kavetsos, 2016; Schork et al., 2021). However, it is unknown if responses to questions on self-perceived discrimination are also sensitive to survey mode and survey context. This raises a concern about whether or how the GSS can be used to monitor changes in the prevalence of self-perceived discrimination over time. To provide policy-relevant information, it is imperative to understand the extent to which variation in the prevalence of self-perceived discrimination across GSS cycles is attributable to differences in survey collection mode and thematic context.^Note  Without accounting for these sources of variation, estimates of changes in the prevalence of self-perceived discrimination may be incorrectly attributed to societal change.

This article investigates reasons for variation in the rate of self-perceived racial discrimination across the 2013, 2014, 2019 and 2020 cycles of the GSS—the four most recent cycles that included a question on this issue. In particular, the analysis compares how survey mode, survey context (i.e., thematic topic) and sample design contribute to this variation. The analysis focuses on individuals who were aged 15 to 74 years and self-identified as members of designated visible minority groups. Additional analysis is conducted separately for South Asians, Chinese and Blacks.^Note 

Survey background

In the GSS, self-reported racial discrimination has been measured with a yes or no question. In the 2013, 2014 and 2019 cycles, respondents were asked, “In the past five years, have you experienced discrimination or been treated unfairly by others in Canada because of your race or colour?” This question was modified slightly in the 2020 GSS: “In the 5 years before the Covid-19 pandemic, have you experienced discrimination or been treated unfairly by others in Canada because of your race or colour?” This modification was designed to differentiate respondents’ perceptions of discrimination before and since the pandemic began.^Note  The GSS further defined discrimination as being treated “differently, negatively or adversely” because of one’s race. While the question on racial discrimination was essentially the same, there were considerable differences across the four cycles in collection mode, survey context, sample design and response rates.^Note Note 

Computer-assisted telephone interviewing was the primary collection mode for the 2013 GSS (for 75% of the respondents, with the remaining 25% collected through self-completed EQs) and the only collection mode for the 2014 GSS. The Internet option—an EQ filled by the respondent online—was the primary collection mode for the 2019 and 2020 cycles (59% and 81%, respectively). The comparability of survey data collected with telephone and may be constrained in two respects. Selection into telephone versus Internet surveys is known to differ across sociodemographic characteristics such as age, income, and sex, and Internet connectivity and use also differ across these characteristics (Sarracino et al., 2017; Wavrock et al., 2021). The implication is that over- or under-representation of specific segments of the target population in telephone interviews versus online questionnaires can influence estimates if these selection effects are correlated with the outcome variable and pertinent sociodemographic characteristics are not accounted for in the construction of sampling weights.

Mode-specific response bias is another possible source of variation. It refers to the variation in measurement of an outcome variable across different collection modes among respondents with the same sociodemographic characteristics (Sarracino et al., 2017; Schork et al., 2021). Previous studies have found that respondents tend to provide more positive evaluations of their subjective well-being on telephone interviews than on EQs (Dolan & Kavetsos, 2016; Sarracino et al., 2017; Schork et al., 2021). For several reasons, identically worded questions can yield different responses across survey modes (for an overview, see Dillman & Christian, 2005). For example, social desirability bias refers to the tendency of respondents to provide positive answers about their well-being in the presence of an interviewer, which is based on a cultural expectation to provide favourable responses or to avoid divulging negative or stigmatizing experiences.

Survey thematic context may also affect responses to the question on discrimination. Responses to subjective questions may not have straightforward answers or be definite in the minds of respondents, and the contextual information that a survey provides can frame their thought processes in formulating a response (Dillman & Christian, 2005; Schork et al., 2021). Previous research has found that differences in survey context is a source of variation for differences in the distributions of life satisfaction scores observed across multiple GSS cycles. The best score was observed on the 2009 cycle focused on victimization, and the least favourable scores were observed on the 2005 and 2010 cycles focused on time use (Bonikowska et al., 2014). The combination of survey context and placement of the life satisfaction question accounted for 50% to 100% of yearly variation in life satisfaction scores. In the 2013, 2014, 2019 and 2020 GSS cycles, the questions on self-perceived discrimination were placed near the end of the survey following a battery of thematic questions. The thematic topic of the 2013 and 2020 cycles was social identity, and these surveys collected data on social contacts, trust in people and institutions, community and civic engagement, and values and attitudes. The thematic topic of the 2014 and 2019 cycles was victimization, and these surveys collected data on experiences of criminal victimization, intimate partner violence, contact with the police, social disorder, and perceptions of safety and crime. It is possible that responses to the discrimination question differ between the surveys on social identity and victimization.

The four GSS cycles also differed in sample design. The 2013 GSS and 2014 GSS added oversamples of immigrants and youth to its regular sample. The 2019 added an oversample of Indigenous people, while the 2020 GSS added a large oversample of selected visible minority groups. Relative to the regular GSS sample, the oversamples were based on a separate sample stratification and targeted specific geographic or demographic areas (Statistics Canada, 2014, 2015, 2020, 2021).^Note  It is unknown whether the oversample and the regular sample respondents differ in their responses to the question on self-perceived racial discrimination.

Results

Chart 1 shows the percentage of GSS respondents who reported racial discrimination among all visible minorities and those who identified as South Asian, Chinese and Black. The overall rate of self-perceived racial discrimination decreased from 31% in 2013 to 13% in 2014, and then increased to 22% in 2019 and further to 38% in 2020. Particularly noteworthy are the large year-over-year differences. The percentage of visible minority respondents who reported racial discrimination decreased by 18 percentage points between 2013 and 2014, and increased by 16 percentage points between 2019 and 2020. Similar variations are observed for each of the three sub-groups. Multivariate analyses show that key sociodemographic characteristics (sex, age, marital status, education, immigrant status, ability to speak English or French, employment status, and geographic region) accounted for little of the cross-cycle differences in the rate of self-perceived racial discrimination (model estimates are shown in Appendix Table 1).^Note  This finding makes it clear that changes in the sociodemographic composition of the survey respondents are not the main driver of the observed changes in self-perceived racial discrimination.

Data table for Chart 1



Data table for chart 1
Table summary
This table displays the results of Data table for chart 1 2013, 2014, 2019 and 2020, calculated using percent units of measure (appearing as column headers).

	2013	2014	2019	2020
	percent
All visible minorities	31.3	13.0	21.6	37.9
South Asian	31.3	13.8	16.9	38.0
Chinese	31.4	7.4	22.6	40.4
Black	48.5	23.8	36.3	53.2
Note: The vertical lines overlaid on the bars indicate the 95% confidence interval. Sources: Statistics Canada, General Social Survey, 2013, 2014, 2019 and 2020.

It is implausible that the large, cross-cycle differences in self-perceived racial discrimination observed in Chart 1 are mainly attributable to societal change at the national level. Other explanations, particularly the effects of collection mode and survey context, warrant assessment. Chart 2 compares rates of self-perceived racial discrimination across survey collection modes. Among the data collected by telephone interviews, the observed rates of self-perceived racial discrimination did not vary significantly across the 2013, 2019 and 2020 GSS cycles. This result remained the same when controlling for sociodemographic characteristics in multivariate analysis. The rate of self-perceived racial discrimination was significantly lower in 2014 than in the other cycles, with or without controls for sociodemographic characteristics (detailed model estimates not shown). Among the data collected online, there was no significant difference in the rate between 2013 and 2020. The rate was significantly lower in the 2019 cycle than in the 2013 and 2020 cycles, with or without controls for sociodemographic characteristics. The difference in the shares of telephone and online respondents accounted for all the difference in the rate of self-perceived racial discrimination between the 2013 and 2020 cycles on social identity.^Note 

Data table for Chart 2



Data table for chart 2
Table summary
This table displays the results of Data table for chart 2 Telephone, Online, 2013, 2014, 2019 and 2020, calculated using percent units of measure (appearing as column headers).

	Telephone				Online
	2013	2014	2019	2020	2013	2019	2020
	percent
All	29.6	13.0	26.5	28.0	37.5	20.1	39.5
Note: The vertical lines overlaid on the bars indicate the 95% confidence interval. Sources: Statistics Canada, General Social Survey, 2013, 2014, 2019 and 2020.

The responses also differed by survey context. In the 2013 and 2020 cycles on social identity, the rate of self-perceived racial discrimination among visible minorities was higher among online respondents than among telephone respondents. Compared with telephone respondents, the rate of self-perceived racial discrimination was 8 percentage points (or 10 percentage points with controls for sociodemographic characteristics) higher for online respondents in 2013 and 12 percentage points (or 14 percentage points with controls) higher in 2020. In the 2019 cycle on victimization, the rate of self-perceived racial discrimination was 6 percentage points lower (or 3 percentage points lower with controls, not statistically significant) among online respondents than among telephone respondents.^Note 

There was also an interaction between survey mode and survey context that contributed to inconsistencies in the changes in the rate of self-perceived racial discrimination. Among telephone respondents on the GSS cycles on victimization, there was a 14 percentage point (13 points with controls for sociodemographic characteristics) increase in the rate between 2014 and 2019. In contrast, among telephone respondents on the cycles on social identity, the rate decreased 2 percentage points (3 points with controls) between 2013 and 2020, although this change was statistically nonsignificant. Secondly, between 2019 and 2020, the rate of self-perceived racial discrimination did not significantly change among telephone respondents, but the rate almost doubled among online respondents with or without controls for sociodemographic characteristics.

Survey context may be directly related to the difference in self-perceived racial discrimination between GSS cycles within the same survey mode. For telephone respondents, differences in survey context is a likely reason for the much lower rate of self-perceived racial discrimination in the 2014 cycle on victimization than in the 2013 cycle on social identity. Similarly, survey context is a likely explanation for at least part of the difference in the rate between online respondents in the 2019 cycle on victimization and the 2020 cycle on social identity. The overall rate of self-perceived racial discrimination among online respondents was much higher in 2020 than in 2019, and was also higher in 2020 for South Asians, Chinese and Blacks (Chart 3). Conversely, the rates in 2019 and 2020 were similar for telephone respondents, and thus there was no context effect.^Note 

Data table for Chart 3



Data table for chart 3
Table summary
This table displays the results of Data table for chart 3 2019 and 2020, calculated using percent units of measure (appearing as column headers).

	2019	2020
	percent
South Asian	16.4	40.6
Chinese	20.6	40.8
Black	32.3	57.3
Note: The vertical lines overlaid on the bars indicate the 95% confidence interval. Sources: Statistics Canada, General Social Survey, 2019 and 2020.

Different oversamples could be another potential factor for the variation in rates of self-perceived racial discrimination across GSS cycles. Chart 4 shows the rate of self-perceived racial discrimination for the regular sample and oversample in the 2013 and 2020 GSS cycles.^Note  The oversample consisted of immigrants in the 2013 cycle and visible minorities in the 2020 cycle. The difference in the rate between the regular sample and oversample was statistically significant in 2013, but not in 2020. Visible minority immigrants, particularly recent arrivals, were less likely to report racial discrimination than their Canadian-born counterparts. When immigration status, ability to speak English or French, and survey mode were controlled, the difference in the rate between oversample and regular sample in 2013 became statistically nonsignificant. Furthermore, the differences in oversamples did not affect the observed change in the rate between 2013 and 2020, with or without controls for key sociodemographic characteristics.

Data table for Chart 4



Data table for chart 4
Table summary
This table displays the results of Data table for chart 4 2013, 2020, Regular sample and Oversample, calculated using percent units of measure (appearing as column headers).

	2013		2020
	Regular sample	Oversample	Regular sample	Oversample
	percent
Rate (%)	33.2	27.6	35.9	38.0
Note: The vertical lines overlaid on the bars indicate the 95% confidence interval. Sources: Statistics Canada, General Social Survey, 2013 and 2020.

Summary and discussion

The General Social Survey (GSS) is a primary source of data for monitoring many social indicators in Canada. Since 2013, some GSS cycles have been conducted with both telephone and online questionnaires, and it is the case for all cycles starting in 2018. A multi-mode survey is intended to abate declines in response rates on telephone interviews and reduce survey administration costs (Schork et al., 2021; Statistics Canada, 2019). The GSS rotates survey content on an annual basis to cover different thematic topics. Both survey collection mode and survey context are known to influence responses to subjective questions, which poses a challenge for interpreting trends across GSS cycles.

This study compared the rate of self-perceived racial discrimination across four cycles of the GSS to examine the sensitivity of this indicator to mode effects, survey context and survey design. Clearly, collection mode and survey context have strong effects, while sample design (oversamples) has non-significant effects on the cross-cycle differences in the rate of self-perceived discrimination. The rate of self-perceived racial discrimination in 2014 (victimization) was less than half the rate observed in the previous year (social identity). In 2020 (social identity), the rate was almost triple that observed in 2014 (victimization). This increase could be partly attributable to movements and demonstrations (e.g., Black Lives Matter) that may have heightened sensitivity to perceptions of discrimination and unfair treatment (Cotter, 2022). However, the present study found that the rate of self-perceived discrimination among the Black population and other visible minorities did not significantly increase between 2013 and 2020 within the same collection mode. Hence, societal change was unlikely the primary reason for the fluctuations in self-perceived racial discrimination observed across the 2013, 2014, 2019 and 2020 GSS cycles. To a large extent, these are methodological artefacts that need to be accounted for in the measurement of rates of self-perceived racial discrimination in the GSS.

In the 2013 and 2020 cycles of the GSS (social identity), online respondents reported a higher rate of self-perceived racial discrimination compared with telephone respondents. In contrast, online respondents on the 2019 GSS (victimization) reported a lower rate than telephone respondents. The higher rates observed among online respondents in the 2013 and 2020 cycles are consistent with previous studies that demonstrate that online responses to subjective measures of well-being tend to be less positive than responses from telephone interviews (Dolan & Kavetsos, 2016; Sarracino et al., 2017; Schork et al., 2021). The present study provided a comparison of differences in self-perceived discrimination across collection modes, but was not a statistical mode-effect study. However, the findings are consistent with a study that used a propensity score analysis that aimed to formally isolate mode effects, which found a mode effect on self-perceived discrimination in the 2020 GSS (SSMD, 2022). It is unknown which estimates—telephone versus online responses—are nearest to the actual value in the target population (Schork et al., 2021).

Survey context is another source of variation in the rate of self-perceived racial discrimination across cycles of the GSS, and the largest differences in the rate across GSS cycles are observed when there are context differences. The large decrease between 2013 and 2014 and the large increase between 2019 and 2020 suggest this finding. The 2014 and 2019 cycles focused on victimization, while the 2013 and 2020 cycles focused on social identity. The battery of questions on criminal victimization and social disorder in the 2014 and 2019 cycles may have put the question on perceived racial discrimination in a different context than in the cycles on social identity. For example, it may be that a lack of negative experiences on the questions on crime and victimization could increase respondents’ subjective evaluations in other respects (Bonikowska et al., 2014). Conversely, in the 2013 and 2020 cycles on social identity, the discrimination question was asked after a series of questions on social contact, civic engagement, political participation, and perceptions about Canadian achievements, institutions, and shared values. These preceding questions may set up a high standard for respondents to evaluate their perception of racial discrimination. The observed large survey context effect suggests that the GSS should not be viewed as a repeated annual survey, but rather as a series of distinct surveys on different topics. Furthermore, comparisons on subjective indicators should be made only between surveys with the same theme.

It is recommended that questions on self-perceived racial discrimination be compared with caution across years because of differences in survey mode of administration and survey context across GSS cycles. Also, any comparisons of the rate of self-perceived racial discrimination should be restricted to cycles with the same survey context. Additionally, the GSS cycles administered with CATIs and EQs include a variable for survey collection mode, which can be used to control for mode effects in regression analysis.

Although questions with subjective content are known to be sensitive to survey mode and context effects, questions with objective content are less vulnerable to these effects (Schork et al., 2021). Accordingly, a direction for future research is to investigate indicators of racial discrimination that consist of questions that have more specific or objective content, such as those on the Everyday Discrimination Scale (see Sternthal et al., 2005; Williams et al., 1997), which may improve comparability.



References

Bonikowska, A., Helliwell, J., Hou, F., & Schellenberg, G. (2014). An assessment of life satisfaction responses on recent Statistics Canada surveys. Social Indicators Research, 118(2), 617–643.

Berry, J. W., & Hou, F. (2017). Acculturation, discrimination, and well-being among second generation of immigrants in Canada. International Journal of Intercultural Relations, 61, 29–39.

Breunig, R., & McKibbin, R. (2011). The effect of survey design on household reporting of financial difficulty. Journal of the Royal Statistical Society: Series A (Statistics in Society), 174(4), 991–1005.

Cotter, A. (2022). Experiences of discrimination among the Black and Indigenous populations in Canada, 2019. Juristat 42(1). https://www150.statcan.gc.ca/n1/pub/85-002-x/2022001/article/00002-eng.htm.

Davillas, A., de Oliveira, V. H., & Jones, A. M. (2022). Is inconsistent reporting of self-assessed health persistent and systematic? Evidence from the UKHLS. IZA Discussion Paper No. 15085. IZA Institute of Labor Economics.

Dillman, D. A., & Christian, L. M. (2005). Survey mode as a source of instability in respondents across surveys. Field Methods, 17(1), 30–52,

Dolan, P., & Kavetsos, G. (2016). Happy talk: Mode of administration effects on subjective well-being. Journal of Happiness Studies, 17(3), 1273–1291.

Du Mont, J., & Forte, T. (2016). Perceived discrimination and self-rated health in Canada: An exploratory study. BMC Public Health, 16(1), 1–9.

Golparvar, L. (2021). Cycle 34 mode effect analysis – Canadian’s safety (victimization) of the General Social Survey. [Unpublished internal methodology report]. Social Statistics Methods Division, Statistics Canada.

Houle, R., & Schellenberg, G. (2010). New immigrants’ assessments of their life in Canada. (Analytical Studies Branch Research Paper Series, No. 322). Statistics Canada. https://www150.statcan.gc.ca/n1/pub/11f0019m/2010322/part-partie1-eng.htm.

Oreopoulos, P. (2011). Why do skilled immigrants struggle in the labor market? A field experiment with thirteen thousand resumes. American Economic Journal: Economic Policy, 3(4), 148–171.

Painter, C. V. (2013). Sense of belonging: A Literature Review. Citizenship and Immigration Canada. https://www.canada.ca/content/dam/ircc/migration/ircc/english/pdf/research-stats/r48a-2012belonging-eng.pdf.

Pascoe, E. A., & Smart Richman, L. (2009). Perceived discrimination and health: A meta-analytic review. Psychological Bulletin, 135(4), 531–554.

Safi, M. (2010). Immigrants’ life satisfaction in Europe: Between assimilation and discrimination. European Sociological Review, 26(2), 159–176.

Sarracino, F., Riillo, C. A. F., & Mikucka, M. (2017). Comparability of web and telephone surveys for the measurement of subjective well-being. Survey Research Methods, 11(2), 141–169.

Schmitt, M. T., Branscombe, N. R., Postmes, T., & Garcia, A. (2014). The consequences of perceived discrimination for psychological well-being: A meta-analytic review. Psychological Bulletin, 140(4), 921–948.

Schork, J., Riillo, C. A. F., & Neumayr, J. (2021). Survey mode effects on objective and subjective questions: Evidence from the Labour Force Survey. Journal of Official Statistics, 37(1), 213–237.

Social Statistics Methods Division. (2022). Mode effect analysis of the 2020 General Social Survey on Social Identity. [Unpublished internal methodology report]. Statistics Canada.

Statistics Canada. (2014). General Social Survey, 2013, Cycle 27: Social Identity – Analytical File User Guide.

Statistics Canada. (2015). General Social Survey, 2014, Cycle 28: Canadians’ Safety and Security (Victimization) – Microdata User Guide.

Statistics Canada. (2019). General Social Survey: An overview, 2019. Catalogue no. 89F0115X-2019001. https://www150.statcan.gc.ca/n1/en/pub/89f0115x/89f0115x2019001-eng.pdf?st=l0K5cTj-.

Statistics Canada. (2020). General Social Survey, 2019, Cycle 34: Canadians’ Safety (Victimization) Microdata User Guide.

Statistics Canada. (2021). General Social Survey, 2020, Cycle 35: Social Identity Microdata User Guide.

Sternthal, M. J., Slopen, M., & Williams, D. R. (2011). Racial disparities in health. Du Bois Review, 8(1), 95–113.

Vang, Z., Hou, F., & Elder, K. (2019). Perceived religious discrimination, religiosity, and life satisfaction. Journal of Happiness Studies, 20(6), 1913–1932.

Wavrock, D., Schellenberg, G., & Schimmele, C. (2021). Internet-use typology of Canadians: Online activities and digital skills. (Analytical Studies Branch Research Paper Series, No. 465). Statistics Canada. https://www150.statcan.gc.ca/n1/pub/11f0019m/11f0019m2021008-eng.htm.

Williams, D. R., Yan, Y., Jackson, J. S., & Anderson, N. B. (1997). Racial differences in physical and mental health: Socioeconomic status, stress and discrimination. Journal of Health Psychology 2(3), 355–351.

Wu, Z., & Finnsdottir, M. (2021). Perceived racial and cultural discrimination and sense of belonging in Canadian society. Canadian Review of Sociology, 58(2), 229–249.

Zanutto, E. L. (2006). A comparison of propensity score and linear regression analysis of complex survey data. Journal of Data Science, 4(1), 67–91.