Analytical Studies: Methods and References
An Assessment of Non-probabilistic Online Survey Data: Comparing the Carrot Rewards Mobile App Survey to the Canadian Community Health Survey

Skip to text

Text begins

Acknowledgements

The authors would like to thank CARROT Insights Inc. staff Lauren White, Dr. Marc Mitchell, and Megan Nobrega for their help in the acquisition of the data and their comments on the manuscript.

Abstract

Survey data collection through mobile devices, such as tablets and smartphones, is underway in Canada. However, little is known about the representativeness of the data collected through these devices. In March 2017, Statistics Canada commissioned survey data collection through the Carrot Rewards Application and included 11 questions on the Carrot Rewards Mobile App Survey (Carrot). These questions were drawn from the 2017 Canadian Community Health Survey (CCHS).

The objective of this paper was to assess the sociodemographic and health characteristics of Carrot respondents vis-à-vis the nationally representative sample provided by the 2017 CCHS. The results showed that the characteristics of Carrot and CCHS respondents varied across most sociodemographic and health characteristics. Carrot respondents were disproportionately female, young, single or in a common-law relationship, university-educated, Canadian-born, and residents of larger urban centres. In addition, Carrot respondents assessed their general health, mental health, and sense of community belonging in unfavourable terms relative to CCHS respondents. Moreover, relatively low levels of life satisfaction among Carrot respondents were particularly striking, with this difference remaining even after important covariates were taken into account.

These findings highlight that the purposes for which data are collected are critical to evaluating data quality and fitness for use. CARROT Insights Inc. uses its survey results to improve the health and well-being of its users and these results are certainly appropriate for this purpose. However, given the highly selective and unrepresentative profile of Carrot respondents, such data should not be used to draw inferences regarding the Canadian population more broadly.

1 Introduction

There is growing recognition and discussion among public sector organizations—including national statistical offices—academia and the private sector regarding the challenges facing household surveys (for a review, see Couper 2017). Declining response rates and relatively high costs are among the difficulties faced, with “telephone surveys…arguably experiencing the greatest degree of upheaval in recent times” (Couper 2017, p. 124).

But while the growing use of cell phones and other Internet-based technologies has been one of the factors undermining random digit dialing (RDD) telephone surveys, such technologies also offer new potential for data collection. This is especially true given the increased presence of Internet-enabled mobile devices, such as smartphones and tablets,^Note  which allow surveys to be fielded to respondents anytime, anywhere. Data collection through mobile devices is now prevalent in many countries (Şahin and Yan 2013), and this has been accompanied by a growing amount of research literature on associated issues (e.g., Raento, Oulasvirta and Eagle 2009; Sugie 2016).

Questions pertaining to respondent coverage, representativeness and sampling continue to be central in discussions on data collection through Internet-enabled mobile devices. In 2015, 86% of Canadian households were subscribed to mobile telephone services. Among Canadians aged 18 and older, 73% owned a smartphone and 52% owned a tablet (CRTC 2016). Nonetheless, coverage rates are still below those reached by telephone landlines in past decades, when RDD telephone surveys were the norm (Couper 2017). This raises questions about respondent selection on the basis of who owns and uses a smartphone. Respondent selection issues further arise in terms of which individuals choose to download a specific application (app) and, for those users, yet further selection in terms of which individuals choose to respond to specific survey offerings. Overall, data collection activities using mobile devices based on non-probabilistic samples raise concerns regarding statistical inference and generalizability to broader populations.

Survey data collection through mobile devices is underway in Canada. One organization engaged in this activity is CARROT Insights Inc. CARROT Insights Inc. is a new firm that has garnered considerable attention since its launch in 2016. Carrot Rewards provides a public engagement platform that is designed to incent users to adopt healthy behaviours, such as walking more and eating better, by rewarding them with points from popular consumer loyalty programs (e.g., points redeemed for travel, movies, gas and groceries). Individuals can download the Carrot Rewards mobile app for free from the Apple App Store and from Google Play. As “Carrot Rewards” (2018) writes, “Carrot Rewards is an AI‑driven public engagement platform that leverages behavioural economics (nudge theory) to motivate, promote and reward responsible lifestyle choices. Carrot harnesses the power of the most popular consumer loyalty points programs to maximize its appeal and user engagement.”

CARROT Insights Inc. has received funding from the Public Health Agency of Canada and the Government of British Columbia, and works in collaboration with health-related non-governmental organizations^Note  and the private sector. Since its introduction in British Columbia in 2016, the Carrot Rewards App has been launched in Ontario and in Newfoundland and Labrador. The Carrot Rewards App has generated considerable interest: “Carrot Rewards” (2018) reported “over a million Canadian consumers,” and it was recognized as Canadian App of the Year in 2017.

As part of its operations, CARROT Insights Inc. fields online surveys to its users. CARROT Insights Inc.’s early promotional materials drew attention to the wealth of behavioural and attitudinal data collected from users, and emphasized the value of this information for improving the efficiency, effectiveness and sharpness of interventions and engagement campaigns for its users and partners. CARROT Insights Inc. does not make claims regarding the generalizability of its data to the Canadian population.

In March 2017, Statistics Canada commissioned survey data collection through the Carrot Rewards App. This was part of Statistics Canada’s broader efforts to investigate alternative sources of data that might be used for various purposes (e.g., the direct estimation of population characteristics and the improvement of direct estimations using small area estimation techniques). CARROT Insights Inc. was chosen given early collaboration with the federal government, the popularity of the Carrot Rewards App, and the opportunity it offered to assess non-probabilistic online data collection. The short timeliness and relatively low costs of the Carrot Rewards Mobile App Survey (Carrot) were also considerations.

Statistics Canada included 11 questions on Carrot. These questions were drawn from the 2017 Canadian Community Health Survey (CCHS), which is composed of a nationally representative sample of Canadians. This provided a framework within which the representativeness of Carrot respondents could be assessed. Questions were selected with the aim of assessing the utility of Carrot data for use in small area estimation and other methodological techniques.

The objective of this paper was to assess the sociodemographic and health characteristics of Carrot respondents’ vis-à-vis the nationally representative sample of Canadians provided by the CCHS. Assessing the impact of the Carrot Rewards App on health behaviours and well-being is beyond both the scope of this paper and the mandate of Statistics Canada.

This paper is organized into four sections. In Section 2, the characteristics of Carrot and CCHS respondents are compared across 13 variables,^Note  and conclusions are drawn regarding the representativeness of Carrot respondents. Section 3 further explores the relationships between variables in the Carrot data and compares them with the relationships observed in the CCHS. Section 4 provides a multivariate analysis of life satisfaction reported by Carrot and CCHS respondents. The purpose of this exercise is to document further potential self-selection of Carrot respondents on unobserved characteristics. Section 5 concludes this report.

2 The characteristics of respondents of the Canadian Community Health Survey and the Carrot Rewards Mobile App Survey

In late March 2017, Statistics Canada worked with CARROT Insights Inc. and the Government of British Columbia on the selection, wording and response scales for 11 questions to be included on a Carrot survey (see Appendix Table 1). Questions were either identical to or replicated as closely as possible to the questions on the CCHS. CARROT Insights Inc. collected data in April to June 2017 in three provinces (British Columbia, Ontario, and Newfoundland and Labrador).

In late July 2017, Statistics Canada received a data file containing 297,865 observations. Of this total, 153,508 respondents completed the online survey and 144,357 did not complete it. Of the Carrot Rewards App users who did not complete the survey, 34.6% were male and 63.6% were female.^Note  Of the Carrot Rewards App users who did complete the survey, 31.2% were male and 67.4% were female.^Note  Information is not available on other characteristics of Carrot Rewards App users who did not complete the survey. No survey weights were provided or used for analysis of the Carrot data.

The CCHS is an annual cross-sectional survey that gathers information about health status, health care utilization and health determinants for the Canadian population aged 12 and older living in the 10 provinces and 3 territories. Individuals living on Indian reserves and on Crown lands, institutional residents, full-time members of the Canadian Armed Forces, youth aged 12 to 17 living in foster homes, and residents of certain remote regions were excluded from the survey’s coverage (altogether, 2% of the Canadian population aged 12 and older). The CCHS analysis below was based on 2017 data. A total of 26,730 CCHS respondents from British Columbia, Ontario, and Newfoundland and Labrador were included. CCHS respondents aged 13 or older were retained in the analysis to be consistent with the age range of Carrot respondents. All CCHS estimates were based on weighted data.^Note 

Rates of item-missing data on the CCHS and Carrot data^Note  were generally less than 5%, and below 3% in most instances (see Appendix Table 2). The only notable exception was the high rate of item-missing data on the household income question among Carrot respondents.^Note  Missing data were excluded from the CCHS–Carrot comparisons presented in Section 3.

2.1 Sociodemographic characteristics

The characteristics of the 2017 CCHS and 2017 Carrot respondents are shown in Panel 1, Table 1. Comparable information is also presented for respondents aged 25 to 44 in Panel 2, Table 1. As shown below, younger individuals were overrepresented in the Carrot data, which raises the possibility that differences observed across other variables, such as marital status and educational attainment, were simply the result of this age difference. Panel 2 assessed this possibility.

Consistent with other studies (de Bruijne and Wijnant 2013; Peterson 2012; Wells, Bailey and Link 2014), Carrot respondents were disproportionately female and disproportionately young. While 51% of CCHS respondents were female, 68% of Carrot respondents were female. A difference of a similar magnitude (17 percentage points) was observed among respondents aged 25 to 44.

In terms of age, the share of respondents in their twenties was 20 percentage points larger for Carrot respondents than for CCHS respondents (at 35% and 15% respectively), and the share in their thirties was 11 percentage points larger (at 27% and 16% respectively). Conversely, the share of Carrot respondents who were aged 55 or older was almost 27 percentage points lower than the share of CCHS respondents, at 9% and 35% respectively.

The marital status of respondents in the two data files differed as well. Smaller shares of Carrot respondents were married (a difference of 11 percentage points) compared with CCHS respondents, while larger shares were either single or common-law (differences of 6 to 7 percentage points). Among respondents aged 25 to 44, Carrot respondents were again less likely to be married than CCHS respondents (43% and 52% respectively), with most of this difference attributable to larger shares of Carrot respondents being common-law or single.

Educational attainment was considerably higher among Carrot respondents compared with CCHS respondents, with a 13 percentage point difference in the shares reporting a university degree at the bachelor’s level or higher (40% and 27% respectively). The younger age profile of Carrot respondents accounted for part of this difference. Among Carrot and CCHS respondents aged 25 to 44, the shares with a university degree or higher were 46% and 39% respectively.

The share of Carrot respondents who identified as immigrants to Canada was slightly smaller than the share of CCHS respondents, at 27% and 33% respectively. The shares were similar among respondents aged 25 to 44.

In terms of main activity, Carrot respondents were more likely than CCHS respondents to be employed (69% and 55% respectively) and less likely to be retired (4% and 20% respectively). However, this was largely due to the different age characteristics of the two samples. Among respondents aged 25 to 44, there was little difference in main activities reported.

The distributions of Carrot and CCHS respondents across household income categories were difficult to compare because 19% of all Carrot respondents (and 16% of Carrot respondents aged 25 to 44 did not respond to this question (data not shown). So while Carrot respondents were less likely than CCHS respondents to report that they resided in households with incomes of $100,000 or more (a difference of 18 percentage points), it was not possible to determine whether this was because non-response was especially high among Carrot respondents with higher household incomes.

In all three provinces, Carrot respondents were overrepresented in large urban centres. In British Columbia, 65% of Carrot respondents and 51% of CCHS respondents resided in urban centres with populations over 1 million (i.e., the Vancouver census metropolitan area [CMA]). This difference was reflected in smaller shares of Carrot respondents residing in rural areas or communities with populations under 30,000 (see Appendix Table 3). The overrepresentation of Carrot respondents in the Vancouver CMA was similar, but the difference was slightly smaller—at 11 percentage points—among British Columbia respondents aged 25 to 44 (see Appendix Table 4).

In Ontario, the shares of Carrot and CCHS respondents residing in urban centres with populations over 1 million were 52% and 48% respectively, with the difference again reflected in smaller shares of Carrot respondents in rural areas and small towns (see Appendix Table 3). There was no difference among respondents aged 25 to 44 (see Appendix Table 4).

In Newfoundland and Labrador, 60% of Carrot respondents and 39% of CCHS respondents resided in St. John’s—an urban centre with a population over 100,000. Again, this 21-percentage-point difference was reflected in smaller shares of Carrot respondents residing in rural areas (a difference of 16 percentage points) and small communities (a difference of 5 percentage points; see Appendix Table 3). The pattern was similar among respondents aged 25 to 44 (see Appendix Table 4).

To sum up thus far, the characteristics of Carrot and CCHS respondents varied across most demographic variables. Differences were largest across sex, age, and place of residence, and somewhat more modest across marital status, educational attainment and immigration status. Overall, Carrot respondents were disproportionately female, young, single or in a common-law relationship, university-educated, Canadian-born, and residents of larger urban centres. This was the case among respondents of all ages and among those aged 25 to 44.

These differences in characteristics were evident in all three provinces in which the Carrot survey was fielded, although the magnitude of the difference varied in some cases (see Appendix Table 3). For example, in British Columbia and Ontario, the shares of Carrot respondents who were female were 16 to 17 percentage points higher than the shares of CCHS respondents, while in Newfoundland and Labrador this difference was 24 percentage points. Likewise, in Newfoundland and Labrador, the overrepresentation of Carrot respondents with a university degree was larger than in British Columbia and Ontario, as was the overrepresentation of Carrot respondents whose main activity was employment.

2.2 Health and well-being characteristics

In addition to the sociodemographic differences discussed above, large differences in characteristics were evident on the four well-being measures included on both the CCHS and Carrot surveys. Given that health-behaviour improvement is a primary objective of the Carrot initiative, one might expect that individuals seeking to improve their health would be most likely to download and use the Carrot Rewards App. This might also include individuals who had been advised by a health care professional to improve their health behaviours, or those who were more likely to feel the need to make improvements in their lives.

In terms of respondents’ self-assessments of their general health, 37% of Carrot respondents and 60% of CCHS respondents rated their general health as very good or excellent. This 23-percentage-point difference was accounted for by larger shares of Carrot respondents rating their general health as good or fair. The difference was even larger—at 33 percentage points—among respondents aged 25 to 44 (see Table 1). Similarly, differences of 20 percentage points or more were evident within each of the three provinces (see Appendix Table 3) and among respondents aged 25 to 44 within each province (see Appendix Table 4).

Carrot respondents’ assessments of their mental health were also less positive than those of their CCHS counterparts. While 69% of CCHS respondents rated their mental health as very good or excellent, this was the case for only 44% of Carrot respondents—a difference of 25 percentage points. Further, 8% of CCHS respondents assessed their mental health as fair or poor, compared with 23% of Carrot respondents. The differences were of the same magnitude among respondents aged 25 to 44 (see Table 1), and were evident within all three provinces (see Appendix Table 3) and among respondents aged 25 to 44 within each province (see Appendix Table 4).

Carrot respondents were also less likely to report a strong sense of belonging to their local community. Among CCHS respondents, 72% rated their sense of community belonging as somewhat or very strong, compared with 50% of Carrot respondents. The results were much the same among respondents aged 25 to 44 (see Table 1), among those in each of the three provinces (see Appendix Table 3) and among respondents aged 25 to 44 within each province (see Appendix Table 4). This suggests that Carrot respondents tended to have weaker social ties, since the sense of community belonging is a multidimensional concept, of which social networks are an especially important component (Schellenberg et al. 2018).

Finally, Carrot respondents reported far lower levels of life satisfaction than CCHS respondents. On a response scale from 0 to 10, 71% of CCHS respondents rated their life satisfaction as 8 or above, compared with just 38% of Carrot respondents. Conversely, 29% of Carrot respondents rated their life satisfaction as 5 or below, compared with 8% of CCHS respondents. Again, results were much the same among respondents aged 25 to 44 (see Table 1), among those in each of the three provinces (see Appendix Table 3) and among respondents aged 25 to 44 within each province (see Appendix Table 4).

In addition to these four well-being measures, respondents also reported on being diagnosed with high blood pressure by a health professional. There was a small difference (4 percentage points) between CCHS (18%) and Carrot (14%) respondents who reported being diagnosed with high blood pressure. However, this difference was reversed and doubled among respondents aged 25 to 44 (see Table 1). While small differences between CCHS and Carrot respondents were observed in British Columbia and Ontario, there was a large difference in Newfoundland and Labrador, with 25% of CCHS respondents reporting being diagnosed with high blood pressure, compared with 15% of Carrot respondents (see Appendix Table 3).

Overall, Carrot respondents assessed their well-being in unfavourable terms relative to CCHS respondents, with differences of 25 to 30 percentage points on positive self-assessed ratings of general health, mental health and life satisfaction, and a difference of 20 to 25 percentage points on positive assessment of their sense of belonging to their local community. In terms of being diagnosed with high blood pressure, relatively modest differences were observed, except for in Newfoundland and Labrador.^Note 

3 Correlations among the items in both surveys

In addition to observed differences in the characteristics of Carrot and CCHS respondents, another issue regarding generalizability was the extent to which the relationships observed between variables in the Carrot data were representative of the relationships observed between the same variables among the general population (i.e., representative sample of CCHS respondents). Pearson Correlation coefficients (r) based on respondents aged 25 to 44 are shown in Table 2.^Note 

Most of the correlation coefficients in the two data sets (53 out of 66) showed the same direction of relationship (i.e., either a positive or negative correlation), although some did not. For example, the correlations between educational attainment and marital status variables were negative in the CCHS (r=‑0.09), but positive in the Carrot data (r=0.06).

There were also differences in the strength of some of the correlations observed in the CCHS and Carrot data, particularly those involving well-being measures. For example, the relationship between main activity and self-assessed general health was stronger in the CCHS data than in the Carrot data (r= 0.23 and 0.06 respectively), as were the relationships between life satisfaction and both general health (r=-0.46 and -0.28) and mental health (r= -0.53 and -0.33). Overall, the relationships observed between variables in the Carrot data were generally comparable to those observed between the same variables in the CCHS. That said, some of the correlation coefficients involving health and well-being differed in magnitude between the two data sources.

4 Life satisfaction among respondents in both surveys

Among the questions included on both the Carrot survey and CCHS were covariates of life satisfaction that have been well-established in research literature (e.g., marital status, unemployment status, general health, mental health, social ties). These were included to assess whether significant differences in life satisfaction remained between Carrot and CCHS respondents, net of these covariates. A remaining gap could point to differences that otherwise remained unobserved in the two samples. For example, individuals who feel they want or need to make improvements in their lives may be particularly likely to become Carrot Rewards App users. Indeed, it is this group for whom the app is designed. But if such characteristics, attitudes or motivations further distinguish Carrot Rewards App users from the general population, this may diminish the scope for drawing inferences regarding the general population from the Carrot data.

To compare the life satisfaction of Carrot and CCHS respondents, data from the two sources were combined into a single file. Life satisfaction was then regressed on 12 independent variables: sex, age, education, marital status, immigration status, province of residence, urban/rural place of residence, main activity, household income, self-assessed general health, self-assessed mental health, and sense of belonging to one’s local community. In addition, a dummy variable distinguishing Carrot respondents from CCHS respondents was included as a thirteenth variable, and was the focus of the analysis. The regression was limited to individuals aged 25 to 44^Note  to limit the impacts of age interactions, and the samples were re-weighted to ensure both groups contributed equally to the model.

Model 1 in Table 3 only includes the Carrot–CCHS dummy variable and indicates that, in the absence of any other covariates, life satisfaction was 1.64 points higher among CCHS respondents compared with Carrot respondents, on a scale from 0 to 10. The magnitude of this effect is enormous, similar to that typically observed between individuals in excellent health as opposed to poor/fair health. This difference remained when sociodemographic characteristics were added (Models 2 and 3). The inclusion of self-assessed general health narrowed the difference in life satisfaction among Carrot and CCHS respondents to 1.13 points (Model 4), and the subsequent addition of self-assessed mental health narrowed it further to 0.96 points (Model 5). Finally, the inclusion of the sense of community belonging reduced the Carrot–CCHS covariate to 0.93 points (Model 6).

Overall, even when many well-established and strong correlates of life satisfaction were taken into account, Carrot respondents scored 0.93 points lower than their CCHS counterparts on the 11-point life satisfaction scale. The magnitude of this difference is large. Further sets of ordinal regression analyses conducted with self-assessed general health and mental health yielded a similar magnitude of difference between Carrot and CCHS respondents (results available from the authors upon request).

There are several potential explanations for the difference in life satisfaction. First, life satisfaction responses of Carrot and CCHS respondents may be influenced by the different content of each survey (i.e., survey framing effects). This explanation is unconvincing. Bonikowska et al. (2013) showed that, across successive cycles of the General Social Survey (GSS), life satisfaction responses were lower on GSS time-use surveys and higher on GSS victimization surveys than they were on other cycles. However, the magnitude of the GSS survey framing effects, controlling for most of the same covariates (see Table 4 in Bonikowska et al. 2013), was -0.25 on GSS time-use surveys and +0.28 on the GSS victimization survey—far smaller than the difference of 0.94 observed between Carrot and the CCHS. Furthermore, the substantial differences in survey content across successive cycles of the GSS dwarf any potential framing effects imposed by the dozen or so questions on the Carrot survey. Finally, both the Carrot survey and the CCHS pertained to health, further weakening the argument that survey content drove the observed difference in life satisfaction.

Survey mode effects are a second potential explanation for Carrot–CCHS differences in life satisfaction. Individuals may respond differently to survey questions when they complete the online survey on a mobile device, instead of on a desktop device—such as a personal computer—or via telephone interview. This potential was examined, and it was found that individuals who completed the CCHS through a personal interview scored 0.16 points lower than their CCHS counterparts who participated in a telephone interview. However, Carrot respondents scored 1.70 points lower than their CCHS counterparts who had a telephone interview (see Table 4).

When sociodemographic characteristics (Model 2) were taken into account, there were no significant differences (0.07 points) between the two groups of CCHS respondents (i.e., those who had a personal interview vs. telephone interview), but Carrot respondents still scored 1.63 points lower than their CCHS counterparts who had a telephone interview. Carrot respondents continued to score 0.92 points lower on life satisfaction even after the full set of available covariates were taken into account. This suggests that survey mode effects were not a factor. This finding is consistent with Couper, Antoun and Mavletova (2017, p. 149) who concluded that “a clear positive set of findings in the literature is that there are generally few measurement error differences between those who complete a web survey on a mobile device or on a PC.” This was also highlighted in Couper’s (2017) review of the literature. Instead, the main difference associated with surveys fielded through mobile devices are “lower response rates, higher breakoff rates, and longer completion times than PC web surveys” (Couper, Antoun and Mavletova 2017, p. 149).

5 Conclusions

The survey fielded through the Carrot Rewards Application (App) highlights the potential of online data collection. Over the course of just four months, a short survey was fielded in three provinces, and responses were collected from over 153,500 Canadians. The short timelines, relatively low cost and large sample are likely attractive features for prospective clients looking to online data collection to meet their information needs. However, the results presented above clearly show that users of the Carrot Rewards App who responded to the survey differ from the Canadian general population in important ways.

Given CARROT Insights Inc.’s objective of improving health and wellness, it is not surprising that survey respondents assessed their health in less favourable terms than did the general population. Compared with the representative CCHS sample of Canadians aged 25 to 44, the Carrot Rewards Mobile App Survey (Carrot) respondents were more likely to report that they had been diagnosed with high blood pressure, more likely to rate their general health as fair or poor, and more likely to rate their mental health as fair or poor. One interpretation is that Carrot Rewards does a good job reaching Canadians who, by their own assessments, need to improve their health. In this respect, the un-representativeness of the sample could be taken as an indicator of Carrot Rewards’ success.

Other aspects of the sample also stand out. Compared with the general population, Carrot respondents were more likely to have a weak sense of belonging to their local community. This may reflect weak ties with family, friends and neighbours, although other factors—such as local amenities and duration of residence in the community—may be at play. Unpacking such issues would require more questions than may be feasible on a single online survey. The community belonging results also highlight the relationship between social ties and health, which is a growing area of international research (see Jetten, Haslam and Alexander Haslam 2012).

The relatively low levels of life satisfaction among Carrot respondents was perhaps the most striking feature of the sample. This difference remained, even after important covariates such as unemployment, general health, mental health and community belonging were taken into account. One interpretation is that, in the face of poor health and dissatisfaction with life, Carrot Rewards App users were particularly motivated to make changes. Their acquisition and use of the Carrot Rewards App may be evidence of this drive. At the risk of speculation, their relatively low levels of life satisfaction may reflect their assessment of the distance that remains between their current life and the life they hope to achieve. The Cantril Ladder underlies this assessment^Note  measure of subjective well-being collected by the Gallup World Poll and used in the World Happiness Report (Helliwell, Layard and Sachs 2018). This interpretation also highlights the potential for unobserved characteristics—in this case, motivation to change one’s life—that may further distinguish Carrot respondents from the general population.

The purposes for which data are collected are critical to evaluating data quality and fitness for use. As noted at the outset, CARROT Insights Inc. does not make claims regarding the representativeness of its respondents or the generalizability of its survey results to the Canadian population. Instead, it highlights its use of survey results to improve the effectiveness and precision of initiatives designed to improve the health and well-being of its users. Carrot survey data were certainly methodologically appropriate for this purpose. Beyond this, it would be misguided to use Carrot data to draw inferences regarding the Canadian population, given the highly selective and unrepresentative nature of the sample.

Appendix

References

Bonikowska, A., J.F. Helliwell, F. Hou, and G. Schellenberg. 2013. “An assessment of life satisfaction responses on recent statistics Canada surveys.” Social Indicators Research 118 (2): 617–643.

CRTC (Canadian Radio-television and Telecommunications Commission). 2016. Communications Monitoring Report 2016. Ottawa: Canadian Radio-television and Telecommunications Commission.

“Carrot Rewards.” 2018. CARROT Insights Inc. Available at: https://carrotapp.com/ (accessed February 15, 2019).

Couper, M.P. 2017. “New developments in survey data collection.” Annual Review of Sociology 43: 121–145.

Couper, M.P., C. Antoun, and A. Mavletova. 2017. “Mobile web surveys: A total survey error perspective.” In Total Survey Error in Practice, ed. P. Biemer, E. de Leeuw, S. Eckman, B. Edwards, F. Kreuter, L.E. Lyberg, N.C. Tucker, and B.T. West, p. 133–154. Hoboken, New Jersey: John Wiley & Sons, Inc.

De Bruijne, M., and A. Wijnant. 2013. “Comparing survey results obtained via mobile devices and computers: An experiment with a mobile web survey on a heterogeneous group of mobile devices versus a computer-assisted web survey.” Social Science Computer Review 31 (4): 482–504.

Helliwell, J., R. Layard, and J. Sachs. 2018. World Happiness Report 2018. New York: Sustainable Development Solutions Network.

Jetten, J., C. Haslam, and S. Alexander Haslam, eds. 2012. The Social Cure: Identity, Health and Well-Being. London, U.K.: Psychology Press.

Peterson, G. 2012. “Unintended mobile respondents.” CASRO Technology Conference, CASRO (Council of American Survey Research Organizations), May 31, 2012. Presentation.

Raento, M., A. Oulasvirta, and N. Eagle. 2009. “Smartphones: An emerging tool for social scientists.” Sociological Methods & Research 37 (3): 426–454.

Şahin, F., and Z. Yan. 2013. “Mobile phones in data collection: a systematic review.” International Journal of Cyber Behavior, Psychology and Learning (IJCBPL) 3 (3): 67–87.

Schellenberg, G., C. Lu, C. Schimmele, and F. Hou. 2018. “The correlates of self-assessed community belonging in Canada: Social capital, neighbourhood characteristics, and rootedness.” Social Indicators Research 140 (2): 597-618.

Schenker, N., T.E. Raghunathan, P. Chiu, D.M. Makuc, G. Zhang, and A.J. Cohen. 2006. “Multiple imputation of missing income data in the National Health Interview Survey.” Journal of the American Statistical Association 101 (475): 924–933.

Sugie, N.F. 2016. “Utilizing smartphones to study disadvantaged and hard-to-reach groups.” Sociological Methods & Research 47 (3): 458–491.

Wells, T., J.T. Bailey, and M.W. Link. 2014. “Comparison of smartphone and online computer survey administration.” Social Science Computer Review 32 (2): 238–255.