Social media as a data source for official statistics; the Dutch Consumer Confidence Index
Section 2. Data

• Previous
• Next

2.1  Dutch Consumer Confidence Survey

The Consumer Confidence Index (CCI) is based on a monthly survey, called the Consumer Confidence Survey (CCS), and measures the opinion of households residing in the Netherlands about the economic climate in general and their own financial situation. The CCS is a continuous survey. Each month a self-weighted sample of approximately 2,500 households is drawn by stratified two-stage sampling from a sample frame derived from the Dutch Municipal Register. Households for which a known telephone number is available are contacted by an interviewer who completes the questionnaire by computer assisted telephone interviewing during the first ten working days of the month. On average a net sample of about 1,000 responding households is obtained, which comes down to a response rate of about 40%. A major part of the nonresponse are households for which no known telephone number of a land-line connection is available. The response among households for which a known telephone number is available is about 60%.

The CCI is based on five questions that can be answered positively, neutral or negatively. The questions refer to the economic or financial situation in the last 12 month or the respondents expectations in the future 12 months. Let $P_{1,t}^q,$ $P_{2,t}^q,$ and $P_{3,t}^q,$ denote the percentage of respondents that answered question $q=1,\dots ,5,$ in month $t$ positively, neutral or negatively, respectively. Now the CCI is defined as the difference between the percentage of positive and negative respondents, averaged over the five questions:

$$I_t=\frac{1}{Q}{\displaystyle \sum }_{q=1}^Q\left(P_{1,t}^q-P_{3,t}^q\right).(2.1)$$

Since the sample is self-weighted, and no auxiliary information is used in the estimation procedure, the percentages are estimated with the sample mean, i.e.,

$$P_{j,t}^q=\frac{100}{n_t}{\displaystyle \sum _{i=1}^n{\delta }_{i,j,t}^q},(2.2)$$

for question $q=1, \dots , 5,$ and answer category $j=1, 2, 3.$ In (2.2) $n_t$ is the net sample size in month $t,$ and ${\delta }_{i,j,t}^q$ is a dummy indicator that is equal to one if respondent $i$ chose category $j$ to question $q.$ Assuming simple random sampling without replacement for the households, it can be proved that the variance of (2.1) can be estimated by

$$\begin{array}{ll}\text{Var}\left(I_t\right)\hfill & =\frac{1}{Q^2}{\displaystyle \sum }_{q=1}^Q\left[\text{Var}\left(P_{1,t}^q\right)+\text{Var}\left(P_{3,t}^q\right)\right]-\frac{2}{Q^2}{\displaystyle \sum }_{q=1}^Q{\displaystyle \sum _{q^{\prime }=1}^Q\text{Cov}\left(P_{1,t}^q,P_{3,t}^{q^{\prime }}\right)}\hfill \\ \hfill & +\frac{1}{Q^2}{\displaystyle \sum }_{q=1}^Q{\displaystyle \sum }_{q^{\prime }\ne q}^Q\left[\text{Cov}\left(P_{1,t}^q,P_{1,t}^{q^{\prime }}\right)+\text{Cov}\left(P_{3,t}^q,P_{3,t}^{q^{\prime }}\right)\right],(2.3)\hfill \end{array}$$

with

$$\begin{array}{ll}\text{Var}\left(P_{j,t}^q\right)=\frac{1}{n_t}{P}_{j,t}^q\left(100-P_{j,t}^q\right),\hfill & \text{Cov}\left(P_{j,t}^q,P_{j,t}^{q^{\prime }}\right)=\frac{1}{n_t}\left(P_{jj,t}^{q{q}^{\prime }}-P_{j,t}^q{P}_{j,t}^{q^{\prime }}\right),\hfill \\ \text{Cov}\left(P_{j,t}^q,P_{j^{\prime }\text{},t}^{q^{\prime }}\right)=\frac{1}{n_t}\left(P_{j{j}^{\prime }\text{},t}^{q{q}^{\prime }}-P_{j,t}^q{P}_{j^{\prime }\text{},t}^{q^{\prime }}\right),\hfill & \text{Cov}\left(P_{j,t}^q,P_{j^{\prime }\text{},t}^q\right)=-\frac{1}{n_t}{P}_{j,t}^q{P}_{j^{\prime }\text{},t}^q,\hfill \\ P_{j{j}^{\prime }\text{},t}^{q{q}^{\prime }}=\frac{100}{n_t}{\displaystyle \sum _{i=1}^n{\delta }_{i,j,t}^q{\delta }_{i,j^{\prime }\text{},t}^{q^{\prime }}}.\hfill & \hfill \end{array}$$

Figure 2.1 shows the CCI with a 95% confidence interval calculated using the approach described in this section, observed during the period December 2000 through March 2015. In October 2013, the official publication of the CCI is missing.

2.2  Social media sentiment

In an attempt to reduce administration costs and response burden, Daas and Puts (2014b) developed a sentiment index from social media sources that could be used as an alternative indicator for the CCI. They used messages posted on the most popular social media platforms in the Netherlands, written in the Dutch language. These messages are classified as containing positive, neutral, or negative messages using a variant of sentence-level based classification (Pang and Lee, 2008). An index is calculated by taking the difference between the percentage of positive and negative messages.

Combinations of all Facebook and Twitter messages with and without certain filters on phrases were compared with the CCI. The combination of all publicly available Facebook messages together with filtered Twitter messages containing personal pronouns had the highest correlation with the CCI. The Twitter messages had to be filtered due to the fact that a lot of Twitter messages are not very informative. See Daas and Puts (2014b) for further details. In their research Daas and Puts (2014b) also found that major changes in the behaviour of the public on social media, such as those caused by huge events and changes in the number of messages posted on each platform, have a disturbing effect on the series. The final indicator proposed is the average of the sentiment in the Facebook and Twitter messages during each period.

In Figure 2.2, the Social Media Index (SMI) is compared with the CCI for the period June 2010 until March 2015. Both series are clearly on a different level but show a more or less similar evolution. During the presented period, the CCI is always negative, while the SMI is always positive. The size or amplitude of the movements of the CCI are also considerably larger compared to the SMI. Many factors are responsible for this difference since the CCI is based on a survey where data collection is conducted by telephone and the SMI is based on classifying messages on Twitter and Facebook. The interesting question is to which extent the evolution of both series is similar.

2.3  Quality aspects of the CCI and the SMI

The accuracy of statistics are measured with its variance and bias. For simplicity we only distinguish between selection bias and measurement bias. The variance of survey sample statistics, like the CCI, depends on the sample size and will typically constitute a substantial part of the uncertainty of sample statistics. In big data sources the concept of sampling variance is meaningless since the data generating process is not a probability sample from a finite target population. Variance components of the model used to describe the assumed data generating process could be used as an accuracy measure instead. The model-based variance of statistics obtained with time series models applied to series obtained from internet or social media will always be positive depending of the volatility of the series, which predominantly depends on the frequency of the observed series and the dynamics of the stochastic process instead of the volume of the data.

The selection bias of sample survey statistics is approximately zero under complete response. In practice however, selection bias arises due to selective nonresponse, under coverage of the sample frame and to which extend with the field work strategy the target population is successfully reached. In the case of the CCI, only the population with a known telephone number of a land-line connection is reached and the response among this subpopulation is about 60%. The selection bias of big data sources is generally unknown. In this paper, we apply the concept of cointegration to evaluate to which extend the SMI measures the same concept as the CCI. Note, however, that in the case of cointegration, the SMI might reflect a similar nonresponse and coverage selection bias as the CCI. Baker et al. (2013) pointed out that there are similarities between selection bias in probability samples and the non-probabilistic approach followed with data sources like social media.

The measurement bias in sample statistics typically depends on the extend that the conceptual variables to be measured, are implemented in the questionnaire, but also on data collection mode and the quality of the interviewers. Problems with measurement bias in surveys arises, since measurements of the variables of interest are indirect in that respondents are asked to report about their behaviour, introducing all kind of measurement errors. In the case of the CCI the question can be raised to which extend respondents are capable to express their long-term confidence in the economy and to which extend it is influenced by short-term emotions. These problems do not arise with big data if they contain direct measurements of people behaviour. With an index derived from social media like the SMI the question can be raised to which extend it measures a similar concept as the CCI. In Subsection 2.2, it was already mentioned that major changes in the behaviour of the public on social media have a disturbing effect on the series. Particularly at the end of the series, a sudden change in behaviour on social media will be very hard to distinguish from a real turning point. For example, a Google-trend series on search related to vacancies might track an official series on unemployment. It does measure unemployment, however, search behaviour before the start of the financial crisis in 2009 might be completely different compared to the period directly after the financial crisis, invalidating the concept intended to be measured.

• Previous
• Article main page
• Next