Social media as a data source for official statistics; the Dutch Consumer Confidence Index
Section 4. Results

• Previous
• Next

4.1  Univariate model CCI series

The univariate analysis is based on model (3.8) from Section 3.1 applied to the series of the CCI obtained from December 2000 until March 2015. In Table 4.1, the ML estimates for the hyperparameters of the model are specified.

The average of the standard errors of the direct estimates for the CCI equals 1.21. The standard deviation of the disturbance terms of the measurement equation equals 2.46, as follows from Table 4.1. This illustrates that the population white noise dominates the variance of the measurement disturbance terms as mentioned by the choice of the variance structure for (3.4) in Section 3.1.

In the upper panel of Figure 4.1, the smoothed trend plus interventions are compared with the direct estimates for the CCI. In the lower panel of Figure 4.1, the smoothed signal, defined as trend plus interventions plus seasonals, are compared with the direct estimates for the CCI. In the series of the smoothed trend and interventions, the seasonal effect, the white noise of the population parameter and the sampling error are removed from the original series. It follows from Figure 4.1 that with the time series model a more stable estimate for the CCI can be obtained. The filtered trend plus interventions is compared with the smoothed estimates in Figure 4.2. This filtered series approximates what would be obtained in the production of official statistics if no revisions would be published. It follows that even in this case a considerable part of the high-frequency variation and seasonal fluctuations can be removed. Both figures illustrate that the Kalman filter provides plausible smoothed but also filtered imputations for the missing observation in October 2013.

Figure 4.3 shows the smoothed seasonal pattern of the CCI series. Since the seasonal effects are almost time invariant, the effects are displayed for the 12 months of one year only. There are clear significant negative effects in October, November and December and clear positive effects in January and August. The intention of the CCI is to measure a long-term confidence of respondents, since all questions refer to the respondents financial and economic situation over the last 12 month or the expectations for the future 12 months. The clear significant seasonal pattern, however, indicates that answers given by the respondents are clearly driven by a much shorter emotion, which is, among other things, subject to seasonal fluctuations.

In Figure 4.4, the standard error of the direct estimates for the CCI are compared with the standard errors of the filtered and smoothed trend plus interventions. The spikes in the standard error of the filtered and smoothed estimates are the result of the intervention variables and the missing observation in 2013. If at a certain point in time an intervention variable is activated, a new regression coefficient has to be estimated. This results in additional uncertainty in the model estimates, and shows up as a sudden peak in the standard error of the filtered and smoothed trend. In 2013, one observation is missing, which also results in additional uncertainty since the state space model produces a prediction for this missing value.

The standard errors of the smoothed estimates are slightly larger than the standard errors of the direct estimates. The standard errors of the filtered estimates are considerably larger than the standard errors of the direct estimates. This is a remarkable result. Filtered and smoothed estimates based on the time series model are based on a considerably larger set of information since sample information from preceding periods (in the case of filtered estimates) or the entire series (in the case of smoothed estimates) are used to obtain an optimal estimate for the monthly CCI. The direct estimates, on the other hand, are based on the observed sample in that particular month only. Most applications where structural time series models are applied as a form of small area estimation, result in substantive reductions of the standard error compared to the direct estimates, see e.g., van den Brakel and Krieg (2009, 2015) and Bollineni-Balabay, van den Brakel and Palm (2015, 2017).

The reason that in this application a time series modelling approach results in standard errors for filtered and smoothed times series model estimates that are larger than the standard errors of the direct estimates is a result of a large white noise component in the real population value of the CCI. Recall from Section 3.1 that the disturbance term of (3.8) contains two components; the sampling error and the unexplained high-frequency variation of the real population value, as expressed by (3.4). Recall from Table 4.1 that ${\sigma }_{\upsilon }^{}$ is equal to 2.46 and is twice as large as the average value of the standard errors of the direct estimates. This is a strong indication that the variance of the white noise component in the true population variable is of the same order as the variance of the sampling error. The direct estimator for the CCI derived in Section 2 considers the CCI in each particular month as a fixed but unknown variable. The variance of the direct estimator only measures the uncertainty since a small sample instead of the entire population is observed to estimate the CCI. It does not measure the high-frequency variation of the population value over time. This explains why the time series modelling approach does not result in a reduction of the standard error of the estimated CCI.

Although the gain in precision of level estimates obtained with the time series model is limited, the estimates for the trend are more stable as follows from Figures 4.1 and 4.2. A time series model will therefore still be useful to filter a more stable long term trend from the high-frequency variation in the population parameter and the sampling error. Because the state variables of the trend component of subsequent periods will have a strong positive correlation, more gain from the time series modelling approach can be expected by focussing on month-to-month changes, see e.g., Harvey and Chung (2000). Filtered estimates for the month-to-month change of the CCI are defined as

$${\text{Δ}}_{t|t}=L_{t|t}-L_{t-1|t}+{\beta }_{t|t}^{08}{\delta }_t^{08}-{\beta }_{t-1|t}^{08}{\delta }_{t-1}^{08}+{\beta }_{t|t}^{11}{\delta }_t^{11}-{\beta }_{t-1|t}^{11}{\delta }_{t-1}^{11},(4.1)$$

where the notation ${\text{Θ}}_{t|t^{\prime }}$ stands for the estimate for state variable $\text{Θ}$ for period $t$ given the data observed until period $t^{\prime }.$ The outlier in 2007(9) is, naturally, removed from the signal. Furthermore, the regression coefficients are time invariant. Therefore, ${\beta }_{t|t}^x={\beta }_{t-1|t}^x$ for $x=$  08 and 11. Since $t=$  2008(9) and $t=$  2011(9) are the months that ${\delta }_t^{08}$ and ${\delta }_t^{11}$ change form value, expression (4.1) can be simplified to

$${\text{Δ}}_{t|t}=L_{t|t}-L_{t-1|t}+{\beta }_{t|t}^{08}{\tilde{\delta }}_t^{08}+{\beta }_{t|t}^{11}{\tilde{\delta }}_t^{11},(4.2)$$

with ${\tilde{\delta }}_t^{08}=1$ if $t=$  2008(9) and ${\tilde{\delta }}_t^{08}=0$ for all other periods and ${\tilde{\delta }}_t^{11}=1$ if $t=$  2011(9) and ${\tilde{\delta }}_t^{11}=0$ for all other periods. Smoothed estimates for the month-to-month change of the CCI are defined as

$${\text{Δ}}_{t|T}=L_{t|T}-L_{t-1|T}+{\beta }_{t|T}^{08}{\tilde{\delta }}_t^{08}+{\beta }_{t|T}^{11}{\tilde{\delta }}_t^{11}.(4.3)$$

To compare the month-to-month changes based on (4.2) and (4.3) with the direct estimates, the smoothed seasonal effects in (3.8) are subtracted from the direct estimates. The standard errors of the direct estimates are not corrected for this adjustment.

Figure 4.5 compares the direct estimates for the month-to-month change with the smoothed estimates (upper panel) and the filtered estimates (middle panel) obtained with the time series model. The lower panel compares the standard errors of the smoothed, filtered and direct estimates. The filtered and in particular the smoothed estimates for month-to-month change have a more stable pattern compared to the direct estimates. This is also reflected by the standard errors. The strong positive correlations of the states of the trend component between subsequent periods results in standard errors for filtered and smoothed estimates of the month-to-month change that are clearly smaller compared to the direct estimator. Exceptions are the two periods where a level intervention is required. Introducing a level shift results for a short period in an increased level of uncertainty.

The reduction in standard error is measured as the Mean Relative Difference in Standard Error (MRDSE), and is for filtered estimates defined as $\text{MRDSE}=100/\left(T-t^{\prime }\right)*{\displaystyle {\sum }_{t=t^{\prime }}^T\left[\text{se}\left({\widehat{\Delta }}_t\right)-\text{se}\left({\text{Δ}}_{t|t}\right)\right]}/\text{se}\left({\widehat{\Delta }}_t\right),$ with $\text{se}\left({\widehat{\Delta }}_t \right)$ the standard error for the direct estimate for the month-to-month change. The MRDSE for smoothed estimates is obtained by replacing $\text{se}\left({\text{Δ}}_{t|t}\right)$ for $\text{se}\left({\text{Δ}}_{t|T}\right).$ During the period observed from 2003(1), the MRDSE for smoothed estimates equals 47% and for the filtered estimates 17%.

4.2  Bivariate model for CCI and SMI series

In this section, the bivariate model (3.9) proposed in Section 3.2 is applied to the series of the CCI and SMI, which are available from June 2010 until March 2015. Note that the time series components for the CCI are re-estimated using the shorter series. Maximum likelihood estimates for the hyperparameters are specified in Table 4.2. The model detects a strong positive correlation of about 0.92 between the slope disturbances of the CCI and the SMI. There is, however, no indication that both trends are cointegrated and share one common trend. A likelihood ratio test is applied to further investigate the significance of the correlation between the slope disturbances in the bivariate model. If the correlation parameter is set to zero, the log likelihood drops from -229.9 to -233.9. The $p-$ value of the corresponding likelihood ratio test equals 0.0047, indicating that the correlation between the trends of both series is clearly significantly different from zero and should not be removed from the bivariate model. If the correlation parameter is set equal to one (by choosing $d_2$ in (3.10) equal to zero), the log likelihood drops from -229.9 to -242.1. The $p-$ value of the corresponding likelihood ratio test with one degree of freedom equals zero, indicating that the trends are not cointegrated.

Figure 4.6 compares the smoothed estimates for the slope of the CCI (x-axis) and SMI (y-axis) under the model without correlation, the model with an ML estimate for the correlation $\left({\rho }_{\eta }=0.92\right)$ and the common trend model with ${\rho }_{\eta }=\text{1}\text{.0}.$ The model with uncorrelated slopes shows a clearly positive correlation between the slopes if both series are estimated independently (left panel Figure 4.6). This is picked up by the model that allows for correlation (mid panel Figure 4.6). There is however a clear deviation between the slopes of both series, which can be seen if the cross-plot of the model with a correlation estimated with ML (mid panel Figure 4.6) is compared with the cross-plot of a common factor model (right-panel Figure 4.6).

Figure 4.7 compares the observed SMI series with the smoothed trend obtained under the bivariate model. Figure 4.8 compares the direct estimates for the CCI series with the smoothed trend plus intervention under the univariate model and the bivariate model. As follows from Figure 4.8, the level and evolution of the smoothed estimates for the CCI series are almost identical under the univariate and bivariate models.

Figure 4.9 compares the standard errors of the direct estimates for the CCI series with the smoothed trend plus intervention under the univariate model and the bivariate model. For a fair comparison, the results for the univariate model and bivariate model are based on series of equal length. Therefore, the univariate model is re-estimated with the series from June 2010 until March 2015. As follows from Figure 4.9, the standard error under the bivariate model is slightly smaller compared to the standard error under the univariate model if both models are applied to series of equal length, as expected given the strong and significant positive correlation between the trend disturbance terms of both series. If, however, the univariate model is applied to the series available from December 2000, then the standard errors for the smoothed estimates under the univariate model are slightly smaller compared to the bivariate model as follows from Figure 4.10.

In conclusion, it follows that the bivariate model detects a strong correlation between the CCI and SMI series. Using the SMI series as an auxiliary series slightly improves the precision of the model based estimates for the CCI. Since the series of the CCI is nine years longer than the SMI series, the increased precision obtained with the auxiliary series is compensated in the univariate model with the additional information in the CCI series available before 2010.

The upper panel of Figure 4.11 compares the direct estimates for the month-to-month change with the smoothed estimates obtained with the univariate and bivariate time series models (both based on the series observed from June 2010). The lower panel compares the standard errors of these estimates. During the period observed from 2011(1), the MRDSE for smoothed estimates under the univariate model equals 39% and under the bivariate model 43%. The MRDSE for filtered estimates under the univariate model equals 7% and under the bivariate model 14%. As in the case of the univariate model, the time series modelling approach results in more stable and more precise estimates for the month-to-month change. The use of the SMI series slightly improves the precision of the month-to-month changes compared to the univariate model.

Once the direct estimate for the CCI for month $t$ becomes available, the additional value of the SMI series is limited to improve a time series estimate for the CCI for month $t.$ A drawback of sample surveys, however, is that they generally are less timely compared to social media sources. The additional value of the SMI becomes more clear when the higher frequency of this series is used to produce early predictions or nowcasts for the CCI with the bivariate state space model. If during month $t$ or directly at the end of month $t$ a first early prediction for the CCI is required, the univariate model can only produce a one-step-ahead prediction. As soon as during month $t$ or at the end of month $t$ results for the SMI series become available, the bivariate model exploits the strong correlation between the series to make a more precise prediction for the CCI, already before the direct estimate for month $t$ becomes available.

To illustrate the additional value of the SMI in a nowcast procedure for the CCI, we compare in the upper panel of Figure 4.12, the one-step-ahead predictions for the trend plus intervention of the CCI series obtained with the univariate model with the estimate obtained with the bivariate model if the SMI for month $t$ is available but the direct estimate of the CCI is still missing. The smoothed estimates for the trend plus intervention of the CCI obtained with the univariate model are included as a benchmark. In the lower panel the standard errors of these three estimates are compared.

If the smoothed estimates obtained with the univariate model are used as a benchmark, the Mean Absolute Relative Difference (MARD) between nowcasts and smoothed estimates is used as a measure for the size of the revision and is defined as $\text{MARD}=100/\left(T-t^{\prime }\right)*{\displaystyle {\sum }_{t=t^{\prime }}^T\left|{\theta }_{t|T}-{\theta }_{t|t-1}\right|}/\left|{\theta }_{t|T}\right|,$ where ${\theta }_t=L_t+{\beta }^{11}{\delta }_t^{11}$ denotes the trend plus intervention of the CCI series. Based on the months observed from $t=$  2013(1) the MARD for nowcasts obtained with the univariate model equals 35% and for the bivariate model 31%. This shows that the size of the revisions is a bit smaller and thus more stable with nowcasts for the CCI with the bivariate model. The difference in precision between the nowcasts obtained with the univariate model and the bivariate model are measured with the MRDSE and is in this case defined as $\text{MRDSE}=100/\left(T-t^{\prime }\right)*{\displaystyle {\sum }_{t=t^{\prime }}^T\left[\text{se}\left({\theta }_{t|t-1}^{\text{uni}}\right)-\text{se}\left({\theta }_{t|t-1}^{\text{biv}}\right)\right]}/\text{se}\left({\theta }_{t|t-1}^{\text{biv}}\right).$ Based on the months observed from $t=$  2013(1) the difference in precision of both nowcasts based on this MRDSE equals 17%. Figure 4.12 as well as the MARD and the MRDSE illustrate that the SMI improves the stability and precision of nowcasts for the CCI.

• Previous
• Article main page
• Next