Growth Rates Preservation (GRP) temporal benchmarking: Drawbacks and alternative solutions
Section 5. Empirical test

• Previous
• Next

In this section an illustration exercise is conducted on real-life data, in order to find out whether or not the problems mentioned in Section 3 do occur in a realistic, practical application.

5.1  Data sets

The data set used for the illustration is obtained from quarterly and annual trade as published on the website of United Nations (UN).

The United Nations Commodity Trade Statistics Database (UN Comtrade) contains data from statistical authorities of reporting countries, or are received via partner organizations like the Organisation for Economic Co-operation and Development (OECD). The United Nations Totaltrade (UN Tottrade) data are mostly taken from the International Financial Statistics (IFS), published monthly by the International Monetary Fund (IMF). Differences between both sources emerge because of differences in data collection methods and purposes (United Nations, 2017). All data are publicly available at http://comtrade.un.org/.

We use UN Tottrade as data source for quarterly data and both UN Tottrade and UN Comtrade as sources for annual data. Both data sources include imports and exports for approximately 200 UN member states.

For our application all series were selected that include three annual totals and twelve quarterly values for 2002-2004. The variables of interest are total imports and exports. Series with quarterly or annual values smaller than 0.1 million dollars were deleted, as multiplicative benchmarking methods cannot be considered appropriate for zero or near zero values (see Subsection 3.2). Since the series are in million dollars, the cutoff value only excludes “extreme” cases and still leaves some real-life cases of singularity issues.

We end up with 238 time series for Comtrade and 253 series for Tottrade. The average year to year growth rates discrepancy between the annualized quarterly series and their benchmarks are 5.9%-point and 2.7%-point for Comtrade and Tottrade benchmarks, respectively. For the majority of series the discrepancy can be considered small. The percentage of series with a maximum discrepancy below 5%-point are 79% and 87%, respectively.

5.2  Results

Our first aim is to assess overall performance. We will compare the degree of preservation of the preliminary values and their growth rates for the various methods that are discussed in this paper.

Table 5.1 shows for the five methods the median values over all series, for the functions $f_{\text{F }}^{\text{GRP}}\text{},$ $f_{\text{B}}^{\text{GRP}}\text{},$ $f_{\text{S}}^{\text{GRP}}$ for forward, backward and simultaneous movement preservation and $f_{ }^{\text{Level}}$ for preliminary value preservation. The latter function measures total squared relative adjustment, defined by

$$f_{ }^{\text{Level}}\left(x\right):={\displaystyle \sum }_{t=1}^n{\left(\frac{x_t}{p_t }-1\right)}^2.(5.1)$$

It can be seen from Table 5.1 that the GRPF method, that is designed to preserve forward growth rates, results in relatively poor backward movement preservation. The opposite is also true: GRPB does not preserve forward movements very well. From these results, we can conclude that time reversibility actually matters. Table 5.1 also demonstrates that the time symmetric methods, Denton PFD, GRPS and GRPL, perform well on all measures and that difference between those methods are only marginal.

To assess forward, backward and simultaneous growth rate preservation, a relative criterion is used that compares the values of the objective functions $f_{\text{F}}^{\text{GRP}}\left(x\right),$ $f_{\text{B}}^{\text{GRP}}\left(x\right)$ and $f_{\text{S}}^{\text{GRP}}\left(x\right)$ with their optimum values, which are obtained from GRPF, GRPB and GRPS, respectively. Analogous to the standards in Di Fonzo and Marini (2012), movement preservation is consided acceptable if it lies within 10% of the optimum value. That is, if $f^{\text{method}}\left(x\right)/f^{\text{optimum}}\left(x\right)\le \text{1}\text{.1},$ where $f$ is one of the previously mentioned objective functions.

For the five methods considered, Table 5.2 shows the percentage of time series with acceptable forward, backward and simultaneous movement preservation.

For Denton PFD an acceptable degree of simultaneous movement preservation is found for more than 95% of all cases. Thus, one can conclude that Denton PFD can be considered as a very good approximation for the optimal GRPS method; the approximation is even better than the GRPF and GRPB methods, for which acceptable performance is found for around 80% of all cases.

So far, we focused on performance for entire time series. Below we will consider the occurrence of large and extreme reconciliation adjustments made to single values and growth rates.

To measure the adjustments made to growth rates, the absolute difference $\left|g_{it}\left(x\right)-g_{it}\left(p\right)\right|*100\%$ is used, where $g_{it}$ is a growth rate for series $i$ and period $t.$ Tables 5.3 and 5.4 compare the occurrence of large and extremely large adjustments to forward, backward and simultaneous growth rates.

These tables show minor differences between methods.

Small differences between methods are also in observed in Table 5.5, which shows large and extreme corrections to preliminary values, as measured by the relative criterion $\left(x_{\text{it}}/p_{it}\right)*100\%.$

Hence, one can conclude that the problems caused by singularity do not translate into more often occurring large corrections.

Most remarkable in Table 5.5 are the negative benchmarked values obtained for GRPF in the TOT data. An example of this is illustrated in Figure 5.3.

Despite the similar results of the five benchmarking methods in Tables 5.3-5.5, there are clear differences in smoothness of reconciliation adjustments. To demonstrate this, we will use the smoothness indicator (Temurshoev, 2012).

$$\text{Smoothness}={\displaystyle \sum }_{t=2}^{n-2}{\left[{\text{BI}}_t-\overline{{\text{BI}}_t}\right]}^2\text{},(5.2)$$

where ${\text{BI}}_t$ is the so-called benchmark-to-indicator ratio, i.e., $x_t/p_t$ and $\overline{{\mathrm{BI}}_{t }}$ is the 5-terms moving average $\frac{1}{5}{\displaystyle {\sum }_{k=t-2}^{k=t+2}{\text{BI}}_k}.$

According to this indicator, we find in Table 5.6 that the smoothest results are obtained for Denton PFD and GRPL. Conversely, the asymmetric GRPF and GRPB methods yield the most irregular adjustments. It follows that the time-symmetric method GRPS, but most so GRPL, suffers less from singularity than the asymmetric methods GRPF and GRPB do. These results most clearly illustrate the problems with the singularity of GRP’s objective function that were described in Subsection 3.2.

5.3  Examples

Below we show two examples to demonstrate that the problems in Section 3 do occur in a real-life application.

The first example, in Figures 5.1 and 5.2, illustrates that non-symmetric GRP methods may change the timing of the most important economic events. When considering the first nine quarters, the two highest values occur at different time periods. GRPF’s peak periods are at quarters 6 and 7 and those of GRPB are at quarters 5 and 6. Closely related to this, is that GRPF moves away relatively slowly from the close-to-zero values at quarters 1-4.

The second example illustrates the complications of a singular objective function. As shown in Figure 5.4, GRPF closely preserves growth rates of the quarters 6-10. This comes however at the expense of an irregular peak in quarter 5 and negative benchmarked values in the quarters 11 and 12.

• Previous
• Article main page
• Next