Multilevel time series modelling of antenatal care coverage in Bangladesh at disaggregated administrative levels
Section 7. Model assessment

• Previous
• Next

In this study, models were selected based on the WAIC, DIC and graphical comparisons of their trend predictions at three hierarchical levels. In addition to these model diagnostics, three discrepancy measures are defined to evaluate and compare the time-series multilevel models. The first two measures are the Relative Bias (RB) and Absolute Relative Bias (ARB), which express the differences between model estimates and direct estimates, as percentage of the latter. For a given model, ${\text{RB}}_{it}$ and ${\text{ARB}}_{it}$ for domain $i$ and (survey) year $t$ are defined as 

$${\text{RB}}_{it}\mathrm{<mo>=</mo>}\frac{\mathrm{<mo\; stretchy="false">(</mo>}{\widehat{\theta }}_{it}-{\widehat{Y}}_{it}\mathrm{<mo\; stretchy="false">)</mo>}}{{\widehat{Y}}_{it}}\times \mathrm{100<mi>\%</mi>},(7.1)$$

$${\text{ARB}}_{it}\mathrm{<mo>=</mo>}\frac{\left|{\widehat{\theta }}_{it}-{\widehat{Y}}_{it}\right|}{{\widehat{Y}}_{it}}\times \mathrm{100<mi>\%</mi>}(7.2)$$

with ${\widehat{\theta }}_{it}$ the model prediction and ${\widehat{Y}}_{it}$ the direct estimate. The third discrepancy measure is the Relative Reduction of the Standard Errors (RRSE), which measures the percentage of reduction in standard error of the model-based estimates compared to the direct estimates, i.e., 

$${\text{RRSE}}_{it}=\mathrm{100<mi>\%</mi>}\times \left(\text{se}\mathrm{<mo\; stretchy="false">(</mo>}{\widehat{Y}}_{it}\mathrm{<mo\; stretchy="false">)</mo>}-\text{se}\mathrm{<mo\; stretchy="false">(</mo>}{\widehat{\theta }}_{it}\mathrm{<mo\; stretchy="false">)</mo>}\right)/\text{se}\mathrm{<mo\; stretchy="false">(</mo>}{\widehat{Y}}_{it}\mathrm{<mo\; stretchy="false">)</mo>}\mathrm{.}(7.3)$$

The RRSE measure should not be interpreted too strictly, since design-based and model-based standard errors are conceptually different quantities. However, both are commonly used as measures of uncertainty, and once reasonable models that sufficiently account for variations over all levels of interest have been selected, based on other criteria, it is informative to use the RRSE as one of the comparison measures.

These three discrepancy measures are calculated at national, division and district (i.e., most detailed) levels. The distributions of these measures are presented in terms of the minimum value, 1^st quartile $(Q_1),$ median, mean, 3^rd quartile $(Q_3)$ and maximum value.

Additionally, observed coverage rate (CR expressed in %) for 95% confidence interval of the considered cross-sectional FH and MTS models are calculated at division and district levels by identifying whether the estimated 95% confidence interval (CI) of ${\widehat{\theta }}_{it}$ contains the direct estimates $({\widehat{Y}}_{it}).$ Coverage at the district level is the percentage of district by year combinations (about $7\times 64$ domains) where the direct estimate is included in the CI of ${\widehat{\theta }}_{it}.$ Coverage at the division level is the percentage of division by year combinations $(7\times 7$ domains) where the direct estimate is included in the CI of ${\widehat{\theta }}_{it}.$ Coverage rates are defined in a similar way for each survey year by averaging over all available districts in one particular survey year. Finally coverage is calculated for each division seperately by averaging over the 7 survey years.

The distributions of the ${\text{RB}}_{it}$ (7.1), ${\text{ARB}}_{it}$ (7.2) and ${\text{RRSE}}_{it}$ (7.3) for three administrative levels are provided in Tables 7.1, 7.2, and 7.3 for ANC0 and ANC4 for the cross-sectional FH, MTS-I, MTS-II, and MTS-III models. Table 7.1 shows that FH and MTS-I models provide lower mean RB for ANC0 and ANC4 at all three levels, while MTS-II provides slightly lower mean RB compared to MTS-III model at the district level. The ARB distributions in Table 7.2 show that the performance of MTS-II is in between MTS-I and MTS-III for all administrative levels except the national level for ANC4. The ARB values are the lowest for the cross-sectional FH model. It is also observed that the ARB increases as the domain sample size becomes smaller. Table 7.3 shows that MTS-II has the highest RRSE values at national and division levels, while at district level this model shows slightly lower RRSE than the MTS-III model for both ANC0 and ANC4. The variance reduction increases as the domain sample sizes become smaller. The reason that standard errors for the trends at national and division level under MTS-II are smaller than MTS-III is because under MTS-II the covariances between the cross-sectional FH predictions at the district level in the input series are ignored. These covariances are predominantly positive and therefore the standard errors of trends at aggregated levels are higher and more realistic under MTS-III. The higher RB, ARB and RRSE values for models MTS-II and MTS-III are a consequence of the more smooth trends obtained under both models. Small variances under smooth trends imply a larger amount of bias with respect to the direct estimates. As discussed in Section 6, these trends are more plausible compared to the cross-sectional FH model and MTS-I model, since from a subject matter point of view a smooth decreases for ANC0 and increase for ANC4 are expected.

This conclusion is confirmed by the CR values shown in Table 7.4. The CRs for the cross-sectional FH models are too high, indicating that the FH predictions tend too much to the direct estimates. The CR levels are reasonably good for MTS-I, substantially lower for MTS-II and the lowest for MTS-III. The lower coverage rates of MTS-II and MTS-III at the district level is reflected by the corresponding higher ARB and higher RRSE. These findings show that MTS-I model predictions are more volatile and tend to the direct estimates, MTS-III model predictions are highly smoothed, and MTS-II model predictions seem like a reasonable compromise between MTS-I and MTS-III model predictions, particularly at the district level.

• Previous
• Article main page
• Next