Bayesian spatial models for estimating means of sampled and non-sampled small areas
Section 5. Application to the Current Population Survey data

• Previous
• Next

In this section, we evaluate the spatial models in terms of their prediction accuracy for some state-level population median incomes. The U.S. Department of Health and Human Service (HHS) annually needed accurate data for median incomes for states to implement a welfare program. While accurate national median income data are available from the Current Population Survey (CPS), the CPS data do not provide accurate state-level median income data. To supply accurate statistics to the HHS, the U.S. Census Bureau considered model-based small area estimation methods by utilizing auxiliary data from other federal programs. We apply proposed spatial models to estimate 1989 four-person family median incomes for the contiguous forty-nine U.S. states, including the District of Columbia. We use the direct estimates of 1990 CPS and compare our predictions with the more reliable statistics from the long form from the 1990 Census, i.e., we consider the statistics from the long form from the 1990 Census as the true values. Prediction performances are measured using all small areas and a subset of areas after leaving out multiple direct estimates.

5.1   Four-person family median income estimation

Let ${\theta }_i$ be the true four-person family median income of the $i^{\text{th}}$ state for the year 1989, where $i=\mathrm{1,}\dots \mathrm{,}49.$ The states of Alaska and Hawaii are excluded as they are not geographically connected to the mainland. Let $Y_i$ be the direct estimate of ${\theta }_i$ from the 1990 CPS. The covariates of interest are 1980 Census median income $x_{i1}$ and an adjusted 1980 Census median income $x_{i2}$. The adjusted Census median income $x_{i2}$ is defined as $\left({\text{PCI}}_{i\mathrm{,}1989}/{\text{PCI}}_{i\mathrm{,}1979}\right)x_{i1},$ $i=\mathrm{1,}\dots \mathrm{,}m,$ where ${\text{PCI}}_{i\mathrm{,}1979}$ and ${\text{PCI}}_{i\mathrm{,}1989}$ are the 1979 and 1989 per capita incomes of the $i^{\text{th}}$ state provided by the Bureau of Economic Analysis of the U.S. Department of Commerce. It has been known that the adjusted Census median income is a good covariate which very effectively accounts for the variability of the small area median income.

With the noninformative prior (2.14) with $\alpha =0,$ we fit all five models as described by (2.8)-(2.10) with $X=\mathrm{<mo\; stretchy="false">[</mo>}{1}_m\mathrm{,}{x}_1\mathrm{,}{x}_2\mathrm{<mo\; stretchy="false">]</mo>}$ and $X=\mathrm{<mo\; stretchy="false">[</mo>}{1}_m\mathrm{,}{x}_1\mathrm{<mo\; stretchy="false">]</mo>},$ where for the second covariate setting, we exclude the adjusted Census median income from the fitted model. For each model considered, we run 4 parallel HMC chains for 2,500 iterations after 5,000 burn-in iterations using rstan (Stan Development Team, 2018). We retain every ${10}^{\text{th}}$ iteration and concatenate the 4 chains to obtain a posterior sample of size 10,000. For all models and parameters, the potential scale reduction factors $(\widehat{R};$ Gelman and Rubin, 1992) are all one indicating no lack of convergence. The potential scale reduction factors are provided in Figure 5.1 and Table 5.1.

Using the posterior means, ${\widehat{\theta }}_i\text{’s},$ we calculate the squared prediction errors from respective ${\theta }_i\text{’s}$ and obtain the mean squared prediction error (MSPE), defined in Section 4, by averaging the $m=49$ squared deviations. The average posterior standard deviations (APSD) associated with ${\widehat{\theta }}_i\text{’s}$ are used to quantify the uncertainty of predictions, and the widely applicable information criterion (WAIC; Watanabe and Opper, 2010) is used to evaluate and compare the models, where a smaller WAIC value indicates a better model fit.

Table 5.2 summarizes various evaluation measures we considered and the respective percentage improvements (PI) of MSPE, APSD. When both covariates are available, the LCAR model has approximately 14% smaller MSPE and 4% smaller APSD than the independent FH model. In terms of MSPE, the second best performing model is the SAR having approximately 9.5% smaller MSPE. When only $x_1$ (week covariate) is included in the fitted model, the SAR model has approximately 40% smaller MSPE and 14% smaller APSD than the independent FH model. The CAR and LCAR models show competitive performances having approximately 36% smaller MSPE and 13% smaller APSD over the independent FH model. By removing the strong covariate $x_2$ from the full model, the MSPE of the SAR and LCAR models increase approximately 66% and 84%, respectively, whereas the MSPE of the independent FH model increases more than 150%.

In terms of the goodness of fit, the independent FH model shows the best fit (the smallest WAIC) when both covariates are included. Conversely, when only $x_1$ (week covariate) is included in the fitted model, the independent FH model shows the worst fit having the largest WAIC value. However, considering the standard errors given in parentheses, there is no significant difference in the model fit.

Table 5.3 summarizes the posterior distributions of $\rho$ in terms of the posterior mean, mode, and standard deviation. When all covariates are included in the fitted model, the posterior distributions of $\rho$ indicate no strong spatial dependency having posterior means centered around zero with large standard deviations. In contrast, when only $x_1$ (week covariate) is included in the fitted model, $\rho$ becomes very significant illustrating the posterior distributions concentrated near the upper limit of its support.

In summary, when the weak covariate does not adequately explain existing spatial variation, the spatial models produce significantly better predictions accounting for the spatial variation. When no substantial spatial variation remains in the residual, they make marginally better predictions than the independent FH model without sacrificing model fit.

5.2   Estimation of some non-sampled state means excluding their CPS values

In this section, we evaluate the spatial models in terms of their prediction accuracy for non-sampled small areas using the 1980 Census median income $x_1.$ Specifically, we randomly exclude CPS estimates (direct estimates) of multiple states at each instance and make predictions for ${\theta }_i\text{’s}$ of the excluded states. As there are 49 small areas (states), we created 12 datasets that lack direct estimates for $m_1=4$ or 5 areas, where $m_1$ is the number of non-sampled small areas as in Section 2.2. Excluded states for each dataset are listed in Table 5.4.

For each dataset, we fit the independent FH model and four spatial models as specified in (2.11)-(2.13) with the noninformative prior (2.13) with $\alpha =0$ running HMC chains under the same setting as in Section 5.1. The $\widehat{R}$  values show no evidence of lack of convergence, where detailed values are provided in Figure 5.2 and Table 5.5. For each non-sampled area, the squared prediction error ${\text{SPE}}_i={\mathrm{<mo\; stretchy="false">(</mo>}{\widehat{\theta }}_i-{\theta }_i\mathrm{<mo\; stretchy="false">)</mo>}}^2$ and posterior standard deviation $\mathrm{sd}\mathrm{<mo\; stretchy="false">(</mo>}{\theta }_i\mathrm{<mo\; stretchy="false">)</mo>}$ are obtained for each model. Based on these, the prediction performance is compared with the following ratio: for $i=\mathrm{1,}\dots \mathrm{,}m$ and $k=\mathrm{2,}\dots \mathrm{,}5,$

$$\text{SPE}-{\text{Ratio}}_{ki}\mathrm{<mo>=</mo>}\frac{{\text{SPE}}_{ki}}{{\text{SPE}}_{1i}}\mathrm{,}\text{PSD}-{\text{Ratio}}_{ki}\mathrm{<mo>=</mo>}\frac{{\text{sd}}_k\mathrm{<mo\; stretchy="false">(</mo>}{\theta }_i\mathrm{<mo\; stretchy="false">)</mo>}}{{\text{sd}}_1\mathrm{<mo\; stretchy="false">(</mo>}{\theta }_i\mathrm{<mo\; stretchy="false">)</mo>}}(5.1)$$

where ${\text{sd}}_k\mathrm{<mo\; stretchy="false">(</mo>}{\theta }_i\mathrm{<mo\; stretchy="false">)</mo>}$ is the posterior standard deviation of ${\theta }_i$ under the $k^{\text{th}}$ model. A value of $\text{SPE}-{\text{Ratio}}_{ki}(\text{PSD}-{\text{Ratio}}_{ki})$ less than one indicates that the $k^{\text{th}}$  spatial model has a smaller squared prediction error (posterior standard deviation) than the independent FH model. In Figures 5.3 and 5.4, we display the ratios using a red and blue color scheme to denote ratios greater than and less than one, respectively, where a darker color represents a more extreme value.

Overall, SAR, SCAR, CAR, and LCAR models exhibit smaller SPEs in 35, 41, 36, and 36 states, respectively. The SCAR model has the greatest number of states in which its predictions perform better than those of the independent FH model, but the overall improvements are the least. In more than 35 states, the SAR, CAR, and LCAR models produce more accurate predictions than the independent FH model, and in three states (New Mexico, Oregon, and Wisconsin), their SPEs are more than 100 times smaller. For California, Minnesota, and South Carolina, all spatial models make worse predictions than the independent FH model. California and Minnesota have much higher median incomes than surrounding states, whilst South Carolina has a significantly lower median income. Among the 49 states, these three states have the second, seventh, and nineteenth smallest local Moran’s I values. This illustrates that if the small area mean of an non-sampled area is significantly different from the means of surrounding areas, then the spatial models may produce inferior predictions. The model that demonstrates the best fit in terms of WAIC is either the SCAR, CAR, or LCAR model, where the exact numbers are provided in Table 5.4.

• Previous
• Article main page
• Next