A new double hot-deck imputation method for missing values under boundary conditions
Section 2. Double hot-deck boundary information matching proportioned residual draw

• Previous
• Next

Suppose data are composed of a fully observed explanatory variable vector $X_i$ and response variable $Y_i$ for $i\mathrm{<mo>=</mo>}\mathrm{1,}\dots ,n,$ for which some of $Y_i’s$ are missing (i.e., item missingness). Regardless of the missingness, let $Y_i$ be individually bounded and values of the boundaries be given by

$$C_{i\mathrm{,}L}\le Y_i\le C_{i\mathrm{,}U}(2.1)$$

where $C_{i\mathrm{,}U}$ and $C_{i\mathrm{,}L}$ are the upper and lower boundaries of $Y_i,$ $i\mathrm{<mo>=</mo>\; 1,}\dots ,n_0\mathrm{,}{n}_0+\mathrm{1,}\dots ,n,$ and the first $n_0$ $Y_i’s$ are observed and remaining $\left(n-n_0\right)$ $Y_i’s$ are missing.

Following Rubin (1987), we generate the regression coefficients ${\beta }^{*}$ and variance ${\sigma }^{*2}$ from the posterior distributions given by

$${\sigma }^{*2}~{\widehat{\sigma }}_{\text{OLS}}^2\left(n_{\text{obs}}-q\right)/{\chi }_{n_{\text{obs}}-1}^2\text{}\mathrm{,}{\beta }^{*}~N\left({\widehat{\beta }}_{\text{OLS}}\mathrm{,}{\sigma }^{*2}{\left(X^{T}X\right)}^{-1}\right)(2.2)$$

where $X$ is the fully observed $q$ covariates, and ${\widehat{\beta }}_{\text{OLS}}$ and ${\widehat{\sigma }}_{\text{OLS}}^2$ are the OLS estimates of regression coefficients and variance, respectively, from the regression model fitted to the observations. We then obtain the predictive means denoted by ${\widehat{Y}}_i^{\text{obs}}$ for observed $Y_i$ and ${\widehat{Y}}_j^{\text{miss}}$ for missing $Y_j.$ Then each ${\widehat{Y}}_i^{\text{obs}}$ or ${\widehat{Y}}_j^{\text{miss}}$ is located in one of following three intervals $S^k$ where $k\mathrm{<mo>=</mo>}-\mathrm{,}\mathrm{0,}+:$

$$S^{-}\mathrm{<mo>=</mo>}\left(-\infty \mathrm{,}{C}_{iL}\right)\mathrm{,}{S}^0\mathrm{<mo>=</mo>}\left(C_{iL}\mathrm{,}{C}_{iU}\right)\mathrm{,}{S}^{+}\mathrm{<mo>=</mo>}\left(C_{iU}\mathrm{,}\infty \right)\mathrm{.}$$

For observed $Y_i$ (i.e., $i\mathrm{<mo>=</mo>}\mathrm{1,}\dots ,n_0),$ we define the upper and lower proportioned residuals ${\tilde{r}}_{i\mathrm{,}U}$ and ${\tilde{r}}_{i\mathrm{,}L}\text{}:$

${\tilde{r}}_{i\mathrm{,}U}\mathrm{<mo>=</mo>}\frac{Y_i-{\widehat{Y}}_i^{\text{obs}}}{C_{iU}-{\widehat{Y}}_i^{\text{obs}}}\mathrm{<mi>\; </mi>}$  and  ${\tilde{r}}_{i\mathrm{,}L}\mathrm{<mo>=</mo>}\frac{Y_i-{\widehat{Y}}_i^{\text{obs}}}{C_{iL}-{\widehat{Y}}_i^{\text{obs}}}(2.3)$

where we assume that there is no $\widehat{Y}$ which is exactly equal to its own upper or lower boundary.

These ${\tilde{r}}_{i\mathrm{,}U}$ and ${\tilde{r}}_{i\mathrm{,}L}$ in equation (2.3) are then both divided into three sets based on the $S^k$ to which ${\widehat{Y}}^{\text{obs}}$ belongs. For $k\mathrm{<mo>=</mo>}-\mathrm{,}\mathrm{0,}+,$

$R_U^k\mathrm{<mo>=</mo>}\left\{{\tilde{r}}_{i\mathrm{,}U}\mathrm{<mo>;</mo>}{\widehat{Y}}_i^{\text{obs}}\in S^k\right\}$  and  $R_L^k\mathrm{<mo>=</mo>}\left\{{\tilde{r}}_{i\mathrm{,}L}\mathrm{<mo>;</mo>}{\widehat{Y}}_i^{\text{obs}}\in S^k\right\}\mathrm{.}$

Finally, we impute the missing $Y_j$ for $j\mathrm{<mo>=</mo>}{n}_0+\mathrm{1,}\dots ,n$ with

$Y_{j\mathrm{,}U}^{*}\mathrm{<mo>=</mo>}{\widehat{Y}}_j^{\text{miss}}+{\tilde{r}}_{j\mathrm{,}U}^{*}\left(C_{j\mathrm{,}U}-{\widehat{Y}}_j^{\text{miss}}\right)$  or  $Y_{j\mathrm{,}L}^{*}\mathrm{<mo>=</mo>}{\widehat{Y}}_j^{\text{miss}}+{\tilde{r}}_{j\mathrm{,}L}^{*}\left(C_{j\mathrm{,}L}-{\widehat{Y}}_j^{\text{miss}}\right).(2.4)$

In order to select ${\tilde{r}}_{j\mathrm{,}U}^{*}$ or ${\tilde{r}}_{j\mathrm{,}L}^{*}$ in (2.4), we now employ a hot-deck method that considers similar values as their candidates (i.e., possible donors). Hot-deck is a method for handling missing data in which each missing value is replaced with an observed response randomly selected from a donor containing similar units (Andridge and Little, 2010). We employ the following double hot-deck scheme.

1. [The first hot-deck] If ${\widehat{Y}}_j^{\text{miss}}\in S^k$ for $k\mathrm{<mo>=</mo>}-\mathrm{,}\mathrm{0,}+$ and ${\widehat{Y}}_j^{\mathrm{miss}}$ is closer to $C_{jU}$ than $C_{jL},$ then we select the corresponding upper proportioned residual set $R_U^k$ as the set of possible donors for sampling ${\tilde{r}}_{jU}^{*}.$ Likewise, if ${\widehat{Y}}_j^{\text{miss}}$ is closer to $C_{jL},$ we select $R_L^k$ for sampling ${\tilde{r}}_{jL}^{*}.$
2. [The second hot-deck] We construct possible donors from the $R_U^k$ or $R_L^k$ selected in the first hot-deck. The possible donors for sampling ${\tilde{r}}_{jU}^{*}$ consist of ${\tilde{r}}_{i\mathrm{,}U}’s$ whose $C_{iU}-{\widehat{Y}}_i^{\text{obs}}$ is close to $C_{jU}-{\widehat{Y}}_j^{\text{miss}}$ for $R_U^k.$ Similarly, possible donors of ${\tilde{r}}_{i\mathrm{,}L}’s$ whose $C_{iL}-{\widehat{Y}}_i^{\text{obs}}$ is close to $C_{jL}-{\widehat{Y}}_j^{\text{miss}}$ for $R_L^k.$
3. [Imputing] Then ${\tilde{r}}_{i\mathrm{,}U}$ or ${\tilde{r}}_{i\mathrm{,}L}$ is randomly sampled from the corresponding possible donors to impute missing $Y_j$ with $Y_{j\mathrm{,}U}^{*}$ or $Y_{j\mathrm{,}L}^{*},$ respectively and the selected ${\tilde{r}}_{i\mathrm{,}U}$ and ${\tilde{r}}_{i\mathrm{,}L}$ for $i\mathrm{<mo>=</mo>}\mathrm{1,}\dots ,n_0$ are denoted by ${\tilde{r}}_{j\mathrm{,}U}^{*}$ or ${\tilde{r}}_{j\mathrm{,}L}^{*}$ for $j\mathrm{<mo>=</mo>}{n}_0+\mathrm{1,}\dots ,n.$ Here the cases with $Y_{j\mathrm{,}U}^{*}\le C_{j\mathrm{,}L}$ and/or $Y_{j\mathrm{,}L}^{*}\ge C_{j\mathrm{,}U}$ are excluded from the possible donor set. This is rare, although it does occur.

Theorem 1 The values $Y_{j\mathrm{,}U}^{*}$  and $Y_{j\mathrm{,}L}^{*}$  always satisfy their boundary conditions.

$C_{j\mathrm{,}L}\le Y_{j\mathrm{,}U}^{*}\le C_{j\mathrm{,}U}$  and   $C_{j\mathrm{,}L}\le Y_{j\mathrm{,}L}^{*}\le C_{j\mathrm{,}U}.$

The proof is given in the Appendix.

Theorem 1 states that the boundary conditions of $Y_j$ for $j\mathrm{<mo>=</mo>}{n}_0+\mathrm{1,}\dots ,n$ are always satisfied as DBM-PRD imputes missing $Y_j$ with $Y_{j\mathrm{,}U}^{*}$ or $Y_{j\mathrm{,}L}^{*}.$ We may assume that there exists only an upper boundary value such that $C_{i\mathrm{,}L}\mathrm{<mo>=</mo>}-\infty$ for $i\mathrm{<mo>=</mo>}\mathrm{1,}\dots ,n.$ Then the first hot-deck is not needed because $R_U^k$ is automatically selected and $Y_{j\mathrm{,}U}^{*}\ge C_{j\mathrm{,}L}\mathrm{<mo>=</mo>}-\infty .$

Corollary 1.1 The DBM-PRD method is reduced to the boundary information matching method in Kwon and Park (2015), when there is only an upper or lower bound.

To examine the double hot-deck procedures used in DBM-PRD, we consider three variations of DBM-PRD. The first variation is a proportioned residual draw method (PRD) which removes the two hot-decksteps from DBM-PRD and the second variation removes the first hot-deck procedure denoted by SPRD. Thus in PRD we randomly sample from all elements in $R_U^k$ and $R_L^k.$ In SPRD, the possible donor set is based on the minimum distance from either boundary to the predictive mean. Among possible donors for ${\tilde{r}}_{j\mathrm{,}U}^{*}$ and ${\tilde{r}}_{j\mathrm{,}L}^{*},$ we select and construct final donors based solely on the distance order, without distinction between upper and lower bounds. The third variation, denoted by ${\text{SPRD}}_2,$ also removes the first hot-deck step as in SPRD and additionally changes the matching method in the second hot-deck step. The possible donor in ${\text{SPRD}}_2$ consists of ${\tilde{r}}_{i\mathrm{,}U}$ and ${\tilde{r}}_{i\mathrm{,}L}$ whose predicted mean ${\widehat{Y}}_i^{\text{obs}}$ is close to ${\widehat{Y}}_j^{\text{miss}}.$

• Previous
• Article main page
• Next