A new double hot-deck imputation method for missing values under boundary conditions
Section 4. Empirical analysis

• Previous
• Next

4.1  Data

Health insurance services in Korea are national and compulsory by law, and the data related to medical information for the entire Korean population are recorded in a national health information database. A sample cohort database is constructed by stratified random sampling from this national health information for research purposes. It maintains a cohort structure continued from a sample in 2002 (Lee, Lee, Park, Shin and Kim, 2016). Based on this recently published medical big data, several medical studies have been conducted (Kwon, Lim and Park, 2017; Kim, Kwon, Yu, Kim, Choi, Baik, Park and Kim, 2017; Kim, Lee, Kim, Kim, Choi, Baik, Choi, Pop-Busui, Park and Kim, 2015; Ko, Yoon, Kim, Kim, Kim and Seo, 2016; Ko, Jo, Park, Kim, Kim and Park, 2016; Rim, Kim, Han and Chung, 2015).

We apply imputation methods to this sample cohort data, in particular, to missing values of self-reported variables. In health screening records, there are variables measured by health-care professionals, such as height, weight, blood pressure, and blood sugar. They are reliable and completely observed. On the other hand, some health screening variables such as smoking period, exercise frequency, and drinking habits are self-reported. They are likely to be incomplete and inaccurate as discussed by Crossley and Kennedy (2002), Cambois, Robine and Mormiche (2007), and Kwon and Park (2016).

Using health screening data of the sample cohort data for 2011 and 2013, we impute the missing values of smoking periods (in years) of current smokers in 2013 whose ages were between 20 and 84 and who had health screening records both in 2011 and 2013. Let $Y_{k\mathrm{,}i}^{\text{self}}$ be the self-reported smoking period and ${\text{AGE}}_{k\mathrm{,}i}$ be the minimum value of age categorized in 5-year classes for person $i$ in year $k,$ respectively.

There is no missing data in $Y_{\mathrm{2013,}i}^{\text{self}}$ as we limited our analysis to those who answered that they were current smokers in 2013. Unreasonable values of smoking periods in $Y_{\mathrm{2013,}i}^{\text{self}}$ are treated as missing, by comparing smoking periods and ages in 2011 and 2013; i.e. if $Y_{\mathrm{2013,}i}^{\text{self}}$ meets any of the conditions $Y_{\mathrm{2013,}i}^{\text{self}}\mathrm{<mo>></mo>}{\text{AGE}}_{\mathrm{2013,}i}-a,$ $Y_{\mathrm{2013,}i}^{\text{self}}\mathrm{<mo><</mo>}{Y}_{\mathrm{2011,}i}^{\text{self}}+2-b_1\mathrm{,}$ or $Y_{\mathrm{2013,}i}^{\text{self}}\mathrm{<mo>></mo>}{Y}_{\mathrm{2011,}i}^{\text{self}}+2+b_2$ where $a$ is a minimum age started smoking, and $b_1$ and $b_2$ are tolerances according to human memory. We denote by $Y_{\mathrm{2013,}i}$ the new smoking period in 2013 to distinguish it from $Y_{\mathrm{2013,}i}^{\text{self}}$ with no missing values.

As Raghunathan et al. (2001) did, we set the upper bound, $C_{Ui},$ to be ${\text{AGE}}_{\mathrm{2013,}i}-$ minimum age started smoking. Although Raghunathan et al. (2001) defined the minimum smoking age to be 18 years old, we take the minimum age to be 10 which is the lowest smoking age observed in the sample cohort data. We set $Y_{\mathrm{2011,}i}^{\text{self}}$ for a lower boundary $C_{Li}.$ When $Y_{\mathrm{2011,}i}^{\text{self}}$ is missing which is 0.09% of data, we set $C_{Li}$ to 0. If $Y_{\mathrm{2011,}i}^{\text{self}}$ is the same as $Y_{\mathrm{2013,}i}$ then we set the $C_{Li}$ to $Y_{\mathrm{2011,}i}^{\text{self}}-0.001$ to ensure that no denominator of the proportioned residual given in equation (2.3) is zero.

By adjusting $b_1$ and $b_2$ which show the extent to which we allow the error due to human memory, the missing rate ranges from 44.0% to 73.5%. By taking $b_1\mathrm{<mo>=</mo>}{b}_2\mathrm{<mo>=</mo>}2$ when we may allow up to two years of human memory error, the rate of missing data is 44.0%. When we do not allow any error at all, that is, letting $b_1\mathrm{<mo>=</mo>}\mathrm{1,}{b}_2\mathrm{<mo>=</mo>}1$ for those who smoked in 2013 and did not smoke in 2011 and letting $b_1\mathrm{<mo>=</mo>}\mathrm{2,}{b}_2\mathrm{<mo>=</mo>}1$ for those who smoked both in 2013 and 2011, the rate of missing data is 73.5%.

In order to fit the regression model for smoking periods, we use sex, age and income level as siginificant predictors, Table 4.1 shows the summary of data we use in this paper. Individual income information used to estimate health insurance premiums was observed in the form of a categorized ordered variable with 11 levels. We re-categorized the income groups into 3 groups: high (top 30%), low (bottom 30%), and medium (others) as this improved the fit.

The average age of people who did not respond to the smoking period question 2013 is about 6 to 9 years older than that of respondents. The distribution and average of the income indicate that the nonrespondent’s income level is lower than the respondent’s income level. Since age and income level are important predictors for smoking period, it is hard to assume that missing mechanism of the smoking period is MCAR.

As seen in Table 4.1, the correlation between $Y_{\mathrm{2013,}i}$ and $Y_{\mathrm{2011,}i}^{\text{self}}$ is as high as 0.99 because of treating $Y_{2013}^{\text{self}}$ as a missing value when not satisfying the logical constraints under the assumption that $Y_{2011}^{\text{self}}$ is correct. However, $Y_{2011}^{\text{self}}$ of course has the same problem as $Y_{2013}^{\text{self}}$ and it is not reliable either. Despite such a very high correlation, this is the reason for not including $Y_{\mathrm{2011,}i}^{\text{self}}$ as a predictor. Erroneous boundary information only affects the individual imputed values, but erroneous boundary information as a predictor affects the overall regression model estimates, which greatly affects the overall reliability of imputation. Hence, we only use the measured variables such as sex, age, and income which are collected by the government as a basis for the collection of national medical insurance premiums. Note that the age variable is also used as upper boundary information.

Since it is highly possible that the self-reported smoking periods in 2011 are not correct, there should be a criticism for setting $Y_{2011}^{\text{self}}$ to be a lower bound. Thus, we consider another lower bound with $C_{Li}\mathrm{<mo>=</mo>}1$ which is the observed smallest smoking period of current smokers.

4.2  Results

The regression model for the smoking period in 2013, $Y_{\mathrm{2013,}i}$ is fitted with fully observed cases as given by Table 4.2.

In Table 4.2, sex(female) is a dummy variable with value 1 indicating female, age is the central value of age categorized in 5-year classes, and income(low) and income(mid) are both dummy variables with value 1 indicating membership to the particular income group.

Table 4.3 shows the mean of $Y_{\mathrm{2013,}i}$ estimated by each of five imputation methods with $M\mathrm{<mo>=</mo>}5$ and $m_d\mathrm{<mo>=</mo>}6.$ A possible donor size $m_d$ is allowed to be smaller than 6 when there is not enough sample to compose a donor, but there is no such case in our data. We consider four different scenarios made up of two settings of lower boundaries and two tolerances of human memory errors.

Different from the simulation results, T-NORM under-estimates more seriously than OBS when $C_{Li}\mathrm{<mo>=</mo>}1.$ The distribution of observed smoking periods is slightly skewed to the right as the distance between Q50 and Q95 is farther than that between Q50 and Q5. However, T-NORM imputes a predictive mean plus a random residual generated from a truncated normal distribution not from the empirical distribution of residuals which is right-skewed. Since the lower bound, $C_{Li}\mathrm{<mo>=</mo>}1,$ is far from the mean, the possibility of selecting a negative error is higher from the truncated normal than from the right skewed empirical distribution of residuals. This leads to the underestimation of T-NORM. All other imputation methods estimate the mean smoking period higher than OBS as they use empirical residuals.

OBS produces higher mean of smoking periods with tolerance than without tolerance as the regression coefficient of age is higher with tolerance than without tolerance as shown in Table 4.2. Except with T-NORM, the estimated smoking periods are longer without tolerance than with tolerance, and the gap is smaller when $C_{Li}\mathrm{<mo>=</mo>}1$ than $C_{Li}\mathrm{<mo>=</mo>}{Y}_{\mathrm{2011,}i}.$

The DBM-PRD method is the most robust regardless of how we define missing values and the lower boundary. This is a desirable property of imputation when the boundary information is unreliable. On the other hands, the estimation results of the existing imputation methods (i.e., T-NORM, T-MV, T-LRD, T-PMM) clearly depend on the choice of boundary and human memory tolerance. The estimated distributions of smoking period by the existing methods move substantially to the right when $C_{Li}\mathrm{<mo>=</mo>}{Y}_{\mathrm{2011,}i}$ relative to when $C_{Li}\mathrm{<mo>=</mo>}1$ since $Y_{\mathrm{2011,}i}$ is a more informative boundary than the constant boundary. However, the distribution with DBM-PRD is only marginally changed for different boundaries.

Figure 4.1 presents the kernel density estimates of smoking periods using OBS and DBM-PRD under two lower boundary settings and two tolerances of human memory errors. Imputation by DBM-PRD moves the distribution of smoking period to the right and spreads it widely, compared to the distribution constructed only by observations (OBS).

• Previous
• Article main page
• Next