Are deep learning models superior for missing data imputation in surveys? Evidence from an empirical comparison
Section 3. Simulation-based evaluation of imputation methods

• Previous
• Next

Methods for missing data imputation are usually evaluated via real-data based simulations (van Buuren, 2018). Namely, one creates missing values from a complete dataset according to a missing data mechanism (Little and Rubin, 2014), imputes the missing values by a specific method, and then compares these imputed values with the original “true” values based on some metrics.

We first give a quick review of Rubin’s MI combination rules. Let $Q$ be the target estimand in the population, and $q^{\left(l\right)}$ and $u^{\left(l\right)}$ be the point and variance estimate of $Q$ based on the $l^{\text{th}}$ imputed dataset, respectively. The MI point estimate of $Q$ is ${\overline{q}}_L={\displaystyle {\sum }_{l\mathrm{<mo>=</mo>\; 1}}^L}{q}^{\left(l\right)}/L,$ and the corresponding estimate of the variance is equal to $T_L=\left(1+1/L\right)b_L+{\overline{u}}_L,$ where $b_L\mathrm{<mo>=</mo>}{\displaystyle {\sum }_{l\mathrm{<mo>=</mo>\; 1}}^L}{\left(q^{\left(l\right)}-{\overline{q}}_L\right)}^2/\mathrm{<mo\; stretchy="false">(</mo>}L-\mathrm{1\; <mo\; stretchy="false">)</mo>},$ and ${\overline{u}}_L\mathrm{<mo>=</mo>}{\displaystyle {\sum }_{l\mathrm{<mo>=</mo>\; 1}}^L}{u}^{\left(l\right)}/L.$ The confidence interval of $Q$ is constructed using $\left({\overline{q}}_L-Q\right)~t_{\nu }\left(\mathrm{0,}{T}_L\right),$ where $t_v$ is a $t$ -distribution with $\nu =\left(L-1\right){\left(1+{\overline{u}}_L/\left[\mathrm{<mo\; stretchy="false">(</mo>}1+1/L\mathrm{<mo\; stretchy="false">)</mo>}{b}_L\right]\right)}^2$ degrees of freedom.

The first step in our simulation-based evaluation procedure is choosing a dataset with all values observed, which is taken as the “population”. We then choose a set of target estimands $Q$ and compute their values from this population data, which are taken as the “ground truth”. The estimands are usually summary statistics of the variables or parameters in a down-stream analysis model, e.g., a coefficient in a regression model (Tang, Song, Belin and Unützer, 2005; Huque, Carlin, Simpson and Lee, 2018). Second, we randomly draw without replacement $H$ samples of size $n$ from the population data, and in each of sample $\left(h=\mathrm{1,}\dots ,H\right)$ create missing data according to a specific missing data mechanism and pre-fixed proportion of missingness. Third, for each simulated sample with missing data, we create $L$ imputed datasets using the imputation method under consideration and construct the point and interval estimate of each estimand using Rubin’s rules. Lastly, we compute performance metrics of each estimand from the quantities obtained in the previous step.

In the empirical application, we select a large complete subsample from the American Community Survey (ACS) − a national survey that bears the hallmarks of many big survey data − as our population. Since discrete variables are prevalent in the ACS, as well as in most survey data, we focus on the marginal probabilities of binary and categorical variables; e.g., a categorical variable with $K$ categories has $K-1$ estimands. To evaluate how well the imputation methods preserve the multivariate distributional properties, similar to Akande et al. (2017), we also consider the bivariate probabilities of all two-way combinations of categories in binary and categorical variables. Another useful metric is the finite-sample pairwise correlations between continuous variables. For continuous variables, the common estimands are mean, median or variance. To facilitate meaningful comparisons of the results between the categorical and continuous variables, we propose to discretize each continuous variable into $K$ categories based on the sample quantiles. We then evaluate these binned continuous variables as categorical variables based on the aforementioned estimands of marginal and bivariate probabilities.

For each estimand $Q,$ we consider three metrics. The first metric focuses on bias. To accommodate close-to-zero estimands that are prevalent in probabilities of categorical variables, we consider the absolute standardized bias (ASB) of each estimand $Q:$

$$\text{ASB}\text{=}{\displaystyle \sum _{h\mathrm{<mo>=</mo>}1}^H}\left|{\overline{q}}_L^{\left(h\right)}-Q\right|/\left(H\cdot Q\right)\mathrm{,}(3.1)$$

where ${\overline{q}}_L^{\left(h\right)}$ is the MI point estimate of $Q$ in simulation $h.$

The second metric is the relative mean squared error (Rel.MSE), which is the ratio between the MSE of estimating $Q$ from the imputed data and that from the sampled data before introducing the missing data:

$$\text{Rel}\text{.MSE}\text{=}\frac{{\displaystyle {\sum }_{h\mathrm{<mo>=</mo>\; 1}}^H{\left({\overline{q}}_L^{\left(h\right)}-Q\right)}^2}}{{\displaystyle {\sum }_{h\mathrm{<mo>=</mo>\; 1}}^H{\left({\tilde{Q}}^{\left(h\right)}-Q\right)}^2}}\mathrm{,}(3.2)$$

where ${\overline{q}}_L^{\left(h\right)}$ is defined earlier, and ${\tilde{Q}}^{\left(h\right)}$ is the prototype estimator of $Q,$ i.e., the point estimate from the complete sampled data in simulation $h.$

The third metric is coverage rate, which is the proportion of the $\alpha \mathrm{<mi>\%</mi>}$ (e.g., 95%) confidence intervals, denoted by ${\text{CI}}_h^{\alpha }$ $\left(h\mathrm{<mo>=</mo>}\mathrm{1,}\dots ,H\right),$ in the $H$ simulations that contain the true $Q:$

$$\text{Coverage}\mathrm{<mo>=</mo>}{\displaystyle \sum _{h\mathrm{<mo>=</mo>\; 1}}^H}1\left\{Q\in {\text{CI}}_h^{\alpha }\right\}/H\mathrm{.}(3.3)$$

We recommend conducting a large number of simulations (e.g., $H\ge 100)$ to obtain reliable estimates of MSE and coverage. This would not be a problem for deep learning algorithms, which can be typically completed in seconds even with large sample sizes. However, it can be computationally prohibitive for the MICE algorithms when each of the simulated data is large (e.g., $n\mathrm{<mo>=</mo>}\text{100,000}$ in some of our simulations). In the situation that one has to rely on only a few or even a single simulation for evaluation, we propose a modified metric of bias. Specifically, for each categorical variable or binned continuous variable $j,$ we define the weighted absolute bias (WAB) as the sum of the absolute bias weighted by the true marginal probability in each category:

$$\text{Weighted}\text{absolute}\text{bias}\mathrm{<mo>=</mo>}{\displaystyle \sum _{k\mathrm{<mo>=</mo>}1}^K}{Q}_{jk}\left|{\overline{q}}_{jk}^{\left(h\right)}-Q_{jk}\right|\mathrm{,}(3.4)$$

where $K$ is the total number of categories, $Q_{jk}$ is the population marginal probability of category $k$ in variable $j,$ and ${\overline{q}}_{jk}^{\left(h\right)}$ is its corresponding point estimate in simulation $h.$ We can also average the weighted absolute bias over a number of repeatedly simulated samples.

The above procedure and metrics differ from the common practice in the machine learning literature. For example, many machine learning papers on missing data imputation conduct simulations on benchmark datasets, but these data often have vastly different structure and features from survey data and thus are less informative for the goal of this paper. One such dataset is the Breast Cancer dataset in the UCI Machine Learning Repository (Dua and Graff, 2017), which has only 569 sample units and no categorical variables. Also, these simulations are usually based on randomly creating missing values of a single dataset repeatedly rather than on drawing repeated samples from a population, and thus fails to account for the sampling mechanism. Moreover, these evaluations often use metrics focusing on accuracy of individual predictions rather than distributional features. Specifically, the most commonly used metrics are the root mean squared error (RMSE) and accuracy (Gondara and Wang, 2018; Yoon, Jordon and Schaar, 2018; Lu et al., 2020). Both metrics can be defined in an overall or variable-specific fashion, but the machine learning literature usually focuses on the overall version. The overall RMSE is defined as

$$\text{RMSE}\mathrm{<mo>=</mo>}\sqrt{\frac{{\displaystyle {\sum }_{i\mathrm{<mo>=</mo>}1}^{}{\displaystyle {\sum }_j{M}_{ij}{\left({\widehat{Y}}_{ij}-Y_{ij}\right)}^2}}}{{\displaystyle {\sum }_{i\mathrm{<mo>=</mo>}1}^n{\displaystyle {\sum }_j{M}_{ij}}}}}\mathrm{,}(3.5)$$

where $Y_{ij}$ is the value of continuous variable $j$ for individual $i$ in the complete data before introducing missing data, and ${\widehat{Y}}_{ij}$ is the corresponding imputed value. For non-missing values (i.e., $M_{ij}\mathrm{<mo>=</mo>}1),$ $Y_{ij}={\widehat{Y}}_{ij}.$ The (overall) accuracy is defined for categorical variables, namely it is the proportion of the imputed values being equal to the corresponding original “true” value:

$$\text{Accuracy}\text{=}\frac{{\displaystyle {\sum }_{i\mathrm{<mo>=</mo>}1}^n{\displaystyle {\sum }_{j\in S_{\text{cat}}}{M}_{ij}\text{1}\left({\widehat{Y}}_{ij}\mathrm{<mo>=</mo>}{Y}_{ij}\right)}}}{{\displaystyle {\sum }_{i\mathrm{<mo>=</mo>}1}^n{\displaystyle {\sum }_{j\in S_{\text{cat}}}{M}_{ij}}}}\mathrm{,}(3.6)$$

where $S_{\text{cat}}$ is the set of categorical variables.

A number of caveats are in order for the RMSE and accuracy metrics. First, they are usually computed on a single imputed sample as an overall measure of an imputation method, but this ignores the uncertainty of imputations. Second, both RMSE and accuracy are single value summaries and do not capture the multivariate distributional feature of data. Third, RMSE does not adjust for the different scale of variables and can be be easily dominated by a few outliers; also, it is often computed without differentiating between continuous and categorical variables. Lastly, when there are multiple $\left(L\right)$ imputed data, a common way is to use the mean of the $L$ imputed value as ${\widehat{Y}}_{ij}$ in (3.5), but the statistical meaning of the resulting metrics is opaque. This is particularly problematic for categorical variables. For these reasons, we warn against using the overall RMSE and accuracy as the only metrics for comparing imputation methods, and one should exercise caution when interpreting them.

• Previous
• Article main page
• Next