Are deep learning models superior for missing data imputation in surveys? Evidence from an empirical comparison
Section 6. Conclusion
Recent years have seen the development of many machine learning based methods for imputing missing data, raising the hope of improving over the more traditional imputation methods such as MICE. However, efforts in evaluating these methods in real world situations remain scarce. In this paper, we adopt an evaluation framework real-data-based simulations. We conduct extensive simulation studies based on the American Community Survey to compare repeated sampling properties of two MICE methods and two deep learning imputation methods based on GAN (GAIN) and denoising autoencoders (MIDA).
We find that the deep learning models hold a vast computational advantage over MICE methods, partially because they can leverage GPU power for high-performance computing. However, our simulations as well as evaluation on several “benchmark" data suggest that MICE with CART specification of the conditional models consistently outperforms, usually by a substantial margin, the deep learning models in terms of bias, mean squared error, and coverage under a wide range of realistic settings. In particular, GAIN and MIDA tend to generate unstable imputations with enormous variations over repeated samples compared with MICE. One possible explanation is that deep neural networks excel at detecting complex sub-structures of big data, but may not suit for data with simple structure, such as the simulated data used here. Another possibility is that the sample sizes in our simulations are not adequate to train deep neural networks, which usually required much more data compared to traditional statistical models.
These results contradict previous findings based on the single performance metric of overall mean squared error in the machine learning literature (e.g., Gondara and Wang, 2018; Yoon, Jordon and Schaar, 2018; Lu et al., 2020). This discrepancy highlights the pitfalls of the common practice in the machine learning literature of evaluating imputation methods. It also demonstrates the importance of assessing repeated-sampling properties on multiple estimands of MI methods. An interesting finding is that ensemble trees (e.g., RF) do not improve over a single tree (e.g., CART) in the context of MICE, which matches the findings in another recent study (Wongkamthong and Akande, 2021). Combined with the fact that the former is more computationally intensive than the latter, we recommend using MICE with CART instead of RF in practice.
Our study has a few limitations. First, there are many deep learning methods that can be adapted to missing data imputation and all may have different operating characteristics. We choose GAIN and MIDA because both generative adversarial network and denoising autoencoders are immensely popular deep learning methods, and the imputation methods based on them have been advertised as superior to MICE. Nonetheless, it would be desirable to examine other deep learning based imputation methods in future research. Second, performance of machine learning methods is highly dependent on hyperparameter selection. So it can be argued that the inferior performance of GAIN and MIDA may be at least partially due to sub-optimal hyperparameter selection. However, practitioners would most likely rely on default hyperparameter values for any machine learning based imputation methods, which is indeed what we have adopted in our simulations and thus represents the real practice. Third, we did not consider the joint distribution between any categorical and continuous variables, but our evaluations within categorical and continuous variables have yielded consistent conclusions. Lastly, as any simulation study, one should exercise caution in generalizing the conclusions. By carefully selecting the data and metrics, we have attempted to closely mimic the settings representative of real survey data so that our conclusions are informative for practitioners who deal with similar situations. Additional evaluation studies based on different data are desired to shed more insights on the operating characteristics and comparative performances of different missing data imputation methods. Data, code, and supplementary material for the paper are available at: https://github.com/zhenhua-wang/MissingData_DL.
Acknowledgements
Poulos and Li’s research is supported by the National Science Foundation under Grant DMS-1638521 to the Statistical and Applied Mathematical Sciences Institute.
References
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y. and Zheng, X. (2015). TensorFlow: Large-scale machine learning on heterogeneous systems [Software available from tensorflow.org], https://www.tensorflow.org/.
Akande, O., Li, F., and Reiter, J. (2017). An empirical comparison of multiple imputation methods for categorical data. The American Statistician, 71(2), 162-170.
Arnold, B.C., and Press, S.J. (1989). Compatible conditional distributions. Journal of the American Statistical Association, 84, 152-156.
Barnard, J., and Meng, X.-L. (1999). Applications of multiple imputation in medical studies: From AIDS to NHANES. Statistical Methods in Medical Research, 8(1), 17-36.
Berthelot, D., Schumm, T. and Metz, L. (2017). BEGAN: Boundary Equilibrium Generative Adversarial Networks. CoRR, abs/1703.10717. http://arxiv.org/abs/1703.10717.
Breiman, L. (2001). Random forests. Machine Learning, 45, 5-32.
Breiman, L., Friedman, J.H., Olshen, R.A. and Stone, C.J. (1984). Classification and Regression Trees. Belmont, CA: Wadsworh, Inc.
Burgette, L., and Reiter, J.P. (2010). Multiple imputation via sequential regression trees. American Journal of Epidemiology, 172, 1070-1076.
Cao, W., Wang, D., Li, J., Zhou, H., Li, L. and Li, Y. (2018). BRITS: Bidirectional recurrent imputation for time series. Advances in Neural Information Processing Systems, 6775-6785.
Che, Z., Purushotham, S., Cho, K., Sontag, D. and Liu, Y. (2018). Recurrent neural networks for multivariate time series with missing values. Scientific Reports, 8(1), 1-12.
Chen, S., and Haziza, D. (2019). Recent developments in dealing with item nonresponse in surveys: A critical review. International Statistical Review, 87, S192-S218.
De Leeuw, E.D., Hox, J. and Huisman, M. (2003). Prevention and treatment of item nonresponse. Journal of Official Statistics, Stockholm, 19(2), 153-176.
Doove, L., Van Buuren, S. and Dusseldorp, E. (2014). Recursive partitioning for missing data imputation in the presence of interaction effects. Computational Statistics & Data Analysis, 72, 92-104.
Dua, D., and Graff, C. (2017). UCI Machine Learning Repository. http://archive.ics.uci.edu/ml.
Fortuin, V., Baranchuk, D., Rätsch, G. and Mandt, S. (2020). GP-VAE: Deep probabilistic time series imputation. International Conference on Artificial Intelligence and Statistics, 1651-1661.
Gelman, A., and Speed, T.P. (1993). Characterizing a joint probability distribution by conditionals. Journal of the Royal Statistical Society Series B: Statistical Methodology, 55, 185-188.
Glorot, X., and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. Artificial Intelligence and Statistics, 9, 249-256.
Gondara, L., and Wang, K. (2018). MIDA: Multiple imputation using denoising autoencoders. Pacific-Asia Conference on Knowledge Discovery and Data Mining, 260-272.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. and Bengio, Y. (2014). Generative adversarial nets. Advances in Neural Information Processing Systems, 2672-2680.
Ham, H., Jun, T.J. and Kim, D. (2020). Unbalanced Gans: Pre-Training the Generator of Generative Adversarial Network Using Variational Autoencoder. arXiv preprint arXiv:2002.02112.
Harel, O., and Zhou, X.-H. (2007). Multiple imputation: Review of theory, implementation and software. Statistics in Medicine, 26(16), 3057-3077.
Hastie, T., Tibshirani, R. and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference and Prediction (2nd Ed.), Springer.
Haziza, D., and Vallée, A.-A. (2020). Variance estimation procedures in the presence of singly imputed survey data: A critical review. Japanese Journal of Statistics and Data Science, 3(2), 583-623.
Ho, T.K. (1995). Random decision forests. Proceedings of 3rd International Conference on Document Analysis and Recognition, 1, 278-282.
Honaker, J., King, G. and Blackwell, M. (2011). Amelia II: A program for missing data. Journal of Statistical Software, 45(7), 1-47.
Horton, N.J., Lipsitz, S.R. and Parzen, M. (2003). A potential for bias when rounding in multiple imputation. The American Statistician, 57(4), 229-232.
Huque, M.H., Carlin, J.B., Simpson, J.A. and Lee, K.J. (2018). A comparison of multiple imputation methods for missing data in longitudinal studies. BMC Medical Research Methodology, 18(1), 1-16.
Kingma, D., and Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv:1412.6980.
Li, F., Yu, Y. and Rubin, D. (2012). Imputing Missing Data by Fully Conditional Models: Some Cautionary Examples and Guidelines. Technical report, Duke University Department of Statistical Science Discussion Paper, 11-24.
Li, F., Baccini, M., Mealli, F., Zell, E.R., Frangakis, C.E. and Rubin, D.B. (2014). Multiple imputation by ordered monotone blocks with application to the anthrax vaccine research program. Journal of Computational and Graphical Statistics, 23(3), 877-892.
Lipton, Z.C., Kale, D.C. and Wetzel, R. (2016). Modeling missing data in clinical time series with RNNs. Machine Learning for Healthcare, 56.
Little, R.J., and Rubin, D.B. (2014). Statistical Analysis with Missing Data. Hoboken, NJ: John Wiley & Sons, Inc.
Little, R.J., and Rubin, D.B. (2019). Statistical Analysis with Missing Data, 3rd edition. New York: John Wiley & Sons, Inc.
Lu, H.-M., Perrone, G. and Unpingco, J. (2020). Multiple Imputation with Denoising Autoencoder Using Metamorphic Truth and Imputation Feedback. arXiv preprint arXiv:2002.08338.
Maas, A.L., Hannun, A.Y. and Ng, A.Y. (2013). Rectifier nonlinearities improve neural network acoustic models. Proc. ICML, (1), 3.
Manrique-Vallier, D., and Reiter, J. (2014). Bayesian estimation of discrete multivariate truncated latent structure models. Journal of Computational and Graphical Statistics, 23, 1061-1079.
Monti, F., Bronstein, M. and Bresson, X. (2017). Geometric matrix completion with recurrent multi-graph neural networks. Advances in Neural Information Processing Systems, 3697-3707.
Murray, J.S., and Reiter, J.P. (2016). Multiple imputation of missing categorical and continuous values via bayesian mixture models with local dependence. Journal of the American Statistical Association, 111(516), 1466-1479.
Raghunathan, T.E., Lepkowski, J.M., Van Hoewyk, J. and Solenberger, P. (2001). A multivariate technique for multiply imputing missing values using a sequence of regression models. Survey Methodology, 27, 1, 85-95. Paper available at https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2001001/article/5857-eng.pdf.
Reiter, J.P., and Raghunathan, T.E. (2007). The multiple adaptations of multiple imputation. Journal of the American Statistical Association, 102(480), 1462-1471.
Royston, P., and White, I.R. (2011). Multiple imputation by chained equations (mice): Implementation in Stata. Journal of Statistical Software, 45(4), 1-20.
Rubin, D.B. (1976). Inference and missing data (with discussion). Biometrika, 63, 581-592.
Rubin, D.B. (1987). Multiple Imputation for Nonresponse in Surveys. New York: John Wiley & Sons, Inc.
Rubin, D.B. (1996). Multiple imputation after 18+ years. Journal of the American Statistical Association, 91(434), 473-489.
Schafer, J.L. (1997). Analysis of Incomplete Multivariate Data. London: Chapman & Hall.
Shah, A., Bartlett, J., Carpenter, J., Nicholas, O. and Hemingway, H. (2014). Comparison of random forest and parametric imputation models for imputing missing data using mice: A caliber study. American Journal of Epidemiology, 179, 764-74.
Stekhoven, D.J., and Bühlmann, P. (2012). Missforest − non-parametric missing value imputation for mixed-type data. Bioinformatics, 28(1), 112-118.
Su, Y.-S., Gelman, A.E., Hill, J. and Yajima, M. (2011). Multiple imputation with diagnostics (mi) in r: Opening windows into the black box. Journal of Statistical Software, 45.
Tang, L., Song, J., Belin, T.R. and Unützer, J. (2005). A comparison of imputation methods in a longitudinal randomized clinical trial. Statistics in Medicine, 24(14), 2111-2128.
van Buuren, S. (2018). Flexible Imputation of Missing Data. CRC Press LLC.
van Buuren, S., Brand, J.P.L., Groothuis-Oudshoorn, C.G.M. and Rubin, D.B. (2006). Fully conditional specification in multivariate imputation. Journal of Statistical Computation and Simulation, 76(12), 1049-1064.
van Buuren, S., and Groothuis-Oudshoorn, K. (2011). mice: Multivariate imputation by chained equations in r. Journal of Statistical Software, 45(3), 1-67.
Vincent, P., Larochelle, H., Bengio, Y. and Manzagol, P.-A. (2008). Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th International Conference on Machine Learning, 1096-1103.
Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.-A. and Bottou, L. (2010). Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research, 11(12).
White, I.R., Royston, P. and Wood, A.M. (2011). Multiple imputation using chained equations: Issues and guidance for practice. Statistics in Medicine, 30(4), 377-399.
Wongkamthong, C., and Akande, O. (2021). A comparative study of imputation methods for multivariate ordinal data. Journal of Survey Statistics and Methodology, in press.
Yoon, J., Jordon, J. and Schaar, M. (2018). Gain: Missing data imputation using generative adversarial nets. International Conference on Machine Learning, 5689-5698.
Yoon, J., Zame, W.R. and van der Schaar, M. (2018). Estimating missing data in temporal data streams using multi-directional recurrent neural networks. IEEE Transactions on Biomedical Engineering, 66(5), 1477-1490.
Yuan, Y. (2011). Multiple imputation using SAS software. Journal of Statistical Software, 45(6), 1-25.
- Date modified: