Using Multiple Imputation of Latent Classes to construct population census tables with data from multiple sources
Section 4. Simulation results

• Previous
• Next

First, cell-proportions of univariate and multivariate cross-tables are evaluated in terms of bias and root mean squared error (RMSE) over the 500 simulation replications. Second, these cell-proportions are evaluated in terms of variance by investigating the average of the estimated standard error divided by the standard deviation over the 500 estimates obtained from the 500 simulation replications (SESD). Due to the log-transformations we made in equations 3.2, 3.3 and 3.4 to account for small cell frequencies, the RMSE and SESD are reported on a log scale.

4.1  Results in terms of bias

4.1.1  Univariate marginal frequencies of imputed variables

In Table 4.1, the simulation results can be found that cover the univariate marginal frequencies of the imputed latent variable “Gender” in terms of bias and RMSE. Results from all simulation conditions are shown. Here, it can be seen that a smaller amount of bias is obtained if $Y_{\mathrm{1,1}}$ is used, compared to results obtained using MILC under all conditions. In addition, it can be seen that the RMSE is also smaller if $Y_{\mathrm{1,1}}$ is used instead of the MILC method. Furthermore, it can be seen that both bias and RMSE slightly decrease as $M$ increases, and that the quality of the results appears to be unrelated to the missingness mechanism.

In Table 4.2, the simulation results can be found that cover the univariate marginal frequencies of the imputed latent variable “Type of family nucleus” in terms of bias and RMSE. Here, the results are very different from the results we found for “Gender”, the bias obtained for $Y_{\mathrm{2,1}}$ is much higher compared to the bias obtained using MILC under all conditions and the same holds for RMSE. In addition, whether the results for the MILC method depend on the missingness mechanism differ per category. In terms of bias and RMSE, this is the case for the categories “N.A.” and “Partners”.

In Table 4.3, the simulation results can be found that cover the univariate marginal frequencies of the imputed latent variable “Citizen” in terms of bias and RMSE. Here, the results are comparable to the results we found for “Type of family nucleus”, as the bias obtained when only $Y_{\mathrm{3,1}}$ is used is again much higher compared to the bias obtained using MILC method and the same holds for RMSE. As was also the case for “Type of family nucleus”, whether the results for the MILC method depend on the missingness mechanism differ per category, although this is more the case for the bias here, and not so much in terms of RMSE.

Boeschoten et al. (2017) concluded that the quality of the output when MILC is applied related to how well the latent class model is able to make classifications based on the observed data, which is summarized in the entropy $R^2.$ The entropy $R^2$ values for “Gender”, “Type of family nucleus” and “Citizen” are approximately 0.7352, 0.9191, and 0.8571 respectively under MCAR. So this corresponds to the quality of the results for the latent variables in terms of bias and RMSE. An additional explanation for “Gender” is that the two categories are of comparable size and the amount of misclassification in both categories is approximately equal and behaves symmetrical in our simulation study. This causes that the marginal distribution of $Y_{\mathrm{1,1}}$ is very similar to the marginal distribution of $X_1$ and not so much affected by misclassification.

4.1.2  Joint frequencies of imputed variables

In Table 4.4, the simulation results can be found that cover the joint marginal frequencies of the three imputed latent variables in terms of bias and RMSE. Again, it can be seen here that if only $Y_{v\mathrm{,1}}$ is used, severe bias is present in all cells of the joint frequency table. The results obtained when the MILC method is applied show much lower amounts of bias and RMSE. Here, the differences between different numbers for $M$ or different missingness mechanism are much smaller compared to the differences between MILC and $Y_{v\mathrm{,1}}.$ Furthermore, the differences in the amount of bias for particular cells after applying the MILC method seem to be related to imbalances in cell frequencies within particular variables. More specifically, the variable “Citizen” knows substantive differences in cell frequencies and within Table 4.4, it can be seen that particular the category “not EU” is affected in terms of bias by this imbalance.

4.1.3  Restricted cells

In Table 4.5, the simulation results can be found for the six cells that are restricted in the marginal cross-table between “Age” and “Type of family nucleus”. Under “Frequency”, it can be seen that these six cells should all contain zero observations. A combination of these scores is logically impossible. Furthermore, it can be seen that due to misclassification in $Y_{\mathrm{2,1}},$ observations containing these combinations of scores are present when $Y_{\mathrm{2,1}}$ is used to estimate this cross-table directly. In addition, it can be seen that if the MILC method is applied, such impossible combinations of scores will never be present, regardless of the missingness mechanism or the number of imputations. Furthermore, as the cells in this marginal table contain zero observations, all cells of more detailed tables covering these logically impossible combinations of scores automatically also contain zero observations.

4.1.4  The complete population frequency table

Figures 4.1 and 4.2 show results in terms of bias and root mean squared error (RMSE) when the complete census table, so the cross-table between the six variables, is estimated. As these are 42,000 cells in total, it is not feasible to evaluate them individually. Figure 4.1 and Figure 4.2 give an overview of how size of the cell frequency is related to the quality of the results. Here it can be seen that if $Y_{v\mathrm{,1}}$ are used, results in terms of bias and RMSE are related directly to cell frequency. More specifically, the relationship between cell frequency and absolute bias is approximately linear where the amount of bias is approximately 10% of the cell frequency.

4.2  Results in terms of variance

4.2.1  Univariate marginal frequencies of imputed variables

In Table 4.6, the simulation results can be found that cover the univariate marginal frequencies “Gender” in terms of se/sd. As this ratio measures whether the average standard error estimated at each replication in the simulation correctly describes the uncertainty (standard deviation) that is found over the estimates, it should be close to one. In addition, as a completely observed and finite population is assumed, variance is not estimated when $Y_{v\mathrm{,1}}$ is used. The results obtained using MILC are generally close to one and comparable to the results in terms of bias as only minor differences can be found between different values for $M$ or between the different missingness mechanisms.

In Table 4.7 and 4.8, the simulation results can be found that cover the univariate marginal frequencies for “Type of family nucleus” and “Citizen” respectively in terms of se/sd. The results found here have a very comparable structure compared to the results we found for “Gender”.

4.2.2  Joint frequencies of imputed variables

In Table 4.9, the simulation results can be found that cover the joint marginal frequencies of the imputed latent variables “Gender”, “Type of family nucleus” and “Citizen” in terms of absolute se/sd. The results found for these joint frequencies are very comparable to the results we found for the marginal frequencies. For cells with a relatively low frequency, it can be seen that the ratio is in general larger than one, indicating that the variance estimated for these frequencies (and thereby the differences between the imputations) incorporate more uncertainty than is actually found over different replications. Summarizing, the uncertainty for cells containing low frequencies is overestimated.

Results in terms for variance are not shown for the restricted cells, as a variance term cannot be estimated here.

4.2.3  The complete population frequency table

In Figure 4.3, results can be found in terms of average standard error of the cell frequencies divided by the standard deviation over the frequencies estimated in the 500 replications in the simulation study (se/sd). Here it can be seen that the standard error estimated per cell frequency is especially too large when cell frequencies are close to zero, and become closer to the nominal rate of one as the cell frequencies become larger. Apparently, variability due to missing and conflicting values is overestimated by MILC for cells with a frequency close to zero. In addition, this becomes more apparent when the number of imputations increases and it is not influenced by missingness mechanism.

4.3  Sensitivity to violations of assumptions

The simulation study presented in this paper is aimed at investigating the performance of the MILC method in a situation of misclassification in a finite population setting. When applying the MILC method in practice, a number of assumptions are made and during this simulation study these assumptions were met. To further investigate the sensitivity to violations of these assumptions, additional simulation studies were performed.

An important assumption made when applying the MILC method is that the missingness mechanism is either MCAR or MAR. Therefore, a first sensitivity analysis involves a Missing Not At Random (MNAR) mechanism. More specifically, we generated this mechanism in such a way that the probability of being missing in the survey indicator for “Type of family nucleus” depends on the latent variable “type of family nucleus” and is smallest for the first category and largest for the last category. In Table 4.10, it can be seen that the bias and RMSE increase when the mechanism is MNAR compared to MAR, while the se/sd is not affected. More specifically, it can be seen that the extent of the bias relates to how much the respective class is affected by the mechanism.

A second assumption states that the measurement error present in the indicators is random. To investigate sensitivity to the violation of this assumption, we generated a selective measurement error mechanism where the probability of measurement error in the register indicator for the variable “type of family nucleus” differs per category. Here, again the first category is least affected and the last category most. In Table 4.10 it can be seen that the effect of this selective mechanism are limited. The bias increases in a similar way as the percentage of measurement error in the respective category increases, but these are still relatively low amounts of bias. The se/sd is not affected by the mechanism.

A third assumption is that covariates do not contain measurement error. This assumption is the most remarkable, as it is typically often not the case that a coviarate does not contain measurement error. It is more likely that these variables will be treated as such because no additional information about their measurement error is known. If information was known, for example because additional survey information was present, it would have been incorporated by means of a latent variable. As in practice however there is always a probability that for some variables such information is not known, we investigate the sensitivity of the method to violation of this assumption. More specifically, we generated 5% misclassification in the covariate “marital status”, which has a relatively strong association with the latent variable “type of family nucleus”. Indeed, the bias in some categories is highly affected by this misclassification.

• Previous
• Article main page
• Next