How to decompose the non-response variance: A total survey error approach
Section 4. Simulation study

• Previous
• Next

The sum of the item contributions is expected to be close enough to the estimated variance due to non-response. Simulations were conducted to assess the validity of the proposed score. The goal was then to evaluate if the proposed item contribution is a good approximation of the real contribution to the total variance of a given unit. In order to do so, the total contributions of a random subset of $s_m$ were compared to the difference of the estimated variances where this subset is respectively considered as nonresponding units and responding units.

The following steps explain how simulations were performed.

1. A population was created, starting from an auxiliary variable $x$ generated according to a gamma distribution with a mean of 48 and a variance of 768. The variable of interest $y$ was created conditionally on $x$ from a gamma distribution with a mean of $\text{1}\text{.5}x$ and a variance of $16x.$ These parameters are the same as the ones set by Beaumont and Bissonnette (2011).
2. A simple random sample $s$ was selected from this population and an independent non-response subset $s_m$ was generated using Bernoulli sampling.
   1. The nonresponding units from $s_m$ were imputed using ratio imputation, where $y_k^{*}=x_k\left({\displaystyle {\sum }_{l\in s_r}{y}_l}\right){\left({\displaystyle {\sum }_{l\in s_r}{x}_l}\right)}^{-1}$ and $${\widehat{\sigma }}_k^2=x_k\frac{{\displaystyle {\sum }_{l\in s_r}{\left(y_l-y_l^{*}\right)}^2}}{{\displaystyle {\sum }_{l\in s_r}{x}_l}}.$$
   2. The population total $\widehat{t},$ the variance components ${\widehat{V}}_{•}\left(\widehat{t}\right),$ and the unit-level decompositions ${\delta }_k\left({\widehat{V}}_{•}\left(\widehat{t}\right)\right)$ were estimated, where the subscript ${}_{•}$ represents any of the variance components.

3. A subset, $\Lambda ,$ of units, $\lambda ,$ from $s_m,$ independently selected from a Bernoulli experiment, was moved from $s_m$ to $s_r$ to simulate non-response conversion. Therefore, we have a new partition, $P_s^{\left(\Lambda \right)}\text{},$ with $s_m^{\left(\Lambda \right)}=s_m\backslash \Lambda$ and $s_r^{\left(\Lambda \right)}=s_r\cup \Lambda .$
   1. The nonresponding units $k$ from $s_m^{\left(\Lambda \right)}$ were re-imputed using a ratio model which is given by $y_k^{**}=x_k\left({\displaystyle {\sum }_{l\in s_r^{\left(\Lambda \right)}}{y}_l}\right){\left({\displaystyle {\sum }_{l\in s_r^{\left(\Lambda \right)}}{x}_l}\right)}^{-1}$ and $${\widehat{\sigma }}_k^{*2}=x_k\frac{{\displaystyle {\sum }_{l\in s_r^{\left(\Lambda \right)}}{(y_l-y_l^{**})}^2}}{{\displaystyle {\sum }_{l\in s_r^{\left(\Lambda \right)}}{x}_l}}.$$
   2. The population total, ${\widehat{t}}^{\left(\Lambda \right)}\text{},$ and the variance components, ${\widehat{V}}_{•}\left({\widehat{t}}^{\left(\Lambda \right)}\right),$ were estimated.

4. The total of the unit-level decompositions, ${\displaystyle {\sum }_{\lambda \in \Lambda }{\delta }_{\lambda }\left({\widehat{V}}_{•}\left(\widehat{t}\right)\right)},$ for units $\lambda$ from $\Lambda$ was compared to the difference in the variance component estimates, ${\widehat{V}}_{•}\left(\widehat{t}\right)-{\widehat{V}}_{•}\left({\widehat{t}}^{\left(\Lambda \right)}\right).$ The relative difference in the decomposition error, $\text{DRel},$ was calculated as

$$\text{DRel}=\frac{\left({\widehat{V}}_{•}\left({\widehat{t}}^{\left(\Lambda \right)}\right)+{\displaystyle {\sum }_{\lambda \in \Lambda }{\delta }_{\lambda }\left({\widehat{V}}_{•}\left(\widehat{t}\right)\right)}\right)-{\widehat{V}}_{•}\left(\widehat{t}\right)}{{\widehat{V}}_{•}\left(\widehat{t}\right)}.(4.1)$$

Steps 1 to 4 were independently repeated with different combinations of population size, sample size, response rate, and conversion rate as described in 4.1, 4.2 and 4.3.

4.1  Simulation scenario 1: Fixed parameters

In scenario 1, population size, sample size, response rate, and conversion rate were respectively set to 400, 100, 70%, and 33.3%, with 200 independent iterations. The results are shown in Figures 4.1 and 4.2.

Both Figures 4.1 and 4.2 show that the sum of the unit-level decomposition is a good predictor of the change in the non-response component estimates. The average relative difference in the variance estimates is low at 2.1%, but the standard error is large at 5.8%. Out of 200 relative differences, only 19 are not within the +/ $–$ 10% range but they are all above 10%. If a nonrespondent is converted to a respondent, we conclude that the non-response component of the variance will approximately be reduced by the measured contribution of this unit.

4.2  Simulation scenario 2: Varying population and sample sizes

In scenario 2, the population size ranged from 100 to 50,000, with sample rate, response rate, and conversion set to 20%, 70%, and 33.3% respectively. More iterations (40) were created for the smallest population $\left(N=100\right),$ and less (10) for the largest $\left(N=\text{50,000}\right),$ for operational considerations. The results are shown in Figures 4.3 and 4.4.