Replication variance estimation after sample-based calibration
Section 4. Application

• Previous
• Next

We return to the calibration problem encountered while bridging the two 2016 FHWAR surveys. For both surveys, the population is defined as individuals of ages 16 and older, living in U.S. households. The main data sources for this application are record-level data files, containing weights and replicate weights for both surveys. Using these datasets, we conducted an initial analysis and identified discrepancies in the demographics, which we adjusted by sample-based raking. Estimated population totals constructed using the record-level data from the National survey were considered as random controls, for the crosstabs of census divisions (nine categories) and each of the following demographic variables:

• residency: two categories corresponding to urban and rural classification,
• age: eight categories corresponding to age ranges 16-17; 18-24; 25-34; 35-44; 45-54; 55-64; 65-74, and 75+,
• sex: two categories corresponding to male and female,
• race-ethnicity: four categories corresponding to Hispanic, non-Hispanic White, non-Hispanic African American, and non-Hispanic all other,
• annual income: nine categories corresponding to income ranges -$20,000; $20,000-$29,999; $30,000-$39,999; $40,000-$49,999; $50,000-$74,999; $75,000-$99,999; $100,000-$149,999; $150,000+, and not reported.

For the application in this article, we use the 50-State survey public use file, which does not contain information on income. Therefore, we illustrate the proposed method in a slightly simplified setting here, using the crosstabs of census divisions and residency, age, sex, and race-ethnicity as the raking dimensions. We implemented both the Fuller (1998) method and the proposed calibration method described in Section 2 using the public-use data files available for both surveys. For comparison, we also show the results of calibrating without adjusting the variance estimates for the random controls, referred to below as the “naive” method because it ignores the variability of the controls in the variance estimates. To compare the variance estimation methods for survey variables that are not control variables, we will also show estimates for domains defined by crosstabs of residency and sex.

While the replication methods of the two FHWAR surveys are different, they both use $R\mathrm{<mo>=</mo>}{R}_C\mathrm{<mo>=</mo>}160$ replicates. Referring to expression (2.1), the replication constant for the DAGJK method of the 50-State survey is $A\mathrm{<mo>=</mo>}159/160$ and the corresponding constant for the SDR method of the National survey is $A_C\mathrm{<mo>=</mo>}4/160,$ both available from their respective survey documentation. Hence, the replication adjustment constants $a_r$ in (2.10) are equal to $2/\sqrt{159}$ for $r\mathrm{<mo>=</mo>}\mathrm{1,}\dots \mathrm{,}R.$

The estimates we will consider are all estimated domain counts, so we define the target variable $y_i\mathrm{<mo>=</mo>}{I}_{\mathrm{<mo>\{</mo>}i\in U_d\mathrm{<mo>\}</mo>}}$ for a domain of interest $U_d.$ For the 144 domains defined by the raking dimensions, we write the estimated domain counts as ${\widehat{t}}_k\mathrm{<mo>=</mo>}{\displaystyle {\sum }_s{w}_i{I}_{\mathrm{<mo>\{</mo>}i\in U_k\mathrm{<mo>\}</mo>}}},k\mathrm{<mo>=</mo>}\mathrm{1,}\dots \mathrm{,}144.$ Likewise, the control totals are estimated domain counts from the National survey, so the auxiliary variable vector is $x_i\mathrm{<mo>=</mo>}{I}_i,$ a vector of length 144 containing the indicators for inclusion of respondent $i$ in the control domains $U_k\mathrm{,}k\mathrm{<mo>=</mo>}\mathrm{1,}\dots \mathrm{,}144,$ and let ${\widehat{t}}_{C\mathrm{,}k}\mathrm{<mo>=</mo>}{\displaystyle {\sum }_{s_C}{w}_{Ci}{I}_{\mathrm{<mo>\{</mo>}i\in U_k\mathrm{<mo>\}</mo>}}}.$ We denote the vector of control totals as ${\widehat{t}}_C\mathrm{<mo>=</mo>}{\mathrm{<mo\; stretchy="false">(</mo>}{\widehat{t}}_{C\mathrm{,}1}\mathrm{,}\dots \mathrm{,}{\widehat{t}}_{C\mathrm{,}144}\mathrm{<mo\; stretchy="false">)</mo>}}^T$ and the adjusted replicate control totals are ${\widehat{t}}_C^{*\mathrm{<mo\; stretchy="false">(</mo>}r\mathrm{<mo\; stretchy="false">)</mo>}}={\widehat{t}}_C+2/\sqrt{159}\mathrm{<mo\; stretchy="false">(</mo>}{\widehat{t}}_C^{\mathrm{<mo\; stretchy="false">(</mo>}r\mathrm{<mo\; stretchy="false">)</mo>}}-{\widehat{t}}_C\mathrm{<mo\; stretchy="false">)</mo>\; .}$

In order to implement the Fuller (1998) method, we estimated the variance-covariance matrix of the control totals ${\widehat{V}}_C\mathrm{<mo\; stretchy="false">(</mo>}{\widehat{t}}_C\mathrm{<mo\; stretchy="false">)</mo>}$ using the National survey replicate weights. The spectral decomposition of this matrix resulted in a set of 144 eigenvectors $q_i$ and associated eigenvalues ${\lambda }_i,$ for $i\mathrm{<mo>=</mo>}\mathrm{1,}\dots \mathrm{,}144.$ Following Fuller (1998), we obtain a set of 144 vectors $v_i$ satisfying

$${\widehat{V}}_C\mathrm{<mo\; stretchy="false">(</mo>}{\widehat{t}}_C\mathrm{<mo\; stretchy="false">)</mo>}={\displaystyle \sum _{i=1}^{144}{v}_i^{\prime }{v}_i}\mathrm{,}$$

where $v_i\mathrm{<mo>=</mo>}\sqrt{{\lambda }_i}{q}_i,$ for $i\mathrm{<mo>=</mo>}\mathrm{1,}\dots \mathrm{,}144.$ Finally, the adjusted replicate controls are ${\widehat{t}}_C^{*\mathrm{<mo\; stretchy="false">(</mo>}r\mathrm{<mo\; stretchy="false">)</mo>}}\mathrm{<mo>=</mo>}{\widehat{t}}_C+\frac{\sqrt{160}}{2}{v}_r$ for $r\mathrm{<mo>=</mo>}\mathrm{1,}\dots \mathrm{,}144$ and ${\widehat{t}}_C^{*\mathrm{<mo\; stretchy="false">(</mo>}r\mathrm{<mo\; stretchy="false">)</mo>}}\mathrm{<mo>=</mo>}{\widehat{t}}_C$ for $r\mathrm{<mo>=</mo>}\mathrm{145,}\dots \mathrm{,}160.$ This points to a drawback of the Fuller (1998) method: while our approach perturbs the control totals of all 160 replicates, this method only perturbs a fraction of them in this case. In addition, 30 of the 144 eigenvalues were nearly zero, 18 of which less than zero due to floating point error. We truncated the 18 negative eigenvalues to zero, and left the rest unchanged. Hence, to the extent that not all replicates contribute to variance estimates for some survey estimates (e.g., domain totals), there is a risk that the sample-based calibration will be imperfectly reflected in the variance estimates. In general, we expect that a larger number of replicates will be perturbed using our approach, since the estimated variance-covariance matrix of the control totals can only be reliably estimated if its dimension is suitably smaller than $R_C.$

Tables 4.1 and 4.2 show the estimates and standard errors, respectively, for domains defined by residency and sex, before and after calibration. The first four rows contain the results for marginal totals for raking variables, which are exactly calibrated, while the last four are totals that correspond to the intersection of raking dimensions and are therefore not exactly calibrated.

Both surveys are representative of the same target population, but the estimates and associated standard errors differ, reflecting both sampling variability as well as different calibration approaches applied by the two survey organizations. As Table 4.1 confirms, after the 50-state survey is raked to the National survey, the estimated totals for domains defined as exact calibration domains indeed match exactly between both surveys. For the domains defined by the crosstabulation of residency and sex, the raked estimates for the 50-State survey are close but not identical to those of the National survey.

Table 4.2 shows the standard errors obtained by the two replication methods with adjusted control totals and by the naive method, which does not account for the randomness of the control totals. By construction, the proposed replication-based adjustment method and the Fuller (1998) method lead to identical variance estimates for domains that are used in the calibration. These variance estimates are also equal to those from the control survey in this case. This reflects the fact that the variance component corresponding to the first term in (2.5) is set to 0 for the control totals, while the variance component for the second term is exactly equal to the control survey variance estimate in the case of raking. Because that variance component is ignored in the naive method, the variance estimates are equal to zero. For the estimated totals for domains defined as the crosstabulation of residency and sex, the variance estimates of the two methods are not identical but close (within 8% of each other), reflecting the fact that both are consistent for the asymptotic variance (2.5). The variance estimates under the naive method are smaller than the variance estimates under the other two calibration methods, leading to an obviously incorrect result due to not accounting for the variance in the random control totals. For other variables, the variance is still expected to be underestimated under the naive method, due to the fact that the second term in the asymptotic variance (2.5) is ignored.

• Previous
• Article main page
• Next