Conditional calibration and the sage statistician
Section 6. The conditionally calibrated statistician’s evaluation of procedure $P$

Table of contents

With the basic definitions of Sections 4 and 5 for $C_{k}$ (the local calibration of $P$ for $Q_{k}$ under $T_{k})$ and the $M_{k}^{*}$ ( the matching ability of $T_{k}$ for $Y^{*})$ now established, we are prepared to consider the concept of Conditional Calibration (CC) and to make the connection to being a sage statistician after having observed $Y^{*} .$

Conditional Calibration starts at the same place as Neymanian unconditional calibration, but is sensitive to the Bayesian and Fiducial arguments by discounting results from possible truths whose drawn data sets $Y_{j k}$ are not close to $Y^{*}$ as assessed by their match rates $M_{k}^{*} .$ The point of doing this is: Why should we care about calibration for a priori possible Truths that could not have generated the observed $Y^{*},$ i.e., truths that are a posteriori implausible? But this CC perspective does not go to the full Bayesian extreme, which, first, ignores all aspects of the simulation except the Truths that generated data sets exactly matching the observed data set, and second, explicitly uses the often weakly justified prior distribution, $W_{k},$ to weight the local matching rates.

Hence, we are left summarizing our simulation, which evaluates a procedure $P$ for inference about the estimand $Q$ from observed data $Y^{*},$ using a two dimensional criterion: The local calibration of $P$ for $Q_{k}$ under truth $T_{k},$ that is $C_{k};$ and the local match rate for Truth $T_{k},$ the proportion of generated $Y_{j k}$ under truth $T_{k},$ that are accepted as matching $Y^{*},$ $M_{k}^{*} =$ the fraction of $Y_{j k}$ that are considered equal to $Y^{*} .$ Those possible truths with $M_{k}^{*}$ near one are clearly more relevant to the situation with observed data $Y^{*}$ than those truths with values of $M_{k}^{*}$ near zero.

Thus procedure $P$ applied to each possible truth with observed data $Y^{*}$ can be displayed as $K$ points in a two-dimensional graph where the horizontal axis is the average match to $Y^{*}$ of the data sets generated by truth $T_{k},$ $M_{k}^{*},$ and the vertical axis is the local calibration of $P$ for the data sets generated by Truth $T_{k},$ $C_{k} .$ We call this the “conditional calibration plot” and it is illustrated and discussed in the Section 7. Of major relevance to drawing sage inferences for possible truths from observed data $Y^{*},$ many different procedures can be displayed on the same conditional calibration plot for a fixed data set $Y^{*}$ and a fixed set of possible truths.

ISSN : 1492-0921

Editorial policy

Survey Methodology publishes articles dealing with various aspects of statistical development relevant to a statistical agency, such as design issues in the context of practical constraints, use of different data sources and collection techniques, total survey error, survey evaluation, research in survey methodology, time series analysis, seasonal adjustment, demographic studies, data integration, estimation and data analysis methods, and general survey systems development. The emphasis is placed on the development and evaluation of specific methodologies as applied to data collection or the data themselves. All papers will be refereed. However, the authors retain full responsibility for the contents of their papers and opinions expressed are not necessarily those of the Editorial Board or of Statistics Canada.

Submission of Manuscripts

Survey Methodology is published twice a year in electronic format. Authors are invited to submit their articles in English or French in electronic form, preferably in Word to the Editor, (statcan.smj-rte.statcan@canada.ca, Statistics Canada, 150 Tunney’s Pasture Driveway, Ottawa, Ontario, Canada, K1A 0T6). For formatting instructions, please see the guidelines provided in the journal and on the web site (www.statcan.gc.ca/SurveyMethodology).

Note of appreciation

Canada owes the success of its statistical system to a long-standing partnership between Statistics Canada, the citizens of Canada, its businesses, governments and other institutions. Accurate and timely statistical information could not be produced without their continued co-operation and goodwill.

Standards of service to the public

Statistics Canada is committed to serving its clients in a prompt, reliable and courteous manner. To this end, the Agency has developed standards of service which its employees observe in serving its clients.

Copyright

Published by authority of the Minister responsible for Statistics Canada.

Use of this publication is governed by the Statistics Canada Open Licence Agreement.

Catalogue No. 12-001-X

Frequency: Semi-annual

Ottawa

Date modified:: 2019-07-04

Language selection

Search and menus

Search

Conditional calibration and the sage statistician
Section 6. The conditionally calibrated statistician’s evaluation of procedure $P$