Conditional calibration and the sage statistician
Section 3. On the elusive goal of being calibrated and sage
Bayesians condition on what is observed, and so in principle, try to be
appropriate to the data at hand. True Bayesian
calibration, however, in the sense of creating interval estimates that have
accurate Bayesian coverage of the true posterior distribution no matter what “Truth” generated the observed data, is essentially
impossible in practice. This was illustrated to
me in fairly trivial examples, first in Rubin (1983) when
I was attempting to demonstrate the superiority of the Bayesian approach in the
context of survey inferences, then in Rosenbaum and Rubin
(1984), which documented the relevance of stopping rules on the Bayesian
validity of Bayesian inferences, unless all model and prior distributions were
correct, and more recently in Ferriss (Harvard PhD. Thesis,
2018), which considered the implications of re-randomization in experiments on
the Bayesian validity of Bayesian inference. But despite this inability to
approach the Bayesian ideal when there is the absence of knowledge of correct
models, a statistician can still seek to be calibrated, in some important sense,
and sage in the fiducial sense of avoiding conclusions that are contradicted by
the data set actually being analyzed. I refer to this as being “conditionally
calibrated” and explicate this surprisingly elusive idea here.
A personal aside relevant to this idea of
being conditionally calibrated: When I was visiting the University of California, Berkeley in the 1970’s and had a visitor’s office next to, the then retired, but still
intellectually vibrant and feisty, Jerzy Neyman, he clearly
expressed to me his view that such conditioning for statistical inference was
essentially impossible to define correctly, at least in the context of our 1970’s discussion of Fisher’s desiderata to condition on ancillary statistics
when drawing inferences.
Another relevant aside: my reading is that fundamentally, both Neyman and Fisher wanted, at least in their youths, to be effectively Bayesian in that they both sought a distribution for the estimand
conditional on the observed data, but took very different mathematical
approaches to finding that distribution, as discussed in (Rubin, 2016). Fisher (1956) was totally forthright about this fiducial
objective: “The Fiducial argument uses the observations to change the logical
status of the parameter [the unknown estimand] from one in which
nothing is known of it, and no probability statement can be made, to the status
of a random variable having a well-defined distribution”. Values of the
estimand with little support in this fiducial distribution, were those values
that were stochastically contradicted by the observed data, that is, if true,
they were unlikely to generate the observed data
a stochastic proof by contradiction. Despite the intuitive appeal
of this approach, mathematical foundations for it have not enjoyed universal
acceptance (e.g., Dempster, 1967; Martin and Liu, 2016).
Neyman was not direct as Fisher when
seeking a distribution for the estimand, but consider his original definition of “confidence intervals” (Neyman, 1934), which
was openly based on some Bayesian logic:
Description for Figure 3.1
Neyman’s
(1934, pages 589-590) definition of confidence intervals. Suppose we are
taking samples,
from some population
We are interested in a certain collective
character of this population, say
Denote by
a collective character of the sample
and suppose that we have been able to deduce
its frequency distribution, say
in repeated samples and that this is dependent
on the unknown collective character,
of the population
Denote
now by
the unknown probability distribution a priori
of
…[T]he
probability of our being wrong is less than or at most equal to
and this whatever the probability law a priori,
The
value of
chosen in a quite arbitrary manner, I propose
to call the “confidence coefficient”. If we choose, for instance,
and find for every possible
the intervals
having the properties defined, we could
roughly describe the position by saying that we have 99 per cent confidence in
the fact that
is contained between
and
…[I] call the intervals
the confidence intervals, corresponding to the
confidence coefficient
ISSN : 1492-0921
Editorial policy
Survey Methodology publishes articles dealing with various aspects of statistical development relevant to a statistical agency, such as design issues in the context of practical constraints, use of different data sources and collection techniques, total survey error, survey evaluation, research in survey methodology, time series analysis, seasonal adjustment, demographic studies, data integration, estimation and data analysis methods, and general survey systems development. The emphasis is placed on the development and evaluation of specific methodologies as applied to data collection or the data themselves. All papers will be refereed. However, the authors retain full responsibility for the contents of their papers and opinions expressed are not necessarily those of the Editorial Board or of Statistics Canada.
Submission of Manuscripts
Survey Methodology is published twice a year in electronic format. Authors are invited to submit their articles in English or French in electronic form, preferably in Word to the Editor, (statcan.smj-rte.statcan@canada.ca, Statistics Canada, 150 Tunney’s Pasture Driveway, Ottawa, Ontario, Canada, K1A 0T6). For formatting instructions, please see the guidelines provided in the journal and on the web site (www.statcan.gc.ca/SurveyMethodology).
Note of appreciation
Canada owes the success of its statistical system to a long-standing partnership between Statistics Canada, the citizens of Canada, its businesses, governments and other institutions. Accurate and timely statistical information could not be produced without their continued co-operation and goodwill.
Standards of service to the public
Statistics Canada is committed to serving its clients in a prompt, reliable and courteous manner. To this end, the Agency has developed standards of service which its employees observe in serving its clients.
Copyright
Published by authority of the Minister responsible for Statistics Canada.
© Her Majesty the Queen in Right of Canada as represented by the Minister of Industry, 2019
Use of this publication is governed by the Statistics Canada Open Licence Agreement.
Catalogue No. 12-001-X
Frequency: Semi-annual
Ottawa
