My chancy life as a Statistician
Section 3. Life in USA: 1958-68
Undoubtedly, ISU was among the best (if not the best) applied statistics departments at that time. (I believe it still is.) I even had the chance to take the last course on statistical methods with George Snedecor before he retired. He was the founder of the Statistics Department at ISU and his close association with R.A. Fisher led to the well-known Snedecor’s F and also Fisher going to ISU as a visiting professor. It was most rewarding to learn from great statisticians like Hartley and Kempthorne at ISU and also from others who visited ISU regularly. Professor Hartley was my mentor and Ph.D. supervisor and I learnt from him that the development of statistical theory should be motivated by practical applications. I took economics as a minor in my Ph.D. program and I was fortunate to work with Gerhard Tintner who was a pioneer in Econometrics and one of the inventors of the Variate Difference Method for finding the order of difference that makes a time series stationary. I even wrote two papers and a small monograph with him on this topic. For several years I tried to keep up with the developments in Econometrics.
I stayed at ISU for 5 years, three years as a student and two years as Assistant Professor, before returning to India in 1963 for family reasons. This period was most exciting and professionally rewarding. At that time unequal probability sampling without replacement was a “hot” topic and people were looking for practical procedures. Hartley and I published a paper on this topic in the Annals of Statistics (1962) developing an asymptotic theory for randomized probability proportional to size (PPS) systematic sampling (Hartley and Rao, 1962) After finishing my Ph.D. in 1961, I published a paper with Hartley and W.G. Cochran, in the Journal of the Royal Statistical Society, Series B, 1962, on a very simple procedure of unequal probability sampling without replacement that has many desirable properties (Rao, Hartley and Cochran, 1962). This method is now known as the RHC method and many papers on this method have appeared since then. Both the PPS systematic sampling method and the RHC method have been used in the Canadian Labour Force Survey for the past 25 years or so. Professor Arijit Chaudhuri of the Indian Statistical Institute has used the RHC method extensively for large-scale sample surveys in India. I also wrote a paper in the Journal of the American Statistical Association on composite estimation for repeated surveys with my Canadian friend, Jack Graham, who was also a student at ISU at that time (Rao and Graham, 1964). Jack became my colleague after I joined Carleton University in Ottawa in 1973. More recently, I got back to composite estimation in the context of the Canadian Labour Force Survey and developed a new method in association with Wayne Fuller and Avi Singh, that is currently being used in Canada (Fuller and Rao, 2001 and Singh, Kennedy and Wu, 2001). I shared an office with Wayne Fuller at ISU and he has been a close friend for the past 55 years.
I worked as a sample survey expert at the National Council of Applied Economic Research in New Delhi for one year after I returned to India. During my stay there I was involved in the development of the design and analysis of an All India Consumer Expenditure Survey. But I was very frustrated because there were no facilities there for research. I returned to United States in August 1964 and worked for one year in Dallas in a research group headed by D.B. Owen before joining Hartley at Texas A&M University. (Hartley moved to Texas A&M in 1963 to create an Institute of Statistics there.) My stay at Texas A&M was also most rewarding and professionally exciting. I worked closely with Hartley and also supervised Ph.D. students. I was promoted to Full Professor rank in 1967 and things were going great. My son, Sunil, was born in April 1967 and we were well settled. But I had to return to India in June 1968 due to unexpected family problems. I took leave from Texas A&M and joined the Indian Statistical Institute (ISI) in Calcutta as Visiting Professor. (I might mention here that my son Sunil is currently Director of Biostatistics Division and Interim Chair of the Department of Public Health Sciences at the University of Miami. He was elected ASA Fellow in 2011 and we two belong to the very small group of father-son ASA Fellows!)
I would like to briefly mention four significant contributions I made during my stay at Texas A&M. In my Biometrika 1967 paper with Hartley, we gave a matrix formulation of general ANOVA mixed models that was instrumental to the derivation of maximum likelihood (ML) estimators of both fixed effects and variance components (Hartley and Rao, 1967), We also developed an EM algorithm in this paper but did not pursue it further due to computational limitations at that time. (EM algorithm became popular after the appearance of Dempster, Laird and Rubin (1977)). Patterson and Thompson (1971) modified our ML method and developed restricted maximum likelihood (REML) estimation. Many extensions and refinements have been made over the past 40 years, and several software packages implemented those methods. An excellent review paper by Harville (1977) contributed to the extensive use of those methods. I also worked with Hartley on variance estimation when only one unit is sampled from each stratum (Hartley, Rao and Kiefer, 1969). In this case, standard design-based methods are not applicable and it is necessary to resort to models. We used a linear regression model with unequal error variances and expressed the variance of the stratified mean as a linear combination of the error variances. We then developed a new method of estimating the error variances that in turn led to a new variance estimator for the stratified mean. We submitted this paper for publication in 1968 before I left for India. I gave a seminar talk at ISI on this work. After my talk, Professor C.R. Rao felt that he could establish some optimality properties for our method. This led to C.R. Rao’s well-known MINQUE method (Rao, 1970), and Professor Rao notes “The motivation for writing this article is a recent contribution by Hartley, Rao and Kiefer (1969) who obtained unbiased estimator when all the variances are unequal …” (page 161).
In the 1960’s, V.P. Godambe was giving talks at various professional meetings on his important contributions to survey sampling inference; in particular, on the non-existence of a best estimator in a general class of linear unbiased estimator of a total and on the flat likelihood caused by the label property of a finite population. Those negative results are indeed fundamental, but Hartley and I felt that some of the alternative criteria proposed for the choice of an estimator, such as admissibility and hyper-admissibility for any sampling design, are unsatisfactory. In our Biometrika 1968 paper we suggested that some aspects of the sample data, depending on the situation at hand, need to be ignored to arrive at an informative likelihood (Hartley and Rao, 1968). We proposed such a non-parametric likelihood that is now called Empirical Likelihood (Owen, 1988). We also showed how to incorporate known population totals of auxiliary variables, and showed that the empirical likelihood (EL) estimator of a total is close to a regression estimator. I gave several lectures on the foundations of inference in survey sampling at ISI and Professor C.R. Rao wrote a nice article afterwards (Rao, 1970) that seems to agree with our approach: “In situations like the one we are considering where the full likelihood does not satisfy our purpose, we may have to depend on a statistic which for every observed value supplies information (however poor it may be) on parameters of interest.” Our Biometrika 1968 paper also contained a short section on Bayesian inference for the mean obtained by combining our likelihood with a diffuse conjugate prior. Ericson (1969) combined Godambe’s flat likelihood with an informative prior to produce informative posterior inferences on the mean. Our results are algebraically identical to Ericson’s, but fundamentally different in the sense that our inferences depend on the probability distribution induced by the survey design, unlike Ericson’s results.
While I was working on my Ph.D. thesis at ISI, I analyzed some farm survey data where the farms were selected with probabilities proportional to farm sizes. I found that some variables of interest, in particular poultry size, was unrelated to farm size and that the use of the widely used Horvitz-Thompson (HT) unbiased estimator in such cases would lead to very large variances. I therefore proposed an alternative estimator that ignores the survey weights but uses the population structure (Rao, 1966). I provided both theoretical and empirical justifications for preferring such an estimator. My result essentially casts doubt on the usefulness of criteria that advocate the HT estimator for ANY design and ANY characteristic. Later, D. Basu used an amusing circus elephant example to demonstrate that the HT estimator leads to absurd results if the sizes are unrelated to the values of interest (Basu, 1971).
- Date modified: