Methodological problems raised by an international survey - The International Adult Literacy Survey - ARCHIVED

This paper discusses in detail issues dealing with the technical aspects of designing and conducting surveys. It is intended for an audience of survey methodologists.

The International Adult Literacy Survey (IALS), co-ordinated by Statistics Canada, was conducted in some 20 countries between 1994 and 2000. Based on the survey's findings, a wrap-up report containing a comparative analysis of reading skills in participating countries was published in 2000 through funding from the Organisation for Economic Co-operation and Development (OECD). Unfortunately, there are very serious methodological problems with this survey that make it unusable for comparative purposes.

This presentation discusses the survey's weaknesses and, more generally, the pitfalls of applying a universal measure of skills in countries with different cultures and different languages. Analyses of the survey results reveal the extent to which translation and wording altered the difficulty of the test questions. The range of scores on the various items among the participating countries confirms this linguistic bias. Moreover, respondents' attitudes (motivation, attentiveness, refusal, etc.) are not only culturally marked and tightly bound to the survey tradition in their own countries, but are also a determining factor in a survey intended to construct a measure of skills. Only highly detailed coding could separate the respondents' attitudes toward this long and difficult survey and differentiate between a deficiency of skills or a lack of motivation or attention.

This paper is based on an assessment of the survey within the European Community. Its findings have led to the publication of a series of articles and a book on the subject.