Question Evaluation Methods E-Book

Jennifer Madans

• Herausgeber: John Wiley & Sons
• Kategorie: Wissenschaft und neue Technologien
• Serie: Wiley Series in Survey Methodology
• Sprache: Englisch

Seitenzahl: 682

Veröffentlichungsjahr: 2011

Beschreibung

Insightful observations on common question evaluation methods and best practices for data collection in survey research

Featuring contributions from leading researchers and academicians in the field of survey research, Question Evaluation Methods: Contributing to the Science of Data Quality sheds light on question response error and introduces an interdisciplinary, cross-method approach that is essential for advancing knowledge about data quality and ensuring the credibility of conclusions drawn from surveys and censuses. Offering a variety of expert analyses of question evaluation methods, the book provides recommendations and best practices for researchers working with data in the health and social sciences.

Based on a workshop held at the National Center for Health Statistics (NCHS), this book presents and compares various question evaluation methods that are used in modern-day data collection and analysis. Each section includes an introduction to a method by a leading authority in the field, followed by responses from other experts that outline related strengths, weaknesses, and underlying assumptions. Topics covered include:

• Behavior coding
• Cognitive interviewing
• Item response theory
• Latent class analysis
• Split-sample experiments
• Multitrait-multimethod experiments
• Field-based data methods

A concluding discussion identifies common themes across the presented material and their relevance to the future of survey methods, data analysis, and the production of Federal statistics. Together, the methods presented in this book offer researchers various scientific approaches to evaluating survey quality to ensure that the responses to these questions result in reliable, high-quality data.

Question Evaluation Methods is a valuable supplement for courses on questionnaire design, survey methods, and evaluation methods at the upper-undergraduate and graduate levels. it also serves as a reference for government statisticians, survey methodologists, and researchers and practitioners who carry out survey research in the areas of the social and health sciences.

Leseprobe

Table of Contents

Cover

Wiley Series in Survey Methodology

Title page

Copyright page

CONTRIBUTORS

PREFACE

1 Introduction

PART I: Behavior Coding

2 Coding the Behavior of Interviewers and Respondents to Evaluate Survey Questions

2.1 INTRODUCTION

2.2 A BRIEF HISTORY

2.3 HOW BEHAVIOR CODING IS DONE

2.4 EVIDENCE FOR THE SIGNIFICANCE OF BEHAVIOR CODING RESULTS

2.5 STRENGTHS AND LIMITATIONS OF BEHAVIOR CODING

2.6 THE ROLE OF BEHAVIOR CODING IN QUESTION EVALUATION PROTOCOLS

2.7 CONCLUSION

3 Response 1 to Fowler’s Chapter: Coding the Behavior of Interviewers and Respondents to Evaluate Survey Questions

3.1 WHY DO WE STUDY INTERACTION IN THE SURVEY INTERVIEW?

3.2 QUESTIONS, BEHAVIOR, AND QUALITY OF MEASUREMENT: A CONCEPTUAL MODEL

3.3 CODING THE BEHAVIOR OF INTERVIEWERS AND RESPONDENTS

3.4 CONVERSATION ANALYTIC STUDIES OF INTERACTION IN THE INTERVIEW

3.5 CHARACTERISTICS OF SURVEY QUESTIONS

3.6 CONCLUSION

ACKNOWLEDGMENTS

4 Response 2 to Fowler’s Chapter: Coding the Behavior of Interviewers and Respondents to Evaluate Survey Questions

4.1 INTRODUCTION

4.2 IN RESPONSE TO FOWLER’S CHAPTER

4.3 THE USE OF BEHAVIOR CODING TO EVALUATE QUESTIONS IN MULTILINGUAL SURVEYS

4.4 WHAT TO DO?

4.5 CONCLUSION

PART II: Cognitive Interviewing

5 Cognitive Interviewing

5.1 INTRODUCTION

5.2 THEORETICAL UNDERPINNINGS, OBJECTIVES, AND PRACTICE

5.3 THE CASM MOVEMENT AND THE PRACTICE OF COGNITIVE TESTING

5.4 SHIFTING PARADIGMS: TOWARD A MORE INTEGRATIVE APPROACH TO QUESTION RESPONSE

5.5 ONTOLOGICAL AND EPISTEMOLOGICAL ADVANTAGES TO SURVEY RESEARCH

5.6 INTERPRETIVE QUALITY

5.7 CAPABILITY OF CAPTURING COMPLEXITY

5.8 FINDINGS ARE GROUNDED

5.9 ELEMENTS OF A CREDIBLE COGNITIVE INTERVIEWING STUDY

5.10 DELIBERATE METHOD OF ANALYSIS

5.11 TRANSPARENT PROCESS

5.12 CONCLUSION

6 Response 1 to Miller’s Chapter: Cognitive Interviewing

6.1 INTRODUCTION

6.2 ARE WE LIMITED TO “COGNITIVE PROCESSES”?

6.3 UNIVERSALITY OF RESPONSE PROCESSES

6.4 PURPOSE OF THE COGNITIVE INTERVIEW

6.5 SAMPLE SIZE

6.6 METHODOLOGICAL FOCUS OF COGNITIVE INTERVIEWS

6.7 RESPONSES TO SEVERAL OTHER POINTS IN THE CHAPTER

6.8 CONCLUSION

7 Response 2 to Miller’s Chapter: Cognitive Interviewing

7.1 I AM LARGELY SYMPATHETIC TO THE VIEW PROPOSED BY MILLER

7.2 NOT SUCH A PARADIGM SHIFT

7.3 SOME MISUNDERSTANDINGS, I THINK

7.4 POINTS OF DISAGREEMENT: REACTIVITY AND RELIABILITY

7.5 FINAL COMMENTS

PART III: Item Response Theory

8 Applying Item Response Theory for Questionnaire Evaluation

8.1 INTRODUCTION

8.2 THE IRT MODEL

8.3 IRT MODEL INFORMATION FUNCTION

8.4 IRT MODEL STANDARD ERROR OF MEASUREMENT (SEM) FUNCTION

8.5 FAMILY OF IRT MODELS

8.6 IRT MODEL ASSUMPTIONS

8.7 APPLICATIONS OF IRT MODELING FOR QUESTIONNAIRE EVALUATION AND ASSESSMENT

8.8 CONCLUSION

9 Response 1 to Reeve’s Chapter: Applying Item Response Theory for Questionnaire Evaluation

9.1 INTRODUCTION

9.2 CATEGORY RESPONSE CURVES (CRCS)

9.3 DIFFERENTIAL ITEM FUNCTIONING

9.4 INFORMATION

9.5 COMPUTERIZED-ADAPTIVE TESTING (CAT)

9.6 CHOICE OF IRT MODEL

9.7 PERSON FIT

9.8 UNIT OF ANALYSIS

9.9 CONCLUSION

ACKNOWLEDGEMENT

10 Response 2 to Reeve’s Chapter: Applying Item Response Theory for Questionnaire Evaluation

10.1 BACKGROUND

10.2 IRT: LIMITATIONS AND POTENTIAL USES IN SURVEY RESEARCH

10.3 AN EXAMPLE EVALUATING LATENT CLASS MODELS

10.4 CONCLUSION

PART IV: Latent Class Analysis

11 Some Issues in the Application of Latent Class Models for Questionnaire Design

11.1 INTRODUCTION

11.2 BASIC LC MODEL FOR THREE INDICATORS

11.3 SOME ISSUES IN ESTIMATION

11.4 LOCAL DEPENDENCE

11.5 CONCLUSION

12 Response 1 to Biemer and Berzofsky’s Chapter: Some Issues in the Application of Latent Class Models for Questionnaire Design

12.1 INTRODUCTION

12.2 PRACTICAL PROBLEMS WHEN APPLYING LCM

12.3 THE ROLE OF LCM IN THE QUESTIONNAIRE DEVELOPMENT CONTEXT

12.4 DIFFERENT PERSPECTIVES ON THE USE OF LCMS

12.5 GUIDELINES OR STANDARDS WHEN APPLIEDAND USED FOR Q-BANK

13  Response 2 to Biemer and Berzofsky’s Chapter: Some Issues in the Application of Latent Class Models for Questionnaire Design

13.1 INTRODUCTION

13.2 LATENT AND MANIFEST ASPECTS OF QUESTIONNAIRE DESIGN, IMPLEMENTATION, AND TESTING

13.3 MEASUREMENT EVENTS AND MEASUREMENT OCCASIONS

13.4 MEANINGS AND CONTEXTS

13.5 THE RELEVANCE OF LCMs FOR QUESTIONNAIRE DESIGN AND TESTING

13.6 LCM ADVANTAGES

13.7 CONSTRAINTS ON LCM AS AN ASSESSMENT TOOL

13.8 FUTURE OPPORTUNITIES

13.9 CONCLUSION

PART V: Split-Sample Experiments

14  Experiments for Evaluating Survey Questions

14.1 INTRODUCTION

14.2 THE LOGIC OF EXPERIMENTATION

14.3 HOW EXPERIMENTS CAN BE VALUABLE FOR EVALUATING QUESTIONS

14.4 HOW CAN EXPERIMENTS BE MISUSED OR CONDUCTED INCORRECTLY

14.5 COORDINATING EXPERIMENTATION WITH OTHER QUESTION EVALUATION METHODS

14.6 METHODOLOGICAL CRITERIA FOR INCLUDING RESULTS OF EXPERIMENTS IN Q-BANK

14.7 CONCLUSION

15  Response 1 to Krosnick’s Chapter: Experiments for Evaluating Survey Questions

15.1 INTRODUCTION

15.2 EXPERIMENTS FOR PRETESTING

15.3 USING VPA IN A COGNITIVE INTERVIEW PRETEST EXPERIMENTAL DESIGN

15.4 AN OVERVIEW OF VPA

15.5 THE STRUCTURE OF A COGNITIVE INTERVIEW PRETEST EXPERIMENT

15.6 SURVEY QUESTION TASK ANALYSIS

15.7 EXAMPLE APPLICATION: PUBLIC HOUSING RENT SYSTEMS SURVEY

15.8 DISCUSSION

16  Response 2 to Krosnick’s Chapter: Experiments for Evaluating Survey Questions

16.1 INTRODUCTION

16.2 FACTUAL VERSUS SUBJECTIVE MEASUREMENTS

16.3 INTEGRATION OF EXPERIMENTS WITH OTHER METHODOLOGIES

PART VI: Multitrait-Multimethod Experiments

17 Evaluating the Reliability and Validity of Survey Interview Data Using the MTMM Approach

17.1 INTRODUCTION

17.2 BACKGROUND

17.3 THE MTMM MATRIX5

17.4 THE MEANING OF CONSTRUCT VALIDITY

17.5 COMPONENTS OF VARIANCE

17.6 RECENT RESEARCH USING THE MTMM APPROACH

17.7 A CRITIQUE OF THE MTMM DESIGN

17.8 THE ROLE OF MEMORY

17.9 ARE THERE ALTERNATIVE APPROACHES?

17.10 CONCLUSION

18 Response to Alwin’s Chapter: Evaluating the Reliability and Validity of Survey Interview Data Using the MTMM Approach

18.1 INTRODUCTION

18.2 Q: A POINT QUALITY INDICATOR EXAMPLE

18.3 PROCESS QUALITY VERSUS POINT QUALITY

18.4 INSTRUMENT TECHNICAL DESIGN

18.5 REVISITING Q

PART VII: Field-Based Data Methods

19 Using Field Tests to Evaluate Federal Statistical Survey Questionnaires

19.1 INTRODUCTION

19.2 WHAT ARE FIELD TESTS?

19.3 WHY SHOULD FIELD TESTS BE CONDUCTED?

19.4 THE FEDERAL CONTEXT: STANDARDS AND REVIEW PROCESS FOR FEDERAL SURVEYS

19.5 SOME USES OF FIELD TESTS BY FEDERAL AGENCIES

19.6 WHAT ARE THE WEAKNESSES OF FIELD TESTS?

19.7 PARADATA ANALYSIS AS AN ALTERNATIVE OR COMPLEMENT TO FIELD TESTS

19.8 CONCLUSION

INDEX

Copyright © 2011 by John Wiley & Sons, Inc. All rights reserved.

Published by John Wiley & Sons, Inc., Hoboken, New Jersey.

Published simultaneously in Canada.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, [phone], fax [phone], or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, [phone], fax [phone], or online at http://www.wiley.com/go/permissions.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at [phone], outside the United States at [phone] or fax [phone].

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com.

Library of Congress Cataloging-in-Publication Data

Question evaluation methods : contributing to the science of data quality / Jennifer Madans ... [et al.].

p. cm. – (Wiley series in survey methodology)

 Includes bibliographical references and index.

 ISBN 978-0-470-76948-5 (pbk.)

 1. Questionnaires–Design. 2. Social surveys–Methodology. 3. Social surveys–Evaluation. 4. Sampling (Statistics)–Evaluation. I. Madans, Jennifer H.

 HM537.Q45 2011

 301.072'3–dc22

2010048273

oBook ISBN: 9781118037003

ePDF ISBN: 9781118036983

ePub ISBN: 9781118036990

CONTRIBUTORS

DUANE F. ALWIN, Pennsylvania State University, University Park, Pennsylvania

MARCUS BERZOFSKY, RTI International, Research Triangle Park, North Carolina

PAUL P. BIEMER, RTI International, Research Triangle Park, and University of North Carolina at Chapel Hill, Chapel Hill, North Carolina

JOHNNY BLAIR, Abt Associates, Washington, District of Columbia

FREDERICK G. CONRAD, University of Michigan, Ann Arbor, Michigan

JAMES M. DAHLHAMER, National Center for Health Statistics, Hyattsville, Maryland

THERESA DEMAIO, U.S. Bureau of the Census, Washington, District of Columbia

JENNIFER DYKEMA, University of Wisconsin–Madison, Madison, Wisconsin

JENNIFER EDGAR, Bureau of Labor Statistics, Washington, District of Columbia

FLOYD J. FOWLER, JR., University of Massachusetts, Boston, Massachusetts

JANET A. HARKNESS, University of Nebraska–Lincoln, Lincoln, Nebraska

BRIAN A. HARRIS-KOJETIN, U.S. Office of Management and Budget, Washington, District of Columbia

RON D. HAYS, University of California, Los Angeles, California

TIMOTHY P. JOHNSON, University of Illinois at Chicago, Chicago, Illinois

FRAUKE KREUTER, University of Maryland, College Park, Maryland

JON A. KROSNICK, Stanford University, Palo Alto, California

JENNIFER MADANS, National Center for Health Statistics, Hyattsville, Maryland

AARON MAITLAND, National Center for Health Statistics, Hyattsville, Maryland

BRIAN MEEKINS, Bureau of Labor Statistics, Washington, District of Columbia

KRISTEN MILLER, National Center for Health Statistics, Hyattsville, Maryland

PETER PH. MOHLER, Universität Mannheim, Mannheim, Germany

BRYCE B. REEVE, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina

NORA CATE SCHAEFFER, University of Wisconsin–Madison, Madison, Wisconsin

ALISÚ SCHOUA-GLUSBERG, Research Support Services, Chicago, Illinois

CLYDE TUCKER, Bureau of Labor Statistics, Washington, District of Columbia

GORDON WILLIS, National Cancer Institute, Bethesda, Maryland

STEPHANIE WILLSON, National Center for Health Statistics, Hyattsville, Maryland

PREFACE

The goal of this book is to present an interdisciplinary examination of existing methods of question evaluation. Researchers with various backgrounds are faced with the task of designing and evaluating survey questions. While there are several methods available to meet this objective, the field of question evaluation has been hindered by the lack of consensus on the use of terminology and by the technical difficulty of many of the methods available. We have made a concerted effort to start bridging gaps in the knowledge required to utilize the diversity of methods available. Hence, a major challenge for this book was presenting the material in a format that reaches a wide audience.

The idea for the book grew out of discussion among researchers in federal statistical agencies about the need to bring together leading experts to discuss the strengths and weaknesses of different question evaluation methods. During the fall of 2008, Jennifer Madans, Kristen Miller, and Aaron Maitland from the National Center for Health Statistics (NCHS) and Gordon Willis from the National Cancer Institute (NCI) began contacting experts about their willingness to present papers on particular evaluation methods at a workshop. The response was overwhelmingly in favor of such an event, and several key figures in the field elected to participate.

The Workshop on Question Evaluation Methods (http://www.cdc.gov/qbank) was held at NCHS in Hyattsville, Maryland, October 21–23, 2009. Financial support was provided by NCHS and NCI. The workshop agenda was designed to fulfill the objectives of providing a review of the background of each question evaluation method and of identifying and discussing current issues surrounding each method. Each session included three leading experts on a specific method. The initial presenter wrote a primary paper that outlined the background and current issues on the particular method. This was followed by two papers written in response to the primary paper. The timeline for the papers was developed so that the responding authors had a few months to write a paper in response to the primary papers. Time was set aside for discussion between the presenters and audience members following the presentation of all three papers for each method.

Presentations were made on seven question evaluation methods. This organization of the papers at the workshop serves as an outline for the book. This book begins with a brief introduction, and each of the sessions at the workshop is a section in this book. The sections are as follows:

Section I: Behavior Coding

Section II: Cognitive Interviewing

Section III: Item Response Theory

Section IV: Latent Class Analysis

Section V: Split-Sample Experiments

Section VI: Multitrait-Multimethod Experiments

Section VII: Field-Based Data Methods

Following the workshop, the organizers of the workshop constructed a book manuscript out of the papers. Each organizer served as an editor for some of the sections in the manuscript. Jennifer Madans edited Section II; Kristen Miller edited Section VII; Aaron Maitland edited Sections I, III, and IV; and Gordon Willis edited Sections V and VI.

The editors would like to thank all of the workshop participants for their contributions at the workshop and for including their work in this book. Their expertise and willingness to share that expertise made the workshop both exciting and enjoyable. We also thank Ken Waters of the U.S. Department of Agriculture’s Animal and Plant Health Inspection Service for performing the thankless role of moderator. He carried out his tasks in a most constructive way. We are most grateful to the National Center for Health Statistics and the National Cancer Institute for providing the funding for the meeting. Finally, we thank Jacqueline Palmieri and Steven Quigley at Wiley for their guidance throughout the production process.

JENNIFER MADANS

KRISTEN MILLER

AARON MAITLAND

GORDON WILLIS

1

Introduction

JENNIFER MADANS, KRISTEN MILLER, and AARON MAITLAND

National Center for Health Statistics

GORDON WILLIS

National Cancer Institute

If data are to be used to inform the development and evaluation of policies and programs, they must be viewed as credible, unbiased, and reliable. Legislative frameworks that protect the independence of the federal statistical system and codes of conduct that address the ethical aspects of data collection are crucial for maintaining confidence in the resulting information. Equally important, however, is the ability to demonstrate the quality of the data, and this requires that standards and evaluation criteria be accessible to and endorsed by data producers and users. It is also necessary that the results of quality evaluations based on these standards and criteria be made public. Evaluation results not only provide the user with the critical information needed to determine whether a data source is appropriate for a given objective but can also be used to improve collection methods in general and in specific areas. This will only happen if there is agreement in how information on data quality is obtained and presented. In November 2009, a workshop on Question Evaluation Methods (QEM) was held at the National Center for Health Statistics in Hyattsville, Maryland. The objective of the workshop was to advance the development and use of methods to evaluate questions used on surveys and censuses. This book contains the papers presented at that workshop.

To evaluate data quality it is necessary to address the design of the sample, including how that design was carried out, as well as the measurement characteristics of the estimates derived from the data. Quality indicators related to the sample are well developed and accepted. There are also best practices for reporting these indicators. In the case of surveys based on probability samples, the response rate is the most accepted and reported quality indicator. While recent research has questioned the overreliance on the response rate as an indicator of sample bias, the science base for evaluating sample quality is well developed and, for the most part, information on response rates is routinely provided according to agreed-upon methods. The same cannot be said for the quality of the survey content.

Content is generally evaluated according to the reliability and validity of the measures derived from the data. Quality standards for reliability, while generally available, are not often implemented due to the cost of conducting the necessary data collection. While there has been considerable conceptual work regarding the measurement of validity, translating the concepts into measurable standards has been challenging. There is a need for a critical and creative approach to evaluating the quality of the questions used on surveys and censuses. The survey research community has been developing new methodologies to address this need for question evaluation, and the QEM Workshop showcased this work. Since each evaluation method addresses a different aspect of quality, the methods should be used together. Some methods are good at determining that a problem exists while others are better at determining what the problem actually is, and others contribute by addressing what the impact of the problem will be on survey estimates and the interpretation of those estimates. Important synergies can be obtained if evaluations are planned to include more than one method and if each method builds on the strength of the others. To fully evaluate question quality, it will be necessary to incorporate as many of these methods as possible into evaluation plans. Quality standards addressing how the method should be conducted and how the results are to be reported will need to be developed for each method. This will require careful planning, and commitments must be made at the onset of data collection projects with appropriate funding made available. Evaluations cannot be an afterthought but must be an integral part of data collections.

The most direct use of the results of question evaluations is to improve a targeted data collection. The results can and should be included in the documentation for that data collection so that users will have a better understanding of the magnitude and type of measurement error characterizing the resulting data. This information is needed to determine if a data set is fit for an analytic purpose and to inform the interpretation of results of analysis based on the data. A less common but equally if not more important use is to contribute to the body of knowledge about the specific topic that the question deals with as well as more general guidelines for question development. The results of question evaluations are not only the end product of the questionnaire design stage but should also be considered as data which can be analyzed to address generic issues of question design. For this to be the case, the results need to be made available for analysis to the wider research community, and this requires that there be a place where the results can be easily accessed.

A mechanism is being developed to make question test results available to the wider research community. Q-Bank is an online database that houses science-based reports that evaluate survey questions. Question evaluation reports can be accessed by searching for specific questions that have been evaluated. They can also be accessed by searching question topic, key word, or survey title. (For more information, see http://www.cdc.gov/qbank.) Q-Bank was first developed to provide a mechanism for sharing cognitive test results. Historically, cognitive test findings have not been accessible outside of the organization sponsoring the test and sometimes not even shared within the organization. This resulted in lost knowledge and wasted resources as the same questions were tested repeatedly as if no tests had been done. Lack of access to test results also contributed to a lack of transparency and accountability in data quality evaluations. Q-Bank is not a database of good questions but is a database of test results that empowers data users to be able to evaluate the quality of the information for their own uses. Having the results of evaluations in a central repository can also improve the quality of the evaluations themselves, resulting in the development of a true science of question evaluation. The plan is for Q-Bank to expand beyond cognitive test results to include the results of all question evaluation methods addressed in the workshop.

The QEM workshop provided a forum for comparing question evaluation methods, including behavior coding, cognitive interviewing, field-based data studies, item response theory modeling, latent class analysis, and split-sample experiments. The organizers wanted to engage in an interdisciplinary and cross-method discussion of each method, focusing specifically on each method’s strengths, weaknesses, and underlying assumptions. A primary paper followed by two response papers outlined key aspects of a method. This was followed by an in-depth discussion among workgroup participants. Because the primary focus for the workgroup was to actively compare methods, each primary author was asked to address the following topics:

Description of the methodHow it is generally used and in what circumstances it is selectedThe types of data it produces and how these are analyzedHow findings are documentedThe theoretical or epistemological assumptions underlying use of the methodThe type of knowledge or insight that the method can give regarding questionnaire functioningHow problems in questions or sources of response error are characterizedWays in which the method might be misused or incorrectly conductedThe capacity of the method for use in comparative studies, such as multicultural or cross-national evaluationsHow other methods best work in tandem with this method or within a mixed-method designRecommendations: Standards that should set as criteria for inclusion of results of this method within Q-Bank

Finally, closing remarks, which were presented by Norman Bradburn, Jennifer Madans, and Robert Groves, reflected on common themes across the papers and the ensuing discussions, and the relevance to federal statistics.

One of the goals for the workshop was to support and acknowledge those doing question evaluation and developing evaluation methodology. Encouragement for this work needs to come not only from the survey community but also from data users. Funders, sponsors, and data users should require that information on question quality (or lack thereof) be made public and that question evaluation be incorporated into the design of any data collection. Data producers need to institutionalize question evaluation and adopt and endorse agreed-upon standards. Data producers need to hold themselves and their peers to these standards as is done with standards for sample design and quality evaluation. Workshops like the QEM provide important venues for sharing information and supporting the importance of question evaluation. More opportunities like this are needed. This volume allows the work presented at the Workshop to be shared with a much wider audience—a key requirement if the field is to grow. Other avenues for publishing results of evaluations and of the development of evaluation methods need to be developed and supported.

PART I: Behavior Coding

2

Coding the Behavior of Interviewers and Respondents to Evaluate Survey Questions

FLOYD J. FOWLER, JR.

University of Massachusetts

2.1 INTRODUCTION

Social surveys rely on respondents’ answers to questions as measures of constructs. Whether the target construct is an objective fact, such as age or what someone has done, or a subjective state, such as a mood or an opinion, the goal of the survey methodologist is to maximize the relationship between the answers people give and the “true value” of the construct that is to be measured.

When the survey process involves an interviewer and the process goes in the ideal way, the interviewer first asks the respondent a question exactly as written (so that each respondent is answering the same question). Next, the respondent understands the question in the way the researcher intended. Then the respondent searches his or her memory for the information needed to recall or construct an answer to the question. Finally, the respondent provides an answer in the particular form that the question requires.

Of course, the question-and-answer process does not always go so smoothly. The interviewer may not read the question as written, or the respondent may not understand the question as intended. Additionally, the respondent may not have the information needed to answer the question. The respondent may also be unclear about the form in which to put the answer, or may not be able to fit the answer into the form that the question requires.

In short, the use of behavior coding to evaluate questions rests on three key premises:

1. Deviations from the ideal question-and-answer process pose a threat to how well answers to questions measure target constructs.

2. The way a question is structured or worded can have a direct effect on how closely the question-and-answer process approximates the ideal.

3. The presence of these problems can be observed or inferred by systematically reviewing the behavior of interviewers and respondents.

Coding interviewer and respondent behavior during survey interviews is now a fairly widespread approach to evaluating survey questions. In this chapter, I review the history of behavior coding, describe the way it is done, summarize some of the evidence for its value, and try to describe the place of behavior coding in the context of alternative approaches to evaluating questions.

2.2 A BRIEF HISTORY

Observing and coding behavior has long been part of the social science study of interactions. Early efforts looked at teacher–pupil, therapist–patient, and (perhaps the most developed and widely used) small group interactions (Bales, 1951). However, the first use of the technique to specifically study survey interviews was probably a series of studies led by Charles Cannell (Cannell et al., 1968).

Cannell was studying the sources of error in reporting in the Health Interview Survey, an ongoing survey of health conducted by the National Center for Health Statistics. He had documented that some respondents were consistently worse reporters than others (Cannell and Fowler, 1965). He had also shown that interviewers played a role in the level of motivation exhibited by the respondents (Cannell and Fowler, 1964). He wanted to find out if he could observe which problems the respondents were having and if he could figure out what the successful interviewers were doing to motivate their respondents to be good reporters. There were no real models to follow, so Cannell created a system de novo using a combination of ratings and specific behavior codes. Using a strategy of sampling questions as his unit of observation, he had observers code specific behaviors (i.e., was the question read exactly as worded or did the respondent ask for clarification of the question) for some questions. For others, he had observers rate less specific aspects of what was happening, such as whether or not the respondent appeared anxious or bored.