Cognitive Interviewing Methodology E-Book

• Herausgeber: John Wiley & Sons
• Kategorie: Wissenschaft und neue Technologien
• Serie: Wiley Series in Survey Methodology
• Sprache: Englisch

Seitenzahl: 363

Veröffentlichungsjahr: 2014

Beschreibung

AN INTERDISCIPLINARY PERSPECTIVE TO THE EVOLUTION OF THEORY AND METHODOLOGY WITHIN COGNITIVE INTERVIEW PROCESSES

Providing a comprehensive approach to cognitive interviewing in the field of survey methodology, Cognitive Interviewing Methodology delivers a clear guide that draws upon modern, cutting-edge research from a variety of fields.

Each chapter begins by summarizing the prevailing paradigms that currently dominate the field of cognitive interviewing. Then underlying theoretical foundations are presented, which supplies readers with the necessary background to understand newly-evolving techniques in the field. The theories lead into developed and practiced methods by leading practitioners, researchers, and/or academics. Finally, the edited guide lays out the limitations of cognitive interviewing studies and explores the benefits of cognitive interviewing with other methodological approaches. With a primary focus on question evaluation, Cognitive Interviewing Methodology also includes:

• Step-by-step procedures for conducting cognitive interviewing studies, which includes the various aspects of data collection, questionnaire design, and data interpretation
• Newly developed tools to benefit cognitive interviewing studies as well as the field of question evaluation, such as Q-Notes, a data entry and analysis software application, and Q-Bank, an online resource that houses question evaluation studies
• A unique method for questionnaire designers, survey managers, and data users to analyze, present, and document survey data results from a cognitive interviewing study

An excellent reference for survey researchers and practitioners in the social sciences who utilize cognitive interviewing techniques in their everyday work, Cognitive Interviewing Methodology is also a useful supplement for courses on survey methods at the upper-undergraduate and graduate-level.

Leseprobe

WILEY SERIES IN SURVEY METHODOLOGY

Established in Part by WALTER A. SHEWHART AND SAMUEL S. WILKS

Editors: Mick P. Couper, Graham Kalton, J. N. K. Rao, Norbert Schwarz,

Christopher Skinner

Editor Emeritus: Robert M. Groves

The Wiley Series in Survey Methodology covers topics of current research and practical interests in survey methodology and sampling. While the emphasis is on application, theoretical discussion is encouraged when it supports a broader understanding of the subject matter.

The authors are leading academics and researchers in survey methodology and sampling. The readership includes professionals in, and students of, the fields of applied statistics, biostatistics, public policy, and government and corporate enterprises.

ALWIN Margins of Error: A Study of Reliability in Survey Measurement

BETHLEHEM Applied Survey Methods: A Statistical Perspective

BETHLEHEM, COBBEN, and SCHOUTEN Handbook of Nonresponse in Household Surveys

BIEMER Latent Class Analysis of Survey Error

*

BIEMER, GROVES, LYBERG, MATHIOWETZ, and SUDMAN Measurement Errors in Surveys

BIEMER and LYBERG Introduction to Survey Quality

BIEMER Latent Class Analysis of Survey Error

BRADBURN, SUDMAN, and WANSINK Asking Questions: The Definitive Guide to Questionnaire Design---For Market Research, Political Polls, and Social Health Questionnaires,

Revised Edition

BRAVERMAN and SLATER Advances in Survey Research: New Directions for Evaluation, No. 70

CALLEGARO, BAKER, BETHLEHEM, GÖRITZ, KROSNICK, and LAVRAKAS (editors) Online Panel Research: A Data Quality Perspective

CHAMBERS and SKINNER (editors) Analysis of Survey Data

COCHRAN Sampling Techniques,

Third Edition

CONRAD and SCHOBER Envisioning the Survey Interview of the Future

COUPER, BAKER, BETHLEHEM, CLARK, MARTIN, NICHOLLS, and O'REILLY (editors) Computer Assisted Survey Information Collection

COX, BINDER, CHINNAPPA, CHRISTIANSON, COLLEDGE, and KOTT (editors) Business Survey Methods

*

DEMING Sample Design in Business Research

DILLMAN Mail and Internet Surveys: The Tailored Design Method

FULLER Sampling Statistics

GROVES and COUPER Nonresponse in Household Interview Surveys

GROVES Survey Errors and Survey Costs

GROVES, DILLMAN, ELTINGE, and LITTLE Survey Nonresponse

GROVES, BIEMER, LYBERG, MASSEY, NICHOLLS, and WAKSBERG Telephone Survey Methodology

GROVES, FOWLER, COUPER, LEPKOWSKI, SINGER, and TOURANGEAU Survey Methodology,

Second Edition

*

HANSEN, HURWITZ, and MADOW Sample Survey Methods and Theory, Volume 1: Methods and Applications

*

HANSEN, HURWITZ, and MADOW Sample Survey Methods and Theory, Volume II: Theory

HARKNESS, BRAUN, EDWARDS, JOHNSON, LYBERG, MOHLER, PENNELL, and SMITH (editors) Survey Methods in Multinational, Multiregional, and Multicultural Contexts

HARKNESS, van de VIJVER, and MOHLER (editors) Cross-Cultural Survey Methods

HUNDEPOOL, DOMINGO-FERRER, FRANCONI, GIESSING, NORDHOLT, SPICER, and DE WOLF Statistical Disclosure Control

KALTON and HEERINGA Leslie Kish Selected Papers

KISH Statistical Design for Research

*

KISH Survey Sampling

KORN and GRAUBARD Analysis of Health Surveys

KREUTER (editor) Improving Surveys with Paradata: Analytic Uses of Process Information

LEPKOWSKI, TUCKER, BRICK, DE LEEUW, JAPEC, LAVRAKAS, LINK, and SANGSTER (editors) Advances in Telephone Survey Methodology

LESSLER and KALSBEEK Nonsampling Error in Surveys

LEVY and LEMESHOW Sampling of Populations: Methods and Applications,

Fourth Edition

LUMLEY Complex Surveys: A Guide to Analysis Using R

LYBERG, BIEMER, COLLINS, de LEEUW, DIPPO, SCHWARZ, TREWIN (editors) Survey Measurement and Process Quality

LYNN Methodology of Longitudinal Surveys

MADANS, MILLER, and MAITLAND (editors) Question Evaluation Methods: Contributing to the Science of Data Quality

MAYNARD, HOUTKOOP-STEENSTRA, SCHAEFFER, and VAN DER ZOUWEN Standardization and Tacit Knowledge: Interaction and Practice in the Survey Interview

MILLER, WILLSON, CHEPP, and PADILLA (editors)

Cognitive Interviewing Methodology

PORTER (editor) Overcoming Survey Research Problems: New Directions for Institutional Research, No. 121

PRESSER, ROTHGEB, COUPER, LESSLER, MARTIN, MARTIN, and SINGER (editors) Methods for Testing and Evaluating Survey Questionnaires

RAO Small Area Estimation

REA and PARKER Designing and Conducting Survey Research: A Comprehensive Guide,

Third Edition

SARIS and GALLHOFER Design, Evaluation, and Analysis of Questionnaires for Survey Research,

Second Edition

SÄRNDAL and LUNDSTRÖM Estimation in Surveys with Nonresponse

SCHWARZ and SUDMAN (editors) Answering Questions: Methodology for Determining Cognitive and Communicative Processes in Survey Research

SIRKEN, HERRMANN, SCHECHTER, SCHWARZ, TANUR, and TOURANGEAU (editors) Cognition and Survey Research

SNIJKERS, HARALDSEN, JONES, and WILLIMACK Designing and Conducting Business Surveys

STOOP, BILLIET, KOCH and FITZGERALD Improving Survey Response: Lessons Learned from the European Social Survey

SUDMAN, BRADBURN, and SCHWARZ Thinking about Answers: The Application of Cognitive Processes to Survey Methodology

UMBACH (editor) Survey Research Emerging Issues: New Directions for Institutional Research No. 127

VALLIANT, DORFMAN, and ROYALL Finite Population Sampling and Inference: A Prediction Approach

WALLGREN and WALLGREN Register-based Statistics: Administrative Data for Statistical Purposes,

Second Edition

*

Now available in a lower priced paperback edition in the Wiley Classics Library.

COGNITIVE INTERVIEWING METHODOLOGY

Edited by

Kristen Miller

National Center for Health Statistics

Stephanie Willson

National Center for Health Statistics

Valerie Chepp

Hamline University

José-Luis Padilla

University of Granada, Spain

Copyright © 2014 by John Wiley & Sons, Inc. All rights reserved.

Published by John Wiley & Sons, Inc., Hoboken, New Jersey. Published simultaneously in Canada.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, [phone], fax [phone], or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, [phone], fax [phone], or online at http://www.wiley.com/go/permissions.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at [phone], outside the United States at [phone] or fax [phone].

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com.

Library of Congress Cataloging-in-Publication Data:

Cognitive interviewing methodology / edited by Kristen Miller, National Center for Health Statistics, Stephanie Willson, National Center for Health Statistics, Valerie Chepp, National Center for Health Statistics, Jose-Luis Padilla, University of Granada, Spain.

       pages cm     Includes bibliographical references and index.    ISBN 978-1-118-38354-4 (paperback)  1. Interviewing. 2. Cognition. 3. Questionnaires–Methodology. 4. Social surveys–Methodology. 5. Social sciences–Research–Methodology. 6. Psychology–Research–Methodology. I. Miller, Kristen.     H61.28.C64 2014     001.4′33–dc23

2014011436

Dedicated to Janet Harkness,

friend and pioneer of cross-cultural survey methods

CONTENTS

Foreword

Acknowledgments

Contributors

1 Introduction

1.1 Cognitive Interviewing Methodology

2 Foundations and New Directions

2.1 Introduction

2.2 Sociology and the Interpretivist Tradition

2.3 New Directions: Interpretation and Cognition

2.4 Methodological Implications for Cognitive Interviewing

2.5 Conclusion

3 Data Collection

3.1 Introduction

3.2 Cognitive Interviewing Study Sample

3.3 The Cognitive Interview

3.4 The Role of Interviewer

3.5 Conclusion

4 Analysis

4.1 Introduction

4.2 Analysis of Cognitive Interviews: Overview

4.3 Analytic Steps for Cognitive Interviews

4.4 The Benefits of a Complete Analysis

4.5 Conclusion

Note

5 Assessing Translated Questions via Cognitive Interviewing

5.1 Introduction

5.2 Why Use Cognitive Testing in Multilingual Survey Research

5.3 Translation and Translation Assessment Procedures

5.4 Cognitively Testing Translations of Survey Questions

5.5 Problems Uncovered by Cognitive Testing of Translations

5.6 Conclusion

Note

6 Conveying Results

6.1 Introduction

6.2 Contents of a Cognitive Interviewing Report

6.3 Characteristics of a Cognitive Interviewing Report

6.4 Conclusion

7 Case Study: Evaluation of a Sexual Identity Question

7.1 Introduction

7.2 Background

7.3 Case Study: Cognitive Interviewing Evaluation of the National Health Interview Survey Revised Sexual Identity Question

7.4 Case Study Findings

7.5 Conclusion

Notes

8 Analysis Software for Cognitive Interviewing Studies: Q-Notes and Q-Bank

8.1 Introduction

8.2 Q-Notes Analysis Features

8.3 Project Management Features

8.4 Q-Bank: Making Cognitive Interview Findings Publicly Accessible

8.5 Q-Bank Features

8.6 Q-Bank: Challenges for the Past and Future

8.7 Conclusion

Note

9 Cognitive Interviewing in Mixed Research

9.1 Introduction

9.2 The Mixed Research Paradigm: Characteristics and Design

9.3 Mixed Method Research and Survey Question Evaluation

9.4 Conclusion

10 Conclusion

10.1 Introduction

10.2 Summary of Practices

10.3 New Directions

Key Concepts

Question Evaluation Resources

Online Resources

Checklists and Standards

Academic Journals

Professional Organizations/Meetings

References

Index

End User License Agreement

List of Tables

Chapter 5

Table 5.1

Chapter 6

Table 6.1

Chapter 7

Table 7.1

Table 7.2

Table 7.3

Table 7.4

Chapter 8

Table 8.1

Chapter 9

Table 9.1

Table 9.2

Table 9.3

Table 9.4

Table 9.5

Table 9.6

Table 9.7

Table 9.8

Table 9.9

Table 9.10

List of Illustrations

Chapter 4

FIGURE 4.1

Products of data reduction for analytic steps

FIGURE 4.2

Tiers of theory building for analytic steps

FIGURE 4.3

Visual representation of thematic schema

FIGURE 4.4

Visual representation of thematic schema

FIGURE 4.5

Entire thematic schema

FIGURE 4.6

Advanced schema: comparing across groups

Chapter 7

FIGURE 7.1

Revised sexual identity question and follow-up questions

FIGURE 7.2

Conceptual map of sexual identity constructs

Chapter 8

FIGURE 8.1

Thematic schema for self-care question

Chapter 9

FIGURE 9.1

Research designs

FIGURE 9.2

Anxiety questions examined in the cognitive interviewing study

FIGURE 9.3

Anxiety field test questions

Guide

Cover

Table of Contents

Chapter

Pages

xiii

xiv

xv

xvii

1

2

3

4

5

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

163

164

165

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

FOREWORD

As an early practitioner of cognitive interviewing, I can remember presenting many talks on this new science throughout the 1990s. Occasionally, an audience member would ask a pointed question: Although its proponents spoke of the cognitive interview as an application of psychology, were we perhaps missing something by not taking into account other disciplines as well—like linguistics, sociology, anthropology, and so on? I thought this to be a good point, despite my strong focus on cognitive psychology as an anchoring point. In fact, over the ensuing years, there have been a number of contributions that have emphasized a wider disciplinary perspective—including the argument that responses to survey questions involve more than just the individual mind of the respondent, especially as they incorporate social and cultural phenomena in a social context.

In the current volume, Kristen Miller and her colleagues provide what I believe to be the clearest statement of this truth, and the furthest point in the evolution of cognitive interviewing as a mature expression of qualitative research that provides a rich multidisciplinary perspective. The arguments, illustrations, and examples within this book challenge practitioners of cognitive interviewing—and more broadly, anyone having an interest in the subtleties of questionnaire design—to think in new ways about how survey questions are developed by designers, answered by respondents, and consumed by data users. In particular, as what I believe to be the main contribution of the volume, they expand our fundamental notion of why we choose to conduct a cognitive interview. Rather than viewing this endeavor only as an attempt to “patch up” deficiencies by identifying and remediating flawed survey questions, the authors conceptualize the cognitive testing enterprise as an opportunity to obtain a more comprehensive view of the items under our microscope. This interpretivist viewpoint allows us to alter our underlying research question—so that rather than asking “What's wrong with the survey question?”—we can conversely ask “What's right with it?” More to the point, we can hone that question by asking “How does the question function, and what does this imply about the contexts in which it can profitably be employed?” This expansive viewpoint is clearly of interest across a wide range of applications involving the use of self-report data collection instruments.

Although I use the term “microscopic” above, Miller et al. also further the field of cognitive interviewing by incorporating a vital macroscopic view in leading us to step back and consider the wider context of how survey items function across a range of cultures, languages, countries, and other contexts that are increasingly relevant to survey methodology. The book is the first to tackle the challenges of comparative cognitive interviewing, and takes a head-on approach to providing practical assistance to those who face the myriad challenges of question development and evaluation when faced with requirements of instrument translation, interviewing teams that speak different primary languages, and questionnaires that simply do not apply well due to cultural and structural variation. Having collaborated with Dr. Miller in particular over the recent years in which cross-cultural cognitive interviewing has taken root and grown, I can well appreciate the way she has been able to make use of battle-tested experience to save others from having to learn the same hard lessons over again.

A third unique contribution of this volume relates to analysis—well-recognized as the Achilles Heel of the cognitive interviewing process. In a word, the authors preach transparency: We need to put our cards on the table in demonstrating exactly what we mean when we say we have conducted cognitive interviews, what our data consist of, and most importantly, how we came to the conclusions we present within a cognitive testing report. Following an increasingly salient thread within the qualitative research tradition, the book provides clear examples, and conceptual direction, concerning how the results of cognitive interviews should be systematically and openly processed, so that a complete analysis is conducted. By paying significantly more attention to our analytic processes, we end up with a product that is coherent, defensible, and that sets the stage for replication and further advancement of the field as a whole.

Finally, Miller and colleagues look beyond the cognitive interview to also consider the associated pretesting approaches that exist within our ready toolbox of questionnaire development, evaluation, and testing methods. Although the notion that we can look to alternatives, such as behavior coding, psychometric, and field-based experimental studies, has deep roots in the survey methods field, the current volume advocates tying these roots together, through the use of mixed-method studies that leverage the unique strengths of each approach. In particular, the use of quantitative methods reveals how much, or how often, a phenomenon exists; whereas the overlaying of intensive qualitative methods like the cognitive interview reveals “why this happens” due to the richness of the information the qualitative perspective provides. In summary, the current book provides a clear pathway to new thinking, new methods, and new directions for questionnaire designers, survey managers, and data users.

GORDON WILLIS

National Cancer Institute

ACKNOWLEDGMENTS

This book has taken us somewhat longer to write than we initially anticipated. The additional time, however, brought additional critique, debate, and refinement of our ideas.

We thank Catherine Simile for providing perspective and significant insight, and Mitch Loeb for his helpful review and input. We thank our colleagues from Swan Solutions, Florencia Ramirez and Luis Cortes, for editorial comments and insurmountable help in pulling together the entire manuscript including figures, tables, bibliography, and appendices. Special thanks go to Lee Burch also of Swan Solutions for his many years of inspiration and support, as well as Karen Whitaker—office manager extraordinaire—who continuously reminds us to think about the “big picture” while keeping us on task in the here and now. We are especially grateful for all our colleagues in the Questionnaire Design Research Laboratory at the National Center for Health Statistics (NCHS) who, collectively, have helped to improve cognitive interviewing methodology.

We also thank the members of the question evaluation community who developed and sharpened the field over the past 20 years. We are particularly grateful for conversation (and sometimes loud debate!) with Gordon Willis, Norman Bradburn, Janet Harkness, Jack Fowler, Paul Beatty, Fred Conrad, Terry DeMaio, Jennifer Rothgeb, Peter Mohler, Rory Fitzgerald, and Debbie Collins—all of whom helped to shape our thinking.

Additionally, we thank our institutions: the National Center for Health Statistics along with the NCHS Office of Research and Methodology which, under the direction of Nat Schenker, promoted and prioritized question evaluation methodology, providing us the resources and time to develop this work. The University of Granada and the Spanish National Statistics Institute, particularly, Miguel Angel Martínez Vidal who pushed the cognitive interviewing projects in Spain.

We are appreciative of Wiley and our editors, Sari Friedman and Steve Quigley, for realizing the value of this project.

A most special thank you to the NCHS Associate Director of Science, Jennifer Madans, who for over a decade pushed us, argued with us, forced us to articulate better (and sometimes drove us crazy!) more than anyone else. Without her mentorship and sincere dedication to question evaluation and the advancement of survey methodology, this book would not exist. For this, we are truly grateful.