Qualitative Inquiry in Evaluation E-Book

• Herausgeber: John Wiley & Sons
• Kategorie: Fachliteratur
• Serie: Research Methods for the Social Sciences
• Sprache: Englisch

Seitenzahl: 505

Veröffentlichungsjahr: 2014

Beschreibung

Integrate qualitative inquiry approaches and methods into the practice of evaluation

Qualitative inquiry can have a major effect on evaluation practice, and provides evaluators a means to explore and examine various settings and contexts in need of rich description and deeper understanding. Qualitative Inquiry in Evaluation: From Theory to Practice explores the most important considerations for both students and evaluation professionals. Using various evaluation theories and approaches as a springboard for real-world practice, this reference serves as an accessible text for beginning students and seasoned professionals alike. Readers are given an in-depth view of the key qualities and benefits of qualitative inquiry, which also serves as a crucial counterpart to quantitative analysis.

Chapters in part one focus on the foundations, core concepts, and intersection of evaluation theory and qualitative inquiry.

Part two contains contributions from leading evaluators whose design, implementation, and reporting strategies for qualitative inquiry are centered on common, real-world settings. These case-based chapters point to the strengths and challenges of implementing qualitative evaluations. Key competencies for conducting effective qualitative evaluations are also discussed.

• Explores the role of qualitative inquiry in many prominent approaches to evaluation
• Discusses the method's history and delves into key concepts in qualitative inquiry and evaluation
• Helps readers understand which qualities are necessary to be an effective qualitative evaluator
• Presents the viewpoints and experiences of expert editors and contributing authors with high levels of understanding on the topic

Qualitative Inquiry in Evaluation: From Theory to Practice is a vital tool for evaluators and students alike who are looking to deepen their understanding of the theoretical perspectives and practice considerations of qualitative evaluation.

Leseprobe

Table of Contents

Title Page

Copyright

Figures and Tables

Figures

Tables

Introduction

What Is This Book About?

Who Is This Book For and What Do We Hope You Will Get from Reading It?

How to Read This Book

Quality in Qualitative Evaluation

References

About the Editors

About the Contributors

Acknowledgments

Part 1: The Intersection of Qualitative Inquiry and Evaluation Approaches

Chapter 1: The Origins, Meaning, and Significance of Qualitative Inquiry in Evaluation

The Genesis of Qualitative Evaluation

What Is Qualitative Evaluation?

Enduring Contributions

Conclusion

Key Concepts

Discussion Questions

References

Chapter 2: Qualitative Inquiry in Utilization-Focused Evaluation

The Focus on Evaluation Use as the Primary Evaluation Outcome

Basic Definitions

Involving Intended Users in Making Evaluation Decisions

Qualitative Inquiry into Evaluation Use: Emergence of the Personal Factor

Considerations for Incorporating Qualitative Inquiry into Utilization-Focused Evaluations

Particularly Appropriate Uses of Qualitative Evaluations

Value of Incorporating Qualitative Approaches

Challenges Associated with Qualitative Inquiry in Utilization-Focused Evaluations

Conclusion

Key Concepts

Discussion Questions

References

Chapter 3: Qualitative Inquiry for Policy Makers

Who Are the Policy Makers and How Do They Operate?

Why Qualitative Analysis Is Important to Policy Makers

Principles for Influencing Policy Makers

Conclusion

Key Concepts

Discussion Questions

References

Chapter 4: Qualitative Inquiry within Theory-driven Evaluation: Perspectives and Future Directions

Theory-driven Evaluation: A Brief History

Key Elements in Theory-driven Evaluation (TDE)

Qualitative Inquiry: An Imperative to Theory-driven Evaluation

Conclusion

Key Concepts

Discussion Questions

References

Chapter 5: Conceptual and Practical Intersections between Participatory Evaluation and Qualitative Inquiry

Participatory Evaluation: Concepts and Principles of Practice

Qualitative Inquiry: A Brief Overview of Key Characteristics

Locating Participatory Evaluation and Qualitative Inquiry: Conceptual Intersections

Epistemological Intersections

Methodological Linkages

Methods

Practical Intersections

Key Benefits

Potential Challenges

Conclusion

Key Concepts

Discussion Questions

References

Part 2: Tales from the Field of Qualitative Evaluation

Chapter 6: What Does It Take to Be an Effective Qualitative Evaluator?: Essential Competencies

Competencies for Professional Practice

Toward Competencies for Evaluators

Evaluator Competency Taxonomies

Competencies for Qualitative Evaluators

Core Competency Domains

Commitment to the Qualitative Paradigm

Technical Expertise

The Qualitative Sixth Sense

Conclusion

Key Concepts

Discussion Questions

References

Chapter 7: Dealing with Asymmetric Relations between Stakeholders: Facilitating Dialogue and Mutual Learning through Qualitative Inquiry

Evaluators' Outlook and Identity

The Project

Evaluators' Methodological Choices and Actions

Conclusion

Key Concepts

Discussion Questions

References

Chapter 8: Balancing Insider-Outsider Roles as a New External Evaluator

Responding to the Sponsoring Organization's Request for Proposals

The Use of Qualitative Inquiry

Personal Influences on Evaluation Design

Trends in Health Education

Delving into the Formality of Evaluation Design

Data Collection, Analysis, and Reporting

Reflections on Evaluator Background

Reflections on Our Evaluation Process

Reflections on Our Use of Qualitative Inquiry

Conclusion

Key Concepts

Discussion Questions

References

Chapter 9: Whose Story Is This, Anyway?: Navigating an Ethical Conundrum

The Context: Who's Doing What?

The Moral Lens on Qualitative Inquiry: Principles and Ethical Theories

Challenges: Ethical and Logistic

Conclusion

Acknowledgments

Key Concepts

Discussion Questions

Notes

References

Chapter 10: Elements of Quality in Qualitative Evaluation

Understanding and Acknowledging Personal Identity

Building Trusting Relationships

Reflecting on and Managing Relationships

Employing Sound and Explicit Methodology, and Being Transparent about It

Staying True to the Data

Fostering Learning

Key Concepts

Notes

References

Index

End User License Agreement

Pages

xi

xiii

xiv

xv

xvi

xvii

xviii

xix

xxi

xxii

xxiii

xxv

xxvi

xxvii

xxviii

xxix

xxxi

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

286

287

288

289

290

291

292

293

294

295

296

Guide

Cover

Table of Contents

Introduction

Part 1: The Intersection of Qualitative Inquiry and Evaluation Approaches

Begin Reading

List of Illustrations

Figure 2.1

Figure 10.1

List of Tables

Table 5.1

Table 5.2

Table 5.3

Table 5.4

Table 6.1

Table 6.2

Table 6.3

Table 9.1

QUALITATIVE INQUIRY IN EVALUATION

From Theory to Practice

LESLIE GOODYEAR, JENNIFER JEWISS, JANET USINGER, AND ERIC BARELA

Cover design by Wiley

Cover image : © Corey Holms | Getty

Copyright © 2014 by John Wiley & Sons, Inc. All rights reserved.

Published by Jossey-Bass

A Wiley Brand

One Montgomery Street, Suite 1200, San Francisco, CA 94104-4594— www.josseybass.com

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400, fax [phone], or on the Web at www.copyright.com. Requests to the publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, 201-748-6011, fax [phone], or online at www.wiley.com/go/permissions.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. Readers should be aware that Internet Web sites offered as citations and/or sources for further information may have changed or disappeared between the time this was written and when it is read.

Jossey-Bass books and products are available through most bookstores. To contact Jossey-Bass directly call our Customer Care Department within the U.S. at 800-956-7739, outside the U.S. at 317-572-3986, or fax [phone].

Wiley publishes in a variety of print and electronic formats and by print-on-demand. Some material included with standard print versions of this book may not be included in e-books or in print-on-demand. If this book refers to media such as a CD or DVD that is not included in the version you purchased, you may download this material at http://booksupport.wiley.com. For more information about Wiley products, visit www.wiley.com.

Library of Congress Cataloging-in-Publication Data

Qualitative inquiry in evaluation : from theory to practice / edited by Leslie Goodyear, Eric Barela, Jennifer Jewiss, and Janet Usinger. — 1

pages cm — (Research methods for the social sciences ; 29)

Includes bibliographical references and index.

ISBN 978-0-470-44767-3 (paperback) — ISBN 978-1-118-41815-4 (pdf) — ISBN 978-1-118-41525-2 (epub) 1. Educational evaluation. I. Goodyear, Leslie, 1966-

LB2822.75.Q354 2014

379.1′58 — dc23

2014015898

FIRST EDITION

Figures and Tables

Figures

Figure 2.1 The Iterative and Interdependent Set of Evaluation Questions that Must be Negotiated with Primary Intended Users in a Utilization-Focused Evaluation

Figure 10.1 Cycle of Quality in Qualitative Evaluation

Tables

Table 5.1 A Summary of Differences and Similarities between P-PE and T-PE

Table 5.2 A Summary Description of Key Qualitative Features

Table 5.3 Ladder of Participation

Table 5.4 A Summary of Epistemological Intersections

Table 6.1 Comparison of Origin/Purpose, Structure, and Validation across Evaluator Competency Taxonomies

Table 6.2 Cross-Comparison of Evaluator Competency Taxonomies

Table 6.3 Common Methods for Qualitative Evaluators

Table 9.1 Quarterly Report to Funding Agency

Introduction

What Is This Book About?

To put it simply, this book is about integrating qualitative inquiry—its approaches, methods, stances, and theories—into the practice of evaluation. What do we mean by “qualitative inquiry” and “evaluation”? We agree with Thomas Schwandt (2007) that qualitative inquiry “aims at understanding the meaning of human action” (italics in original) (p. 248). It includes many forms of social inquiry such as case study research, naturalistic inquiry, ethnography, and others. Likewise, many definitions of evaluation are in play these days; however, Peter Dahler-Larsen (2012) summarizes them quite succinctly when he defines evaluation as “basically a systematic, methodological, and thus ‘assisted’ way of investigating and assessing an activity of public interest in order to affect decisions or actions concerning this activity or similar activities” (p. 9).

This book is about the intersection of evaluation and qualitative inquiry; that space in a Venn diagram where qualitative inquiry and evaluation overlap. It is the space where the two colored circles representing qualitative inquiry and evaluation come together to create a new color. That part of the diagram is what we are calling qualitative evaluation. That is what this book is about.

Sometimes, to understand what something is, it is helpful to understand what it is not. This book is not a how-to textbook. This edited volume will not tell you how to develop and implement a qualitative evaluation. Instead, we hope this book will stimulate your thinking on how different evaluation approaches dovetail with qualitative inquiry. In addition, the second part of the book offers “tales of the field” (Van Maanen, 1988) that share the ins and outs of the practice of qualitative evaluation.

Who Is This Book For and What Do We Hope You Will Get from Reading It?

As we surveyed the landscape of available literature, we found a wealth of resources on qualitative research and also on evaluation. However, we found few titles dedicated to an exploration of the intersection of qualitative inquiry and evaluation. This book focuses on that intersection, delving deeply into what it means to carry out qualitative evaluation and examining the practice of qualitative evaluation from various theoretical and practical angles.

This book is geared toward students and evaluators who are relatively new to the practice of qualitative inquiry in evaluation, but who have some foundational knowledge of both evaluation and qualitative inquiry. Its purpose is twofold: as a practical introduction for students and new evaluators and as a reference and resource for practicing evaluators. As an edited volume, this compilation of many voices is meant to spark dialogue among evaluators. Our hope is that the array of perspectives herein provides a window on the world of qualitative evaluation—a travelogue of sorts that describes the terrain and offers examples and lessons learned along the way.

For evaluators who are already using qualitative evaluation approaches, we hope the featured theoretical perspectives and practice considerations will deepen their understanding of the knowledge, skills, and epistemological stances that seasoned qualitative evaluators bring to their work. In short, we hope readers of all backgrounds will find it helpful to “listen in” as the contributing evaluators reflect on the theoretical perspectives that inform their practice and the practice-based experiences that have surfaced new layers of learning about their craft.

Many graduate students receive methodological training in qualitative research but are not adequately prepared to employ those skills out in the field, particularly in the context of an evaluation. In addition, many experienced evaluators with quantitative research backgrounds are interested in learning how to incorporate qualitative approaches into the evaluations they conduct. As many readers are no doubt aware, incorporating qualitative inquiry in evaluation entails much more than simply “doing some interviews” to gather a smattering of quotes to “round out” the quantitative data and analysis. The practicing evaluator must address a substantial number of considerations, and these considerations are not necessarily covered in qualitative research or evaluation textbooks. The chapters in Part 2 of this book illustrate these concepts and tell the tales of practitioners faced with real-world challenges and opportunities as they implement qualitative evaluations.

Going “into the field” to carry out qualitative inquiry—whether facilitating a meeting with evaluation stakeholders or conducting interviews—can be an exciting, uncertain, and even daunting task. We hope this collection of chapters deepens readers' thinking about the endeavor by exploring territory beyond that covered in more methodologically oriented texts and teaching.

How to Read This Book

As we constructed this volume, we thought about our own experiences as we each entered the field of evaluation and what we have encountered in our practice. We specifically divided the volume into two parts: one focuses on the intersection of evaluation approaches and qualitative inquiry; the other focuses on how evaluators think about and navigate their evaluation practice.

Part 1 of this volume is of particular value to people who are not steeped in evaluation theory. The first set of chapters provides grounding for people who may understand the epistemology of qualitative inquiry, but not necessarily how it dovetails with evaluation approaches. On the flipside, Part 1 also provides grounding for evaluators who want to deepen their exploration of the role of qualitative inquiry in evaluation.

Chapter 1, written by Thomas A. Schwandt and Timothy J. Cash, provides an historical journey from the advent of qualitative inquiry through its increasing incorporation into the practice of evaluation. Following an overview of the contested definition of qualitative evaluation, they conclude their chapter by describing the three enduring contributions that qualitative inquiry has made in: (1) shaping the ways an evaluation is focused; (2) how values are understood and portrayed; and (3) how evaluations are communicated.

After this orientation, we turn the discussion over to several evaluators who have been leaders in shaping different evaluation approaches and effectively integrating qualitative inquiry. Michael Quinn Patton has long been a pioneer in designing evaluations that are actually used by program stakeholders. In Chapter 2, he shares how he thinks about the intersection of utilization-focused evaluation and qualitative inquiry and highlights both opportunities and challenges for practice.

For those entering the realm of using evaluation to affect public policy, George F. Grob offers lessons he has learned from many years of work with policy makers at the local, state, and national levels. In Chapter 3, he shares his thoughts and experiences regarding the value of qualitative data in informing policy makers about public programs. His pragmatic approach demonstrates how evaluation operates in the “simultaneously fast paced and glacially slow” arena of policy making.

In Chapter 4, Katrina L. Bledsoe argues that although theory-driven evaluation is often associated with randomized controlled trials and the generalizability of programs, to effectively implement theory-driven evaluation, an evaluator must embrace qualitative methods when developing logic models, working with participants to articulate a program's theory, and also when collecting and analyzing data.

Jill Anne Chouinard and J. Bradley Cousins are leaders in the development of participatory evaluation. Chapter 5 provides an exploration of the intersections among qualitative inquiry, practical participatory evaluation, and transformative participatory evaluation.

Part 2 of this volume is of particular value to professionals new to the field of evaluation and those who are looking for examples of how qualitative evaluation is done “on the ground.” How does one enter an evaluation setting in a manner that portrays professionalism and command of the situation, but also quickly establishes trust? How does one interact with the evaluation funders when their concept of an evaluation design is inconsistent with the questions they want answered? How does one handle an evaluation that gets off track? These questions, and more, are addressed through case studies of evaluation practice by evaluators who bring varying degrees of experience to the endeavor.

Part 2 begins with an introductory chapter written by Laurie Stevahn and Jean A. King, who have spent much of their academic careers guiding students in the development of important skills and competencies for conducting effective evaluations. In Chapter 6, they describe the competencies required to conduct effective evaluations generally, and they delve deeply into the specialized competencies required to conduct effective qualitative evaluations.

In Chapter 7, Vivianne E. Baur and Tineke A. Abma describe how they used a transformative paradigm with responsive evaluation and hermeneutic dialogue to address power differences in the management of residential units for senior citizens. In addition to discussing how they interacted with the Board that funded the evaluation, this chapter provides a rich description of the strategies these evaluators used to level the playing field and address the unspoken conflicts between the staff and resident council members.

Academically prepared as a health educator and researcher, Norma Martínez-Rubin was new to evaluation when she partnered with Stuart Hanson to conduct a community assessment of pediatric obesity. Chapter 8 tells the story of how she navigated her own Latina cultural identity and professional background, in light of collecting data and portraying the health-related needs of families living in diverse Southern California communities.

In Chapter 9, Sharon F. Rallis describes a situation in which she was caught in an ethical dilemma. As the evaluator of a high-profile, federally funded project, she found herself having to navigate between noted researchers and unhappy public school staff who worked where the project was conducted. This case study provides an opportunity to “listen in” as an experienced evaluator considers how to make sense of—and make the best of—an evaluation that is headed down a problematic path.

The last chapter is written by the editors, Leslie Goodyear, Jennifer Jewiss, Janet Usinger, and Eric Barela. It brings together our own reflections and experiences as evaluators with those of the other authors featured in this volume. As our four-person editing team slowly but steadily propelled this book forward over the course of several years, we engaged in extensive and deep conversations about what constitutes quality in qualitative evaluation. In the end, we arrived at five elements that we believe are hallmarks of quality for evaluations that employ qualitative inquiry.

The final chapter elaborates on how we arrived at these elements and discusses each in depth. We briefly introduce them here, so that readers may consider how these elements play out in various chapters of the book.

Quality in Qualitative Evaluation

Acknowledging Who You Are and What You Bring to the Work

The evaluator must bring to the process a clear sense of personal identity and professional role. It is a matter of understanding who you are, what you know, and what you need to learn.

Building Trusting Relationships

The evaluator needs to engage stakeholders and build trusting relationships from the outset and throughout the evaluation.

Employing Sound and Explicit Methodology

High-quality evaluation relies on sound methodology, systematically applied, that is explicitly shared with stakeholders.

Staying True to the Data

Conducting high-quality evaluation can only be accomplished by remaining “true” to the data; in other words, hearing participants as they are, not how the evaluator wants them to be.

Fostering Learning

Skillful facilitation of the process by the evaluator results in learning by all involved.

As is portrayed in the graphic representation presented in the final chapter, these five elements form a cycle with no set starting point or definitive end point. Like so many things in life—and certainly in the world of evaluation—putting these elements into practice is not nearly as linear or as neat and tidy as the model itself.

We recognize that some readers may be eager to dive deeper into the discussion about the elements of quality before proceeding with the intervening chapters. If that is the case for you, we invite you to flip to the corresponding discussion in the final chapter and then return to main parts of the book. Regardless of what path you decide to take through this edited volume, we hope you will consider how these elements appear not only in the chapters of this book but also in your own evaluation practice.

References

Dahler-Larsen, P. (2012).

The Evaluation Society

. Stanford, CA: Stanford University Press.

Schwandt, T. (2007).

Sage Dictionary of Qualitative Inquiry

, 3rd ed. Thousand Oaks, CA: Sage.

Van Maanen, J. (1988).

Tales of the Field: On Writing Ethnography

. Chicago, IL: University of Chicago Press.

About the Editors

Leslie Goodyear has been a practicing qualitative (and mixed-method) evaluator for nearly twenty years, during which time she has evaluated programs focused on HIV/AIDS prevention, arts, youth development, parenting, youth media, youth civic engagement, afterschool, youth substance abuse prevention, and most recently informal STEM education. In addition to conducting evaluations, she works to build organizational evaluation capacity, provide technical assistance, and build communities of practice for evaluators. Before studying program evaluation at Cornell University, Leslie worked as an early childhood educator and as a frontline and outreach staff person in social service organizations. Her most important evaluation training came in the form of learning to be a crisis counselor and learning to balance the demands of multiple five-year-olds who needed their shoes tied, noses wiped, and questions answered simultaneously. Leslie has a keen interest in how programs are understood, how meaning is made from program experiences, and how that meaning is conveyed to stakeholders so that they can better understand the program experience and make informed decisions. In this vein, she has experimented with new and innovative forms of representation of evaluation findings, including poetry and performance. All of these interests and experiences, and her passion for ethical practice in evaluation, have informed her contributions to the development of this edited volume.