Handbook of Practical Program Evaluation E-Book

Kathryn E. Newcomer

• Herausgeber: John Wiley & Sons
• Kategorie: Fachliteratur
• Serie: Essential Texts for Nonprofit and Public Leadership and Management
• Sprache: Englisch

Seitenzahl: 1633

Veröffentlichungsjahr: 2015

Beschreibung

The leading program evaluation reference, updated with the latest tools and techniques The Handbook of Practical Program Evaluation provides tools for managers and evaluators to address questions about the performance of public and nonprofit programs. Neatly integrating authoritative, high-level information with practicality and readability, this guide gives you the tools and processes you need to analyze your program's operations and outcomes more accurately. This new fourth edition has been thoroughly updated and revised, with new coverage of the latest evaluation methods, including: * Culturally responsive evaluation * Adopting designs and tools to evaluate multi-service community change programs * Using role playing to collect data * Using cognitive interviewing to pre-test surveys * Coding qualitative data You'll discover robust analysis methods that produce a more accurate picture of program results, and learn how to trace causality back to the source to see how much of the outcome can be directly attributed to the program. Written by award-winning experts at the top of the field, this book also contains contributions from the leading evaluation authorities among academics and practitioners to provide the most comprehensive, up-to-date reference on the topic. Valid and reliable data constitute the bedrock of accurate analysis, and since funding relies more heavily on program analysis than ever before, you cannot afford to rely on weak or outdated methods. This book gives you expert insight and leading edge tools that help you paint a more accurate picture of your program's processes and results, including: * Obtaining valid, reliable, and credible performance data * Engaging and working with stakeholders to design valuable evaluations and performance monitoring systems * Assessing program outcomes and tracing desired outcomes to program activities * Providing robust analyses of both quantitative and qualitative data Governmental bodies, foundations, individual donors, and other funding bodies are increasingly demanding information on the use of program funds and program results. The Handbook of Practical Program Evaluation shows you how to collect and present valid and reliable data about programs.

Leseprobe

HANDBOOK OF PRACTICAL PROGRAM EVALUATION

FOURTH EDITION

Kathryn E. Newcomer

Harry P. Hatry

Joseph S. Wholey

Cover image: ©iStock.com/naqiewei Cover design: Wiley

This book is printed on acid-free paper.

Copyright © 2015 by Kathryn E. Newcomer and Harry P. Hatry, and Joseph S. Wholey. All rights reserved.

Published by John Wiley & Sons, Inc., Hoboken, New Jersey Published simultaneously in Canada

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, [phone], fax [phone], or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, [phone], fax [phone], or online at www.wiley.com/go/permissions.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with the respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor the author shall be liable for damages arising herefrom.

For general information about our other products and services, please contact our Customer Care Department within the United States at [phone], outside the United States at [phone] or fax [phone].

Wiley publishes in a variety of print and electronic formats and by print-on-demand. Some material included with standard print versions of this book may not be included in e-books or in print-on-demand. If this book refers to media such as a CD or DVD that is not included in the version you purchased, you may download this material at http://booksupport.wiley.com. For more information about Wiley products, visit www.wiley.com.

Library of Congress Cataloging-in-Publication Data:Handbook of practical program evaluation / Kathryn E. Newcomer, Harry P. Hatry, Joseph S. Wholey, editors. — Fourth edition.   pages cm. — (Essential texts for nonprofit and public leadership and management) Includes index. ISBN 978-1-118-89360-9 (hardback : alk. paper) ; ISBN 978-1-118-89361-6 (pdf); ISBN 978-1-118-89369-2 (epub)   1. Policy sciences. 2. Political planning—Evaluation. I. Newcomer, Kathryn E., editor. II. Hatry, Harry P., editor. III. Wholey, Joseph S., editor. H97.H358 2015 658.4—dc23

2015009201

CONTENTS

Preface

Intended Audience

Scope

Need for Program Evaluation

Handbook Organization

Acknowledgments

The Editors

The Contributors

PART ONE Evaluation Planning and Design

The Chapters

CHAPTER ONE Planning and Designing Useful Evaluations

Matching the Evaluation Approach to Information Needs

Supporting Causal Inferences

Planning a Responsive and Useful Evaluation

Using Evaluation Information

Glossary

References

CHAPTER TWO Analyzing and Engaging Stakeholders

Understanding Who Is a Stakeholder—Especially a Key Stakeholder

Identifying and Working with Primary Intended Users

Using Stakeholder Identification and Analysis Techniques

Dealing with Power Differentials

Determining the Evaluation's Purpose and Goals

Engaging Stakeholders

Meeting the Challenges of Turbulent and Uncertain Environments

Conclusion

References

CHAPTER THREE Using Logic Models

What Is a Logic Model?

The Utility of Logic Models

Theory-Driven Evaluation

Building the Logic Model

Conclusion

References

CHAPTER FOUR Exploratory Evaluation

Evaluability Assessment Assesses a Program's Readiness for Evaluation

Rapid Feedback Evaluation Produces Tested Evaluation Designs

Evaluation Synthesis Summarizes What Is Known About Program Performance

Small-Sample Studies May Be Useful in Vetting Performance Measures

Selecting an Exploratory Evaluation Approach

Conclusion

References

CHAPTER FIVE Performance Measurement

Performance Measurement and Program Evaluation

Measurement Systems

Identifying, Operationalizing, and Assessing Performance Measures

Converting Performance Data to Information

Presenting and Analyzing Performance Data

Current Challenges to Performance Measurement

Conclusion: The Outlook

References

CHAPTER SIX Comparison Group Designs

Introduction to Causal Theory for Impact Evaluation

Comparison Group Designs

Conclusion

References

Further Reading

CHAPTER SEVEN Randomized Controlled Trials

History of RCTs

Why Randomize?

Trial Design

Conclusion

References

CHAPTER EIGHT Conducting Case Studies

What Are Case Studies?

Designing Case Studies

Conducting Case Studies

Analyzing the Data

Preparing the Report

Avoiding Common Pitfalls

Conclusion

References

CHAPTER NINE Recruitment and Retention of Study Participants

Planning for Recruitment and Retention

Institutional Review Boards and the Office of Management and Budget

Recruitment and Retention Staffing

Implementing Recruitment and Retention

Monitoring Recruitment and Retention Progress

Cultural Considerations

Conclusion

References

CHAPTER TEN Designing, Managing, and Analyzing Multisite Evaluations

Defining the Multisite Evaluation

Advantages and Disadvantages of Multisite Evaluations

Multisite Approaches and Designs

Strategies for Multisite Data Collection

Assessing Multisite Interventions

Monitoring Multisite Implementation

Quality Control in MSEs

Data Management

Quantitative Analysis Strategies

Qualitative Analysis Strategies

Telling the Story

Final Tips for the MSE Evaluator

References

CHAPTER ELEVEN Evaluating Community Change Programs

Defining Community Change Interventions

Challenges

Guidance for Evaluators and Practitioners

Conclusion

References

Further Reading

CHAPTER TWELVE Culturally Responsive Evaluation

What Is CRE?

Pioneers in the Foundations of CRE

From CRE Theory to CRE Practice

Case Applications of CRE Theory and Practice

Implications for the Profession

Conclusion

Notes

References

PART TWO Practical Data Collection Procedures

The Chapters

Other Data Collection Considerations

CHAPTER THIRTEEN Using Agency Records

Potential Problems and Their Alleviation

Data Quality Control Processes

Other Suggestions for Obtaining Data from Agency Records

Conclusion

Note

References

CHAPTER FOURTEEN Using Surveys

Planning the Survey

Select the Sample

Design the Survey Instrument

Collect Data from Respondents

Prepare Data for Analysis

Present Survey Findings

Conclusion

References

CHAPTER FIFTEEN Role Playing

What Is Role Playing?

Diversity of Uses

Sampling

Data Collection Instruments

Recruiting, Selecting, and Training Role Players

Implementing Role Playing

Practical Problems (and Solutions)

Statistical Analysis

Expanding Applications for Role Playing

Ethical and Legal Issues

Limitations of Role Playing

Conclusion

Notes

References

CHAPTER SIXTEEN Using Ratings by Trained Observers

Uses for Trained Observer Ratings

Is a Trained Observer Method Appropriate for Your Needs?

What You Will Need to Start

Decisions About Ratings and Sampling

Examples of Trained Observer Programs

Presenting Findings for Trained Observations

Quality Control

Using Technology or Paper?

Benefits of the Trained Observer Approach

Conclusion

Notes

References

CHAPTER SEVENTEEN Collecting Data in the Field

Objectives of Field Studies

Design Issues

Field Visit Protocol

Data Maintenance and Analysis

Conclusion

References

Further Reading

CHAPTER EIGHTEEN Using the Internet

Using the Internet for Literature Reviews

Conducting Surveys on the Internet

Putting Your Program Evaluation on the Web

References

Further Reading

CHAPTER NINETEEN Conducting Semi-Structured Interviews

Disadvantages and Advantages of SSIs

Designing and Conducting SSIs

Polishing Interview Techniques

Analyzing and Reporting SSIs

References

CHAPTER TWENTY Focus Group Interviewing

Examples of Focus Group Use

Characteristics of Focus Group Interviews

Responsibilities

Planning

Developing Questions

Recruiting

Moderating

Analysis

Addressing Challenges in Focus Group Interviews

Conclusion

Reference

Further Reading

CHAPTER TWENTY-ONE Using Stories in Evaluation

How Stories Enrich Evaluations

A Definition of an Evaluation Story

How Stories Can Be Used in Evaluation Studies

An Overview of Critical Steps

Strategies of Expert Storytellers: Presenting the Story Effectively

Challenges in Using Stories and How to Manage Them

A Final Thought

Conclusion

References

Further Reading

PART THREE Data Analysis

The Chapters

CHAPTER TWENTY-TWO Qualitative Data Analysis

Types of Evaluation and Analytic Purpose

Application

Application

Application

Application

Framing Analytic Choices

Program Evaluation Standards and Quality criteria for QDA

Conclusion

References

CHAPTER TWENTY-THREE Using Statistics in Evaluation

Descriptive Statistics: Simple Measures Used in Evaluations

Inferential Statistics: From Samples to Populations

Selecting Appropriate Statistics

Reporting Statistics Appropriately

Reporting Statistical Results to High-Level Public Officials

Conclusion

Appendix 23A: An Application of the Chi-Square Statistic Calculated with SPSS

Appendix 23B: An Application of the

t

Test

References

Further Reading Textbooks

Special Topics

Statistical Software

CHAPTER TWENTY-FOUR Cost-Effectiveness and Cost-Benefit Analysis

Step 1: Set the Framework for the Analysis

Step 2: Decide Whose Costs and Benefits Should Be Recognized

Step 3: Identify and Categorize Costs and Benefits

Step 4: Project Cost and Benefits Over the Life of the Program, If Applicable

Step 5: Monetizing (Putting a Dollar Value on) Costs

Costs to the Private Sector

Costs to Participants and Volunteers

Step 6: Quantify (for CEA) and Monetize (for CBA) Benefits

Step 7: Discount Costs and Benefits to Obtain Present Values

Step 8: Compute Cost-Effectiveness Ratio (for CEA) or Net Present Value (for CBA)

Step 9: Perform Sensitivity Analysis

Step 10: Make a Recommendation

Conclusion

Notes

References

CHAPTER TWENTY-FIVE Meta-Analyses, Systematic Reviews, and Evaluation Syntheses

Why Be Conscientious in Reviewing Studies of Intervention Effects?

How Are the Best Approaches to Systematic Reviews Employed at Their Best?

What Resources Can Be Employed to Do the Job Well?

To What End? Value Added and Usefulness

Conclusion

Note

References

PART FOUR Use of Evaluation

The Chapters

CHAPTER TWENTY-SIX Pitfalls in Evaluations

Pitfalls Before Data Collection Begins

Pitfalls During Data Collection

Pitfalls After Data Collection

Conclusion

Note

References

CHAPTER TWENTY-SEVEN Providing Recommendations, Suggestions, and Options for Improvement