Purposeful Program Theory E-Book

Sue C. Funnell

• Herausgeber: John Wiley & Sons
• Kategorie: Fachliteratur
• Serie: Research Methods for the Social Sciences
• Sprache: Englisch

Seitenzahl: 785

Veröffentlichungsjahr: 2011

Beschreibung

Between good intentions and great results lies a program theory—not just a list of tasks but a vision of what needs to happen, and how. Now widely used in government and not-for-profit organizations, program theory provides a coherent picture of how change occurs and how to improve performance. Purposeful Program Theory shows how to develop, represent, and use program theory thoughtfully and strategically to suit your particular situation, drawing on the fifty-year history of program theory and the authors' experiences over more than twenty-five years.

"From needs assessment to intervention design, from implementation to outcomes evaluation, from policy formulation to policy execution and evaluation, program theory is paramount. But until now no book has examined these multiple uses of program theory in a comprehensive, understandable, and integrated way. This promises to be a breakthrough book, valuable to practitioners, program designers, evaluators, policy analysts, funders, and scholars who care about understanding why an intervention works or doesn't work." —Michael Quinn Patton, author, Utilization-Focused Evaluation

"Finally, the definitive guide to evaluation using program theory! Far from the narrow 'one true way' approaches to program theory, this book provides numerous practical options for applying program theory to fulfill different purposes and constraints, and guides the reader through the sound critical thinking required to select from among the options. The tour de force of the history and use of program theory is a truly global view, with examples from around the world and across the full range of content domains. A must-have for any serious evaluator." —E. Jane Davidson, PhD, Real Evaluation Ltd.

Companion Web site: josseybass.com/go/funnellrogers

Leseprobe

CONTENTS

Figures, Tables, and Exhibits

Acknowledgments

The Authors

Introduction: The Promise and Risks of using Program Theory

Part 1 : Key Ideas in Program Theory

Chapter 1 : The Essence of Program Theory

Evaluation Without Program Theory

Evaluation with Program Theory

Summary

Exercises

Chapter 2 : Variations of Program Theory over Time

A Short History of Program Theory

Terminology in Program Theory

Key Ideas in Program Theory

Summary

Exercises

Chapter 3 : Common Myths and Traps

Some Common Myths

Traps to Avoid When Developing and Using Program Theory

Summary

Exercises

Part 2 : Assessing Your Circumstances

Chapter 4 : Scoping Intended Uses

Why Intended Use Matters

Using Program Theory

Summary

Exercises

Chapter 5 : The Nature of the Situation and the Intervention

Simple, Complicated, and Complex

Focus

Governance

Consistency

Necessariness

Sufficiency

Change Trajectory

Summary

Exercise

Part 3 : Developing and Representing Program Theory

Chapter 6 : Processes to Identify or Develop a Program Theory

Process Options for Developing Program Theory

Decision 1: Who Should Be Involved in Developing a Program Theory, and How?

Decision 2: What Is an Appropriate Mix of Approaches for Developing or Eliciting the Program Theory?

Decision 3: How Might Workshops and Interviews Be Used in Developing Program Theory?

Decision 4: As Challenges Arise, How Should They Be Addressed?

Decision 5: How Much Time and Resources Should Be Invested in Developing or Identifying a Program Theory?

Decision 6: When Is It Time to Revisit a Program Theory?

Summary

Exercise

Chapter 7 : Developing a Theory of Change

Features of a Theory of Change

Situation Analysis: Understanding the Problem, Its Causes, and Its Consequences

Focusing and Scoping

Outcomes Chain

Summary

Exercises

Chapter 8 : Developing a Theory of Action

Preparing the Theory of Action

Success Criteria for a Theory of Change

Assumptions About Factors That Affect Successful Achievement of Outcomes

How to Identify Factors That Are Likely to Affect Outcomes

Identifying What the Program Does

Pulling the Theory of Change and the Theory of Action Together in a Matrix

Summary

Exercise

Chapter 9 : Representing Program Theory

Options for Representation

Representing Complicated Program Theory

Representing Complex Program Theory

What Makes a Good Representation of Program Theory

Should Logic Models Include SMART Measures?

Summary

Exercises

Chapter 10 : Critiquing Program Theory

Criteria for Assessing Internal Validity

Criteria for External Validation

Engaging Stakeholders in the Review

Responding to the Results of a Review of a Program Theory

Summary

Exercise

Part 4 : Resources for Developing Program Theory

Chapter 11 : Some Research-Based Theories of Change

Theory of Reasoned Action and Theory of Planned Behavior

Stages of Change Theory

Empowerment Theory

Diffusion Theory

Socioecological Theory

Network Theory

Selecting and Using Theories of Change

Summary

Exercises

Chapter 12 : Some Common Program Archetypes

Some Important Program Archetypes

Advisory, Information, and Education Program Archetype

Carrots and Sticks Program Archetype

Case Management Programs Program Archetype

Community Capacity-Building Program Archetype

Product or Direct Service Delivery Program Archetype

Deciding which Program Archetype Applies to a Program

Summary

Exercise

Chapter 13 : Logic Models Resources

Pipeline Logic Models

Variations of Outcomes Chain Logic Models

Technology for Representing Program Theory

Summary

Exercise

PART 5 : Using Program Theory for Monitoring and Evaluation

Chapter 14 : Developing a Monitoring and Evaluation Plan

Using Program Theory for Performance Monitoring

Making Choices About What to Measure Within the Program Theory

Including Comparisons as Part of the Performance Information System

Using Program Theory to Plan an Evaluation

Considerations When Using Program Theory to Design Evaluations of Complicated and Complex Programs

Summary

Exercises

Chapter 15 : Causal Inference

The Need to Be Scientific and Pragmatic

A Framework for Causal Analysis Using Program Theory

Congruence

Counterfactual Comparisons

Critical Review

Summary

Exercises

Chapter 16 : Synthesis and Reporting

Synthesis and Reporting for a Single Evaluation

Synthesis and Reporting Across Evaluations

Summary

Exercises

New Frontiers for Program Theory

References

Index

Copyright © 2011 by John Wiley & Sons, Inc. All rights reserved.

Published by Jossey-Bass

A Wiley Imprint

989 Market Street, San Francisco, CA 94103-1741—www.josseybass.com

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400, fax [phone], or on the Web at www.copyright.com. Requests to the publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, 201-748-6011, fax [phone], or online at www.wiley.com/go/permissions.

Readers should be aware that Internet Web sites offered as citations and/or sources for further information may have changed or disappeared between the time this was written and when it is read.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

Jossey-Bass books and products are available through most bookstores. To contact Jossey-Bass directly call our Customer Care Department within the U.S. at 800-956-7739, outside the U.S. at 317-572-3986, or fax [phone].

Jossey-Bass also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books.

Library of Congress Cataloging-in-Publication Data

Funnell, Sue C., date.

Purposeful program theory : effective use of theories of change and logic models / Sue C. Funnell and Patricia J. Rogers.

p. cm.

Includes bibliographical references and index.

ISBN 978-0-470-47857-8 (pbk.); ISBN 978-0-470-93988-8 (ebk); ISBN 978-0-470-93989-5 (ebk); ISBN 978-1-118-00433-3 (ebk)

1. Evaluation research (Social action programs) I. Rogers, Patricia J. II. Title.

H62.F855 2011

001.4'2—dc22

2010034870

FIGURES, TABLES, AND EXHIBITS

Figures1.1An Evaluation of An Apple a Day Without Program Theory1.2Simple Pipeline and Outcomes Chain Logic Models1.3A Logic Model Showing a Simple Program Theory for An Apple a Day Based on Improved Vitamin Intake1.4Logic Models Showing Different Possible Causal Mechanisms Involved in Eating an Apple a Day5.1A Complicated Program Theory for a Multilevel Intervention5.2Emergent Focus of the Intervention: Intermediate Intended Outcomes5.3Emergent Focus of the Intervention: Long-Term Intended Outcomes5.4Emergent Activities to Achieve Stable Intended Outcomes5.5An Intervention Dependent on the Contribution of a Subsequent Intervention5.6Outputs from One Intervention Forming the Inputs to a Subsequent Intervention5.7A Complicated Program Theory for an Intervention That Works Only in Conjunction with Other Interventions5.8Complex Situation: Where Small Differences in Context Lead to Unpredictably Different Impacts6.1Amalgamating Different Stakeholder Theories6.2Positive and Negative Program Theories6.3Linear Theories with Branching Structures7.1Outcomes Chain for the Mature Workers Program7.2Conversion of a Flow Diagram of Activities to a Chain of Outcomes8.1Utilization of Evaluations: Outcomes Chain and Definitions of Successful Outcomes9.1A Pipeline Logic Model of a Computer Skills Project9.2An Outcomes Chain Logic Model of the Computer Project9.3Complicated Logic Model Showing Multiple Organizations9.4Logic Model for a Program Involving Multiple Partner Organizations9.5Stacked Logic Model9.6Logic Model Based on Outcomes Mapping9.7Complicated Logic Model Showing Multiple Causal Strands9.8Logic Model Showing Two Different Sets of Intended Outcomes9.9The Seven D Approach to Developing and Using an Emergent Program Theory9.10The U Process Theory of Change9.11A Broad Theory of Change with Emergent Theory of Action: Dealing with Striga9.12A Theory of Change with Emergent Theory of Action: Evaluation Capacity Building9.13A Theory of Change with Cyclic Learning at Each Stage9.14A Theory of Change with Emergent Theory of Action: Mental Health9.15A Systems Dynamics Model of a Tobacco Control Initiative9.16A Poor Logic Model in Terms of Focusing on the Main Elements and Explaining How It Works9.17A Poor Logic Model in Terms of a Dead End9.18Improved Logic Model in Terms of Linking a Dead End to an Ultimate Impact9.19Poor Logic Model in Terms of Meaningful Arrows9.20Improved Logic Model in Terms of Meaningful Arrows9.21More Clearly Drawn Logic Model9.22Poor Logic Model in Terms of Representation of Direction of Intended Change9.23Poor Logic Model in Terms of Representation of Sequence and Consequence9.24Improved Logic Model in Terms of Sequence and Consequence9.25Logic Model Emphasizing Four Causal Strands, Each with Two Stages9.26Poor Logic Model in Terms of Too Many Indiscriminate Feedback Lines9.27Poor Logic Model in Terms of Visual Elements That Do Not Add Meaning11.1Schematic Representation of the Theory of Planned Behavior11.2Temporal Dimension as the Basis for the Stages of Change11.3S Curve of Diffusion Theory11.4Socioecological Model for Child Development11.5Evolving Networks12.1Generic Outcomes Chain for Advisory, Information, and Education Program Archetype and Applications of the Generic Outcomes Chain12.2Generic Outcomes Chain for the Archetype of Motivational Programs That Use Incentives and Applications of That Chain12.3Generic Outcomes Chain for the Archetype of Motivational Programs That Use Deterrence and an Application of That Chain12.4Generic Outcomes Chain for Case Management Archetypal Program12.5Generic Outcomes Chain for Community Capacity-Building Archetype Program12.6Generic Outcomes Chain for Service or Product Archetype Programs and Applications of the Archetype13.1Charities Evaluation Planning Triangle Logic Model of a Community Health Center13.2United Way Logic Model of a Community Health Center13.3W. K. Kellogg Foundation Logic Model of a Community Health Center13.4Bennett’s Hierarchy Logic Model of a Community Health Center13.5University of Wisconsin Logic Model of a Community Health Center13.6Example of a People-Centered Logic Model13.7Example of ActKnowledge/Aspen Institute Theory of Change Logic Model with Assumptions and Interventions13.8Picture Pipeline Logic Model in Word13.9Picture Pipeline Logic Model in Excel13.10Picture Pipeline Logic Model in Visio13.11Outcomes Chain Logic Model in DoView13.12InnoNET Logic Model Builder Example14.1Simplified Outcomes Chain Showing Levels for Which the Crisis Call Center for Problem Gamblers Was Held Accountable14.2The Logic of Planning an Evaluation14.3Overarching Program Theory14.4Nonlinear Program Theory15.1Results Chain Logic Model for An Apple a Day16.1Reporting Findings Using DoView16.2Reporting Findings Using ShockwaveTables1.1Using Program Theory to Interpret Evaluation Findings2.1Distinctions Sometimes Made Between Program Theory and Program Logic2.2Definitions Used in This Book3.1Some Common Myths About Program Theory3.2Seven Traps to Avoid When Developing and Using Program Theory4.1Implications of Different Aspects of Use for Developing and Using Program Theory5.1Simple, Complicated, and Complex Problems5.2Distinguishing Simple, Complicated, and Complex Aspects of Interventions5.3A Framework for Addressing Simple, Complicated, and Complex Interventions6.1Considerations for Choice of Approach, Methods, and Sources of Information7.1Components of a Program Theory7.2Features of a Theory of Change for Addressing Common Traps8.1Components of a Program Theory8.2Theory of Action Features That Can Avoid Common Traps8.3Factors That Affect the Success of Different Types of Services8.4An Alternative Approach for Focusing on Outcomes, Their Desired Attributes, and Factors That Affect Success8.5Application of the Program Theory Matrix to One Outcome from the Outcomes Chain for the Mature Workers Program9.1A Realist Matrix Logic Model of the Computer Project9.2Options for Formatting Logic Models9.3A Realist Summary of Foster and Hope’s Study of the Effects of PEP on Crime in the Hull Experimental Estate9.4Options for Representing Interventions with Important Complex Aspects11.1Stages of Change Theories Useful Strategies for Different Processes of Change13.1Variations on the Layout of Components in Pipeline Logic Models13.2Variations on the Number of Components in Pipeline Logic Models13.3Variations on the Labels Used for Components in Pipeline Logic Models13.4Variations on the Definitions of Labels Used in Pipeline Logic Models13.5Additional Components Sometimes Included in Pipeline Logic Models13.6Variations of Pipeline Logic Models13.7Components of a Logframe13.8Variations of Outcomes Chain Logic Models13.9Technology for Drawing Logic Models13.10Tabular Pipeline Logic Model in Word for Meals on Wheels14.1Some Types and Purposes of Evaluation at Different Stages of a Program and How Program Theory Can Be Situated14.2Performance Information Matrix for One Outcome in the Outcomes Chain14.3Examples of Evaluation Questions, Performance Information, and Methods of Data Collection for the Outcome: Job Retention for Mature Workers14.4Overview of Characteristics of Simple, Complicated, and Complex Programs14.5Some Implications of Various Characteristics for Monitoring and Evaluation14.6Complicated and Complex Aspects of the Stronger Families and Communities Strategy Evaluation14.7Levels of the Evaluation That Provided Opportunities to Address Emergent Issues15.1Methods and Techniques for the Three Components of Causal Analysis15.2Interpreting Findings in Terms of Their Congruence with Program Theory15.3Pattern of Results Suggesting Implementation Failure15.4Pattern of Results Suggesting Engagement or Adherence Failure15.5Pattern of Results Suggesting Theory Failure: Intermediate Outcome15.6Pattern of Results Suggesting Theory Failure: Final Outcome15.7Pattern of Results Suggesting Theory Failure: Alternative Causal PathExhibits6.1Questions for Drawing Out Program Theories6.2Some Challenges When Developing a Program Theory7.1Arguments for Including in the Scope of the Program Some Outcomes That Are Not Its Focus7.2Steps for Developing an Outcomes Chain7.3Some Key Points About Outcomes Chains8.1Useful Features for Inclusion in the Theory of Action9.1Guidelines for Developing Effective Logic Models11.1Examples of Theories of Change11.2Examples of Blending Various Aspects of Different Theories14.1Choosing Which Parts of the Program Theory to Measure Routinely14.2Some Suggestions for Choosing Which Outcomes in the Outcomes Chain to Measure16.1Using an Outcomes Chain as a Structure for an Oral Report: An Evaluation of Road Safety Presentations by Police to High School Students

ACKNOWLEDGMENTS

MANY PEOPLE HAVE helped us along the journey to this book.

Our work builds on the contributions of the pioneers and innovators in program theory evaluation and those whose work underpins key concepts in program theory. The work of Dan Stufflebeam, Carol Weiss, Joseph Wholey, Claude Bennett, Len Bickman, Carol Fitz-Gibbon, and Lynn Morris laid the foundations for program theory. Benjamin Bloom, Thomas Hastings, and George Madaus’s taxonomy of educational objectives, when linked with Bennett’s Hierarchy, gave rise to the notions of outcome chains, and Tom Hastings’s important article, “Curriculum Evaluation: The Why of the Outcomes,” expanded thinking beyond a focus on outcomes. Bryan Lenne of the New South Wales Program Evaluation Unit contributed significantly to the development of the matrix approach to program theory and the classification of program archetypes. More recently, we have learned much from the work of Michael Patton, Rick Davies, and Boru Douthwaite exploring the use of network theory and systems approaches; the development of realist evaluation and realist synthesis by Ray Pawson and Nick Tilley; and outcome mapping by Barry Kibel, Terry Smutlyo, Fred Carden, and Sarah Earl in terms of using program theory for complicated and complex situations and interventions.

Over more than twenty-five years, we have had the good fortune to work with colleagues and clients who have stretched our thinking about program theory. Sue particularly acknowledges those whose work and insights have contributed to the development of the matrix approach: Carolyn Wells for development of the concept of attributes of success as part of program theory matrices; Steve Baxter for his contribution to thinking about managing external risk factors that affect outcomes and the implications of that for developing a program theory; and Larraine Larri for her assistance with portraying and explaining program theory matrices. Sue also acknowledges the contributions of those who helped identify program theories for particular program archetypes: Bryan Lenne for his contribution to the development of the generic theory for advisory programs drawing on the work of Claude Bennett; Ross Homel for his work on regulatory programs and contribution to the development of the sticks archetypal program theory; Harry Hatry’s work on performance incentive programs that contributed to the development of the carrots archetype; Alison Matthews for her case management approach that drew attention to the need to have models that could accommodate different outcomes for different people using different paths, types of service, and activities; Greg Masters for his work on service delivery programs; and Barry Smith for his work on community capacity-building programs. She acknowledges as well the valuable work of Ellen Taylor-Powell, Larry Jones, and Ellen Henert of the University of Wisconsin Extension Center who, through their online course, Enhancing Program Performance with Logic Models, drew her attention to the potential usefulness of research-based theories of change and prompted her in-depth exploration of those theories.

Patricia’s thinking about program theory owes a particular debt to her mentor, Carol Weiss, and colleagues Tim Hacsi, Tracey Huebner, and Anthony Petrosino at the Harvard Project on Schooling and Children, where she completed a postdoctoral fellowship supported by the Spencer Foundation. She is also grateful for the insights about program theory gained from discussions and evaluation projects with Brad Astbury, Fred Carden, Margaret Cargo, Jane Davidson, Rick Davies, Julie Elliott, Gerald Elsworth, Delwyn Goodrick, Irene Guijt, Ernest House, Bron McDonald, Michael Patton, Ray Pawson, Kaye Stevens, Gill Westhorp, Bob Williams, and Jerome Winston. Thanks also to Susie Elliott, Caitlin Nash, and Russell Stanbrough for their help with editing and manuscript preparation.

The book has benefited considerably from advice and encouragement from Andy Pasternack and Seth Schwartz at Jossey-Bass and from the constructive feedback on earlier drafts of the book from Fred Carden, Jane Davidson, Ernest House, Steve Montague, and especially Michael Patton. We bear responsibility for all remaining deficiencies.

Finally, we thank our families for their support and encouragement throughout the writing of this book and the staff and other stakeholders of the many programs from which we drew the experience on which this book is based.

To Peter, my partner and soulmate, for his encouraging, good-humored, and practical support throughout the writing of this book.

—Sue

To Paul, my dear man, for all he does and has done to make this work possible.

—Patricia

THE AUTHORS

SUE C. FUNNELL is a director of Performance Improvement, a company she established in 1992. She has more than thirty-five years of experience in program design, evaluation, and performance measurement. Since the 1980s, she has been one of the key contributors to the development, dissemination, and use of program theory in Australia. She has supported local, state, national, international, and global government and nongovernment organizations in developed and developing countries, and she has successfully used program theory for evaluation, monitoring, planning, and organizational learning. Her program theory work has been with the Joint United Nations Programme on HIV/AIDS, United Nations Center for Human Settlements, World Bank, Australian National Audit Office, Australian Department of Finance, and government agencies and departments in Australia and New Zealand. She has addressed communication technology; education, training, and leadership development; employment; energy and water; environmental protection; evaluation policy; family, youth, and community services, including alcohol and other drugs, disability, housing, mental health, problem gambling, volunteering, and welfare assistance; industry development; legal systems; natural resources management; occupational health and safety; primary industries; and roads. Funnell, a past president of the Australasian Evaluation Society (AES) and now an AES Fellow, has been awarded the AES Evaluation Training and Services Award for outstanding contributions to the profession of evaluation.

PATRICIA J. ROGERS is Professor of Public Sector Evaluation at the Royal Melbourne Institute of Technology, Melbourne, Australia. She has worked in public sector evaluation and research for more than twenty-five years, with government and nongovernment organizations (international, national, state, and local) across a wide range of program areas, including agriculture, community development, criminal justice, early childhood education, health promotion, Indigenous housing, international development, and legal aid. She has worked on projects with the United Nations Development Programme, World Bank Institute, Network of Networks on Impact Evaluation, U.S. Department of Energy, the Office of the Presidency (South Africa), the Public Service Commission (South Africa), and the Development Bank of Southern Africa. She has presented keynote addresses at conferences of the Australasian, Aotearoa/New Zealand, European, United Kingdom, South African, and Swedish evaluation societies and associations and is on the editorial boards of the journals Evaluation and New Directions for Evaluation. She has been awarded the American Evaluation Association’s Myrdal Award for Evaluation Practice, the Australasian Evaluation Society’s Evaluation Training and Services Award for outstanding contributions to the profession of evaluation, the AES Caulley-Tulloch Prize for Pioneering Literature in Evaluation, and (with Sue Funnell) the AES Best Evaluation Study Award.

INTRODUCTION: THE PROMISE AND RISKS OF USING PROGRAM THEORY

THE 1920S ENTREPRENEUR Carl Weeks once wrote, “If you can dream it, you can build it.” This is the key idea that underpins program theory. Having a vision of where we are going and some clarity about how we plan to get there can help us work together to achieve our goals, and learn from both success and failure.

WHAT PROGRAM THEORY IS

A program theory is an explicit theory or model of how an intervention, such as a project, a program, a strategy, an initiative, or a policy, contributes to a chain of intermediate results and finally to the intended or observed outcomes. A program theory ideally has two components: a theory of change and a theory of action. The theory of change is about the central processes or drivers by which change comes about for individuals, groups, or communities—for example, psychological processes, social processes, physical processes, and economic processes. The theory of change could derive from a formal, research-based theory or an unstated, tacit understanding about how things work. For example, the theory of change underpinning some health promotion programs is that changes in perceived social norms lead to behavior changes. The theory of action explains how programs or other interventions are constructed to activate these theories of change. For example, health promotion programs might use peer mentors, advertisements with survey results, or some other strategy to change perceptions of social norms.

Program theory, under all its various labels, including “theories of change,” “logic modeling,” and “intervention logic,” has grown in popularity over the past twenty years or so. Many government and nongovernment organizations across the world now encourage or require its use for planning, monitoring, and evaluating.

When done well, program theory can produce many benefits. It can develop agreement among diverse stakeholders about what they are trying to do and how, or identify where there are legitimately different perspectives. It can help to improve plans by highlighting gaps and opportunities for collaboration with partners. It can help to set realistic objectives. It can support the development of meaningful performance indicators to track progress and report achievements. It can be used to identify where and why unsuccessful programs are failing or what makes successful programs work, and how they might be reproduced or adapted elsewhere. It can provide a framework to bring together information from many sites, many projects, or many evaluations so that it is possible to learn from the past to improve the future.

Program theory, however, is not always done well. And when it is done badly, it misrepresents what an intervention does and what it can achieve. It can lead to monitoring systems and evaluations that produce an incomplete or distorted picture of what is happening and mistaken judgments about what is effective or efficient. It can demotivate staff and deflect attention from what is important to only what can be easily measured. It can silence important voices or fail to touch those who can act on it. It can take up time without adding value.

The promise of good program theory and the risk of bad program theory have motivated us to write this book. Over more than twenty years, we have worked with small and large organizations in countries all over the world; with municipal, state, and federal government agencies, and nongovernment organizations; on tiny local projects, multimillion-dollar national programs, and whole-of-government strategies; with service deliverers, policymakers, and funders; and in many sectors, including health, education, agriculture, justice, infrastructure, natural resources, community services, community development, and emergency management. Over this time, we have seen diverse approaches to program theory.

What we have learned from this experience, and from the expanding library of empirical research on program theory, is that program theory should be developed, represented, and used not in a formulaic way, but thoughtfully and strategically, in ways that suit the particular situation. We call this purposeful program theory.

PURPOSEFUL PROGRAM THEORY

Greek legend tells of the fearsome hotelier Procrustes who would adjust his guests to match the length of his bed, stretching the short and trimming off the legs of the tall. Guides to program theory that are too prescriptive risk creating such a Procrustean bed. When the same approach to program theory is used for all types of interventions and all types of purposes, the risk is that the interventions will be distorted to fit into a preconceived format. Important aspects may be chopped off and ignored, and other aspects may be stretched to fit into preconceived boxes of a factory model, with inputs, processes, outcomes, and impacts.

Purposeful program theory requires thoughtful assessment of circumstances, asking in particular, “Who is going to use the program theory, and for what purposes?” and, “What is the nature of the intervention and the situation in which it is implemented?” It requires a wide repertoire, not a one-size-fits-all approach to program theory.

Purposeful program theory also requires attention to the limitations of any one program theory, which must necessarily be a simplification of reality, and a willingness to revise it as needed to address emerging issues. As the American evaluator Daniel Stufflebeam (2001) has pointed out, evaluators who continue to use an unsuitable program theory are similarly at risk of creating a Procrustean bed for the evaluation.

OVERVIEW OF THE BOOK

The book is designed to help you assess your particular circumstances and develop, represent, and use program theory in appropriate ways. It has options at every stage and examples to help you decide which options to use and how to adapt them to your circumstances. Throughout the book, we draw on examples from our own work and the work of others. (“Our work” refers to projects we have done together and individually.) Each chapter includes exercises to try out new ideas and techniques.

If you are new to program theory, it will be most useful to read the chapters in sequence. If you have some experience or are coming back to the book during an evaluation, you can select the particular chapter you need.

Key Ideas in Program Theory

Part One sets out the key ideas of program theory and how it has developed over time. We explain in Chapter One the essential features of program theory, using the broad policy objective of eating an apple a day to keep the doctor away as an example of how program theory can be used in different ways to learn from success, failure, and mixed results. Chapter Two describes how program theory has developed over time and sorts out the confusion about the different terms that have been used. And Chapter Three introduces seven widespread myths about program theory and seven common traps to avoid.

Assessing Your Circumstances

A key message of this book is the need to approach program theory in a way that suits your circumstances. Therefore, Part Two examines how to analyze the intended uses of program theory and the nature of the situation and intervention.

We explain in Chapter Four why it is important to be clear about who is going to use program theory and for what purposes. A program theory that is useful for developing internal monitoring systems for incremental correction, for example, could be inappropriate for developing performance measures for external accountability. A theory to guide the design of an impact evaluation might not be sufficient to guide a process evaluation that aims to document an unfolding innovation. Being clear about the intended uses of program theory, reviewing this as circumstances change, and considering this when making decisions is an essential part of purposeful program theory.

Chapter Five discusses how to identify simple, complicated, and complex aspects of the program or policy and the situation in which it is being implemented. Program theory can be used for interventions that are simple; that is, they have a single implementing agency and a well-understood causal process that works pretty much the same everywhere. But most interventions have important complicated or complex aspects that program theory needs to address in order not to misrepresent how it works. The implications of complicated and complex aspects of interventions for developing, representing, and using program theory are addressed throughout the book.

Developing and Representing Program Theory

The chapters in Part Three focus on ways of developing and representing program theory.

Chapter Six discusses how to combine three approaches to developing a program theory. A deductive approach focuses on stated policies and procedures and previous research. An inductive approach builds from observing the intervention in action, reviewing previous observations of it, or observing similar interventions. A mental model approach works with stakeholders to articulate their tacit understandings of how the intervention works.

Chapter Seven sets out three steps to develop a program theory. Step 1 is undertaking a situation analysis to identify problems and opportunities and understand the causes and consequences of problems. Step 2 is to decide the program scope: agreeing which aspects of the problem—its causes and consequences—the program will focus on directly and primarily and which will be beyond the direct focus. The more complex the program is, the more fluid the boundaries should be. Step 3 is to articulate an outcomes chain that shows the assumed or hypothesized cause and effect or contingency relationships between immediate and intermediate outcomes and ultimate outcomes or impacts (both short and long term). In this chapter, we address each of these tasks by applying them to an employment program for mature workers, and we provide examples of how these can be done in different ways to suit any situation.

In Chapter Eight, we introduce a structured approach to developing the second part of the program theory, the theory of action, which spells out how the intervention is intended to activate the theory of change. For example, if a program aims to change health behaviors through increasing knowledge of their consequences, will this knowledge be achieved through a public advertising campaign, personal consultations from health professionals, viral marketing from peers, or some other activities? We introduce the program theory matrix: a structured approach that explores systematically the outcomes chain developed in the theory of change. For each of the outcomes in the outcomes chain, the matrix identifies the nature and quantity of program activities that are intended to achieve this and other factors that will affect whether and how well the outcome is achieved. It also defines what success will look like for the outcome. We continue with the example of an employment program for mature workers, introduced in Chapter Seven, to demonstrate the various components of a theory of action.

We look at different types of logic models in Chapter Nine that can be used to represent program theory. Pipeline models show an intervention as a linear series of boxes labeled something like “inputs, processes, outcomes, and impacts.” Outcome chains, which show a series of results leading to the final impacts of interest, have the advantage of being able to represent more complicated and complex interventions where the activities occur throughout the causal chain and are not all present at the beginning of the process. Realist matrices focus on showing how interventions work differently for different groups or in different situations. We discuss what makes a good logic model, do some logic model makeovers, and review some technology for producing these models.

Chapter Ten discusses how to assess the quality of the program theory in terms of its internal coherence and its validity with respect to external considerations. A program theory can be poorly expressed, incompletely expressed, or just plain wrong. It is important to review it systematically during development and periodically throughout its use.

Resources for Developing Program Theory

The chapters in Part Four provide resources to help with the processes of developing and representing program theory. It can be helpful to draw on previous research and planning when developing the outcomes chain.

Chapter Eleven provides information about a number of theories of how change occurs for individuals, organizations, and communities. The theory of reasoned action (Fishbein and Ajzen, 1975), the theory of planned behavior (Ajzen, 1988), and the stages of change theory (Prochaska and DiClemente, 1983) are theories about changing behaviors of individuals. Empowerment theory (Perkins and Zimmerman, 1995) may relate to individuals, groups, or communities. Diffusion theory (E. Rogers, 1995) is largely about changing community behaviors (and behaviors of individuals en masse). Socioecological theory (Bronfenbrenner, 1979) is about mechanisms for change for individuals, families, groups, and communities and the interplay among all of those actors. Network theory (Granovetter, 1973) is about how the relationships, networks, and connections among entities, and not just the characteristics of the entities themselves, affect outcomes. The entities could be individuals, organizations, special issues groups, or even whole countries. There are many other research-based theories of change, and the chapter lists some other potentially relevant theories that could be used as the basis for an intervention’s specific theory of change.

Chapter Twelve outlines some common program archetypes that can be selected, adapted, and combined for particular situations. These include advisory, information, and education programs that seek to change individual behavior by informing decisions; “sticks and carrots,” which work through incentives and sanctions; case management; community capacity development; and direct service delivery.

Chapter Thirteen provides examples of variations on pipeline and outcomes chain logic models.

Using Program Theory for Monitoring and Evaluation

The final part of this book describes how to use program theory specifically for monitoring and evaluation.

Chapter Fourteen explains how to use program theory to identify what aspects of the intervention, the context, and results should be measured and how to use key evaluation questions to focus an evaluation in terms of data collection, analysis, and reporting. Program theory can help to structure a coherent narrative report and a focused analysis, whether reporting the results of a single evaluation or bringing together data from many studies. We provide some suggestions on ways to do this for small and large evaluations.

Even when there is credible evidence that outcomes have occurred, can we be confident that an intervention has caused them or at least contributed to them together with other factors? In recent years there has been a vigorous debate about the suitability of different methods and designs to address the issue of causal analysis. In Chapter Fifteen, we set out a three-part framework for causal analysis when using program theory that can bring to bear the full range of research designs and methods for causal analysis. The starting point is looking for congruence of results with those predicted by program theory. The second part is finding relevant comparisons that indicate the difference that the intervention has made. These can include creating a control group or a comparison group or making other relevant comparisons. The third part is checking out alternative explanations for the results and exceptions to the patterns.

Chapter Sixteen describes ways to bring together information across the different levels of a program theory, or across several interventions that use the same program theory, and how to report this coherently and effectively.

TAKING A STRATEGIC AND ADAPTIVE APPROACH

Program theory can be developed, represented, and used in many ways. Throughout this book, we invite you to take a purposeful approach to program theory, matching it to your situation, checking how it is going, and adapting it as needed to ensure that it contributes to improved interventions and the outcomes you seek.

PART 1

Key Ideas in Program Theory

Chapter 1

The Essence of Program Theory

An apple a day keeps the doctor away—or does it? Thinking about how we would find out if this is true and how we might use those findings shows the value of program theory. In this chapter, we set out the key ideas in program theory and show how program theory can be used to learn from success, failure, and mixed results to improve planning, management, evaluation, and evidence-based policy.

EVALUATION WITHOUT PROGRAM THEORY

Let us imagine that we have implemented a program based on the broad policy objective of an apple a day in order to keep the doctor away. This program, which we dubbed An Apple a Day, involves distributing seven apples a week to each participant. A representation of this program without program theory would simply show the program followed by the intended outcome of improved health (Figure 1.1).

Figure 1.1 An Evaluation of An Apple a Day Without Program Theory

This is what is often referred to as a black box evaluation: one that describes an evaluation that analyzes what goes in and what comes out without information about how things are processed in between.

ORIGINS OF “BLACK BOX”

Different sources have been suggested for the term black box. The current Wikipedia entry for black box traces the term, when used for flight data recorders, to World War II Royal Air Force terminology, when prototypes of new electronic devices were installed in airplanes in metal boxes, painted black to avoid reflections and therefore referred to as black boxes.