Promoting Value in the Public Interest E-Book

• Herausgeber: John Wiley & Sons
• Kategorie: Bildung
• Serie: J-B PE Single Issue (Program) Evaluation
• Sprache: Englisch

Seitenzahl: 268

Veröffentlichungsjahr: 2012

Beschreibung

Do you make evidence-based decisions when designing and conducting evaluations, and use methods validated by experience? Because of the growing importance of evidence-based decision-making for improving programs and policies, this issue examines methods selection: * Which is better? * How can one be improved? * Are the results of the project worth the resources expended? and how that leads to confidence in value-based conclusions. It presents a constructive dialogue on valuing in evaluation, with the goal of developing a pragmatic approach for guiding method choice and for promoting methodology policies that support multiple approaches to valuation being employed in context-appropriate ways so as to serve the public interest. This is the 133rd volume of the Jossey-Bass quarterly report series New Directions for Evaluation, an official publication of the American Evaluation Association.

Leseprobe

Contents

Editor’s Notes

Chapter 1: Managing Valuation

Evaluation as Assisted Valuation

Method of Analysis for Valuing

Method of Synthesis for Valuing

Prefatory Conclusions

Chapter 2: The Logic of Valuing

Preliminaries

Probative Proposal

Chapter 3: The Evaluator’s Role in Valuing: Who and With Whom

Evaluator Contextual Influence on Valuing

Modes of Valuing

Valuing by Stakeholders

Valuing by Evaluator and Stakeholders

Valuing by Evaluator

In Summary

Final Comment

Chapter 4: Step Arounds for Common Pitfalls When Valuing Resources Used Versus Resources Produced

Step Arounds for Pitfalls and Biases in Evaluating Resources Consumed by Programs

Evaluating Program Value as What Is Produced Relative to What Is Consumed

Step Arounds for Pitfalls and Biases in Evaluating Resources Produced by Programs

The Biggest Step Around: Evaluate More Than Resources Consumed and Resources Produced

Conclusions

Chapter 5: The Role of Context in Valuing Federal Programs

GAO’s Role in Supporting Oversight of Federal Programs

Range of GAO Evaluation Questions and Approaches

Evaluation Conclusions Depend on Available Criteria

Problems in Assessing Overall Program Value or Worth

Efforts to Assess Overall Program Value or Worth

Conclusions

Chapter 6: When One Must Go: The Canadian Experience With Strategic Review and Judging Program Value

Understanding the Strategic Review

The Evolving Role of Program Evaluation

Tackling New Evaluation Requirements

Methodological Ramifications

New Stakes

Valuing Programs at the Federal Level—Some Observations

Chapter 7: Valuing, Evaluation Methods, and the Politicization of the Evaluation Process

Agency Involvement With Particular Methods

The Advent of the Single Narrative

The Single Narrative and Its Implications for Methodology

Current Expansion of Efforts to Direct and Control Evaluations

Four Suggestions

Chapter 8: Valuation and the American Evaluation Association: Helping 100 Flowers Bloom, or at Least Be Understood?

Conclusion

Chapter 9: Evaluators in a World of Valuators

What Can I Contribute to the Valuating That Unfolds in the Policy-Making Process?

How Can I Improve My Valuation Competence?

What Should I Not Do?

Conclusion

Chapter 10: Contextual Pragmatics of Valuing

Contributions to Deconstructing Valuing Through Context Sensitivity

The Personal Factor and Concrete Guidance on Becoming More Skilled at Valuing

Author by Author Challenges for Reflection and Professional Development

Chapter 11: Developing Policies to Support Valuing in the Public Interest

Assisted Valuation in the Public Interest

Managing Context-Appropriate Valuation

Implications for Evaluation Policy

Index

Promoting Valuation in the Public Interest: Informing Policies for Judging Value in Evaluation

George Julnes (ed.)

New Directions for Evaluation, no. 133

Sandra Mathison, Editor-in-Chief

Copyright ©2012 Wiley Periodicals, Inc., A Wiley Company, and the American Evaluation Association. All rights reserved. No part of this publication may be reproduced in any form or by any means, except as permitted under sections 107 and 108 of the 1976 United States Copyright Act, without either the prior written permission of the publisher or authorization through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923; [phone]; fax [phone]. The copyright notice appearing at the bottom of the first page of a chapter in this journal indicates the copyright holder’s consent that copies may be made for personal or internal use, or for personal or internal use of specific clients, on the condition that the copier pay for copying beyond that permitted by law. This consent does not extend to other kinds of copying, such as copying for general distribution, for advertising or promotional purposes, for creating collective works, or for resale. Such permission requests and other permission inquiries should be addressed to the Permissions Department, c/o John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030; [phone], fax [phone], www.wiley.com/go/permissions.

Microfilm copies of issues and articles are available in 16mm and 35mm, as well as microfiche in 105mm, through University Microfilms Inc., 300 North Zeeb Road, Ann Arbor, MI 48106-1346.

New Directions for Evaluation is indexed in Education Research Complete (EBSCO), ERIC Database (Education Resources Information Center), Higher Education Abstracts (Claremont Graduate University), SCOPUS (Elsevier), Social Services Abstracts (CSA/CIG), Sociological Abstracts (CSA/CIG), and Worldwide Political Science Abstracts (CSA/CIG).

New Directions for Evaluation (ISSN 1097-6736, electronic ISSN 1534-875X) is part of The Jossey-Bass Education Series and is published quarterly by Wiley Subscription Services, Inc., A Wiley Company, at Jossey-Bass, One Montgomery Street, Suite 1200, San Francisco, CA 94104-4594.

Subscriptions cost $89 for U.S./Canada/Mexico; $113 international. For institutions, agencies, and libraries, $295 U.S.; $335 Canada/Mexico; $369 international. Prices subject to change.

Editorial correspondence should be addressed to the Editor-in-Chief, Sandra Mathison, University of British Columbia, 2125 Main Mall, Vancouver, BC V6T 1Z4, Canada.

www.josseybass.com

New Directions for Evaluation

Sponsored by the American Evaluation Association

Editor-in-Chief

Sandra MathisonUniversity of British Columbia

Associate Editors

Saville KushnerUniversity of the West of EnglandPatrick McKnightGeorge Mason UniversityPatricia RogersRoyal Melbourne Institute of Technology

Editorial Advisory Board

Michael BambergerIndependent consultantGail BarringtonBarrington Research Group Inc.Nicole BowmanBowman ConsultingHuey ChenUniversity of Alabama at BirminghamLois-ellin DattaDatta AnalysisStewart I. DonaldsonClaremont Graduate UniversityMichael DuttweilerCornell UniversityJody FitzpatrickUniversity of Colorado at DenverGary HenryUniversity of North Carolina, Chapel HillStafford HoodArizona State UniversityGeorge JulnesUtah State UniversityJean KingUniversity of MinnesotaNancy KingsburyU.S. Government Accountability OfficeHenry M. LevinTeachers College, Columbia UniversityLaura LevitonRobert Wood Johnson FoundationRichard LightHarvard UniversityLinda MabryWashington State University, VancouverCheryl MacNeilSage CollegeAnna MadisonUniversity of Massachusetts, BostonMelvin M. MarkThe Pennsylvania State UniversityDonna MertensGallaudet UniversityRakesh MohanIdaho State LegislatureMichael MorrisUniversity of New HavenRosalie T. TorresTorres Consulting GroupElizabeth WhitmoreCarleton UniversityMaria Defino WhitsettAustin Independent School DistrictBob WilliamsIndependent consultantDavid B. WilsonUniversity of Maryland, College ParkNancy C. ZajanoLearning Point Associates

Editorial Policy and Procedures

New Directions for Evaluation, a quarterly sourcebook, is an official publication of the American Evaluation Association. The journal publishes empirical, methodological, and theoretical works on all aspects of evaluation. A reflective approach to evaluation is an essential strand to be woven through every issue. The editors encourage issues that have one of three foci: (1) craft issues that present approaches, methods, or techniques that can be applied in evaluation practice, such as the use of templates, case studies, or survey research; (2) professional issues that present topics of import for the field of evaluation, such as utilization of evaluation or locus of evaluation capacity; (3) societal issues that draw out the implications of intellectual, social, or cultural developments for the field of evaluation, such as the women’s movement, communitarianism, or multiculturalism. A wide range of substantive domains is appropriate for New Directions for Evaluation; however, the domains must be of interest to a large audience within the field of evaluation. We encourage a diversity of perspectives and experiences within each issue, as well as creative bridges between evaluation and other sectors of our collective lives.

The editors do not consider or publish unsolicited single manuscripts. Each issue of the journal is devoted to a single topic, with contributions solicited, organized, reviewed, and edited by a guest editor. Issues may take any of several forms, such as a series of related chapters, a debate, or a long article followed by brief critical commentaries. In all cases, the proposals must follow a specific format, which can be obtained from the editor-in-chief. These proposals are sent to members of the editorial board and to relevant substantive experts for peer review. The process may result in acceptance, a recommendation to revise and resubmit, or rejection. However, the editors are committed to working constructively with potential guest editors to help them develop acceptable proposals.

Sandra Mathison, Editor-in-Chief

University of British Columbia

2125 Main Mall

Vancouver, BC V6T 1Z4

CANADA

e-mail: [email protected]

Editor’s Notes1

This issue of New Directions for Evaluation is the product of presentations and sessions I organized at the American Evaluation Association conferences in 2007, 2009, and 2011. The impetus for this work is the recognition that there is little consensus in the evaluation community regarding a critical aspect of what we do—what methods should be used to help judge the value of public programs and policies? We make such value judgments whenever we suggest ways to improve a program or conclude that one policy is better than another.

This current lack of consensus has been both useful and limiting. Its usefulness has stemmed from the resulting flowering of diverse approaches to valuing. However, this default encouragement of diverse methods becomes more problematic if government and foundation funding begins prioritizing particular valuing methods as best suited for their decision-making needs. For example, if a lack of a strong, systematic stance on valuing methods within the evaluation community results in a more organized paradigm, such as economic valuation, becoming more visible and preferred for some funded evaluations, proponents and practitioners of other valuing traditions will feel marginalized and less able to contribute. Not only will this be unfortunate for some evaluators, it will likely also result in a distorted understanding of the public interest and a diminished capacity for evaluation in general to serve this interest.

The evaluation community has been through this before, recently in response to the priority given in government-funded projects for random-assignment experimental designs. To vaccinate ourselves even somewhat from the negative consequences of a conflict on valuing methods, this issue seeks to promote a dialogue on which of our valuing methods appear most promising and which works best with whom and under what circumstances. The goal is for dialogue to lead to some working consensus on multiple approaches to valuing that would help the evaluation community to speak with a stronger voice concerning the need to match different valuing methods with their appropriate contexts.

The issue begins with chapters by Michael Scriven, Marvin C. Alkin and coauthors, Brian T. Yates, and me that set the context for current tensions in understanding how we value alternatives. These background chapters are followed by two chapters on government evaluation policies, one addressing the methods used at the U.S. Government Accountability Office by Stephanie Shipman, and the other by François Dumaine, providing a contrasting look at recently changing evaluation expectations in the Canadian federal government as a result of its Strategic Review initiative.

The issue ends with several chapters that consider the current theory and research on improving the use of our multiple methods of valuing. Specifically, Eleanor Chelimsky, Michael Morris, George F. Grob, Michael Quinn Patton, and I explore the implications of the ideas in this issue for developing policies that promote context-appropriate valuing in evaluation. I hope the issue promotes constructive dialogues on valuing, to which I also hope you will contribute your views and thoughts.

George Julnes

Editor

1I wish to acknowledge the contributions of anonymous reviewers, Sandra Mathison, and Debra Rog in providing feedback on the direction and content of this issue. In addition, Debra Rog and Michael Morris helped me give feedback to other contributors to this issue. As always, remaining flaws are mine.

George Julnes is a professor in the School of Public and International Affairs, University of Baltimore, and his work focuses on evaluation methods, particularly for programs targeting disadvantaged and at-risk populations, and also on supporting government policies on evaluation that promote context-appropriate methodologies.

Julnes, G. (2012). Managing valuation. In G. Julnes (Ed.), Promoting valuation in the public interest: Informing policies for judging value in evaluation. New Directions for Evaluation, 133, 3–15.

Chapter 1

Managing Valuation1

George Julnes

Abstract

This chapter introduces some of the challenges in valuing programs and policies that seek to serve the public interest. After examining variations in current practice, the author addresses ways of managing the use of valuing methodologies according to the contexts in which they work best. This, in turn, supports conclusions about desired types of balance in our methods of valuing in the field of evaluation. ©Wiley Periodicals, Inc., and the American Evaluation Association.

Evaluation can be defined simply as “determining the merit, worth, or value of things” (Scriven, 1991, p. vii) or with more detail as “the systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit or implicit standards, as a means of contributing to the improvement of the program or policy” (Weiss, 1998, p. 4). Either way, an enterprise so defined exists to inform real-world decisions and requires being able to support warranted conclusions about the value of what is being evaluated. That is, making real decisions depends on reaching conclusions about whether, for example, an educational program is good, is better than another, is worth the resources required or risks involved, or could be improved with specified changes. There is, however, little consensus in the evaluation community regarding how we best support these needed conclusions.

This lack of consensus on methods of valuing is becoming more problematic now that (a) evaluation is becoming more central to public-sector decision making and (b) the increasing pressure for evidence-based governance is pushing for more evidence-based, and hence systematic, policies on the methods of valuing appropriate for evaluation, often privileging specific approaches to assessing performance and economic impacts. Some consequences of this pressure for more systematic approaches to valuing are good—evaluators have often been unreflective, and even sloppy, in their approaches to valuing. However, efforts to be systematic often have unintended effects, including premature constriction of useful diversity and a general inflexibility in crafting methodologies to match the needs of specific contexts (Julnes & Rog, 2007; Stake et al., 1997). To respond constructively to pressures for systematization while maintaining the needed flexibility to employ context-appropriate methods of valuing, the evaluation community needs to be proactive in articulating its own working consensus on this issue. This chapter introduces some of the challenges and alternatives to be considered in developing this consensus.

Evaluation as Assisted Valuation

To understand valuing in evaluation is to understand the methods by which we assist our natural abilities to judge the value of alternatives. An example of unassisted valuing could be a common decision where we might prefer a meal that is cheap, tasty, healthy, and convenient. Typically, none of the available options is the best on all four of these criteria, meaning there are pros and cons for each alternative, but we, nonetheless, manage without formal methods to choose what we deem is best. This natural, everyday valuation can be complex but is generally nonproblematic, because the consequences of nonoptimal decisions are minor (Henry & Julnes, 1998).

Essentials of Assisted Valuing

In contrast, when the consequences of a poor decision loom larger, we are often inclined to organize our understanding of the consequences so as to consider the many implications together. That is, we try to assist our natural capacities of valuation, as suggested in a letter from Benjamin Franklin to Joseph Priestley, written September 19, 1772.

Dear Sir,

In the Affair of so much Importance to you, wherein you ask my Advice, I cannot for want of sufficient Premises, advise you what to determine, but if you please I will tell you how . . . divide half a Sheet of Paper by a Line into two Columns, writing over one Pro, and over the other Con. . . . If I judge some two Reasons con equal to some three Reasons pro, I strike out the five; and thus proceeding I find at length where the Balance lies. . . . And tho’ the Weight of Reasons cannot be taken with the Precision of Algebraic Quantities, yet when each is thus considered separately and comparatively, and the whole lies before me, I think I can judge better, and am less likely to take a rash Step. (Labaree & Bell, 1956)

Franklin’s simple explanation addresses most of the essential issues for the current dialogue on valuing policies and programs. First, Franklin acknowledges that the value of his method does not depend on its yielding some exact “Algebraic” answer, only that it reduces the likelihood of our taking “a rash Step.” This recognition is particularly relevant for our claim that evaluation can yield warranted valuative conclusions. Second, we can see that Franklin’s method is based on analyzing the situation (distinguishing important elements, wherein “each is thus considered separately”) and then (after eliminating any combinations of pros and cons that seem to cancel each other out) synthesizing the elements (combining them in some way, so that “the whole lies before me” and I can see “where the Balance lies”) to reach an overall judgment. Significantly, every method used by evaluators for assisting the process of valuation presumes the value of this two-stage approach, though the choices of how to analyze and how to synthesize are points of dispute. After first addressing the warrantability of valuative judgments, these analysis and synthesis stages are addressed in the following sections.

Possibility of Warranted Valuation

The goal of all of these methods of assisted valuation is to yield meaningful judgments of value that support effective decision making. Is it possible for judgments about the value of alternatives to be warranted in some meaningful sense, and, if so, how do we support warranted valuation for actions affecting complex societies? For conclusions to be warranted means that there are strong reasons for accepting the claims. The unease with accepting that judgments of value can be warranted stems in part from a long tradition in philosophy and social science of accepting what is called the fact–value dichotomy, the idea that facts (e.g., “the student average on a the test is 86”) can be objectively verified and validated, whereas values and value judgments (“this curriculum is better than that one”) are subjective and so incapable of validation. As a dichotomy, there is no blurring of lines between these two types of assertions—objective, factual conclusions do not depend at all on value judgments and value judgments become strictly a matter of personal preference. If this dichotomy is accepted, if one judgment about value is not, and cannot, be viewed as better than another, then why prefer one methodology for valuing over another?

Fortunately, philosophers and social scientists have largely rejected this dichotomy. Putnam (2002) allows for a distinction between most facts and value judgments (there is at least some difference between typical fact and value claims) without accepting a strict dichotomy of the two. Similarly, Hurley (1989) has argued that even if there is no “one best way” of valuing, it is possible for specific value judgments to be warranted. Scriven (this issue) adds to this legitimization of valuing by developing an account of the grounds for concluding that judgments of value are warranted. For social scientists, accepting the warrantability of value judgments depends not on objective truth but on the more pragmatic goal of assisted valuing being able to yield specific judgments about policy or program improvements that can be accepted as actionable as a warranted guide for actions (Julnes & Rog, 2008). If the goal of actionable evidence is to avoid rash actions, then valuation methodologies such as benefit-cost analysis are, indeed, “most plausibly justified on cognitive grounds—as a way of counteracting predictable problems in individual and social cognition” (Sunstein, 2001, p. 233).

Method of Analysis for Valuing

As seen in Franklin’s (Labaree & Bell, 1956) distinguishing individual pros and cons, methods of assisted valuing are based on analyzing things in terms of multiple attributes. Further, the attributes distinguished are presumed to be important and capable of evaluative judgment distinct from judgments of other attributes. With Scriven’s term evaluand used for whatever is being evaluated, the first three steps of his four-step logic highlight one analytic approach (quoted from Fournier, 1995, p. 16; the fourth step involves synthesis, the focus of the next section; see Stake et al., 1997, for an alternative view):

1. Establishing the criteria of merit. On what dimensions must the evaluand do well?

2. Constructing standards. How well should the evaluand perform?

3. Measuring performance and comparing with standards. How well did the evaluand perform?

4. Synthesizing and integrating data into a judgment of merit or worth. What is the merit or worth of the evaluand?

Selecting Criteria for Valuing

The analytic approach to valuing requires distinguishing and selecting the criteria, sometimes thought of as dimensions, on which the thing, the evaluand, is to be judged. For this there are two main sources of criteria, with one justification founded on prescriptive values, claims of what should be important based on tradition or authority, and the other on descriptive representation of the expressed values of stakeholders (Shadish, Cook, & Leviton, 1991).

Prescriptive approaches to selecting criteria.

Perhaps the most straightforward approach to selecting the valuing criteria to be used in evaluation is to consider the objectives of the policies or programs in question. Although an overreliance on formal objectives has been criticized for taking attention away from other impacts (Scriven, 1993), this approach has a long history and is commonly used by the U.S. Government Accountability Office (GAO) (Shipman, this issue) in valuing both implementation and effectiveness. One obvious advantage to this approach is that the expected activities and outcomes written into legislation and regulations are fairly objective and easy to defend by GAO analysts.

Less official but still traditional is to judge public policy in terms of two prescriptive values, efficiency and equality (Berlin, 1998; Okun, 1975); although improving outcomes regarding both is good, there is generally a trade-off that complicates valuing. Further complicating the trade-offs, freedom, community, security, and privacy are also often promoted as important criteria in valuing public-sector initiatives (Bardach, 2012). Thus, a challenge for prescriptive approaches is justifying which prescriptive values to address with criteria.

In addition to these outcome-based values, many (e.g., Greene, 1994; Morris & Jacobs, 2000) have argued that evaluators need to be concerned with process issues as well. For example, in addition to promoting efficiency and equality, does a program treat participants with dignity, and is a policy consistent with the cultural values of the range of citizens affected? These process issues are often the focus of program advisory boards and so are balanced with the dominant consequentialist stance in evaluation.

Descriptive approaches to selecting criteria.

Alternatively, criteria can be selected based on stakeholder input. In addition to many methods, there are whole fields of debate around how best to find out what people really value. Surveys or focus groups can be used to identify, or describe, citizen preferences that can then be used to value alternatives. For example, an approach called contingent valuation uses surveys to develop ratings and rankings of respondent preferences as input for economic analyses (Mitchell, 1989). Alternatively, Stake et al. (1997) recommend in-depth discussions with stakeholders to understand their values. Another tradition extends this emphasis on explication by using a cultural lens to emphasize selected values (e.g., emancipation) and inherent value conflicts, such as the conflict that occurs when evaluation is used as a political tool to impose one set of values on more marginalized groups (Fetterman, 2001). A challenge for descriptive approaches is interpreting the reported values. Are some reported values needs, whereas others are merely preferences that have reduced claims on public action? What if different descriptive approaches, such as surveys and dialogue-based focus groups, yield different values?

Valuing on Selected Criteria

The second and third of Scriven’s four steps of valuing involve judgments about the quality of performance on the selected criteria, referencing the “explicit or implicit standards” noted in the Weiss quote above. When criteria are specified in legislation or regulations, there are often associated standards for rating, or grading, performance, as in saying that 80% of the people eligible for services should receive them or that 70% of students should pass a standardized test. As in the ratings on criteria in Consumer Reports, meeting expectations is rated as good; exceeding them is even better.

In the absence of published standards, establishing credible standards is difficult. For example, in a period of high national unemployment, what size of decreased unemployment would warrant judging a responsible policy as “good”? One way to avoid dependence on agreed-upon standards for ratings is to rank the alternatives on the selected criteria. Another approach is to represent performance on criteria in dollar amounts, as when the value of a high school retention project is estimated by calculating the increased lifetime earnings of people who receive a high school diploma. This economic approach to representing values has advantages in appearing objective in the sense of standardized measurement and in matching the financial concerns of many decision makers (Gramlich, 1990; Yates, this issue). However, economic approaches, such as benefit-cost analysis, have their own methodological challenges, reminding us that benefit-cost analysis is an appropriate valuation method in some contexts but should not be supported universally and uncritically by evaluators.

Method of Synthesis for Valuing

It is perhaps telling that the synthesis step in valuing—pulling together the various analytic findings into an overall judgment—is allotted only one of the four steps in Scriven’s logic but is nonetheless the source of the most serious challenges. Indeed, as Scriven contends (1993, p. 72), “‘Pulling it all together’ is where most evaluations fall apart.”

Two Challenges for Synthesis

Scriven (1993) and others identify two fundamental challenges for synthesis in valuing—aggregating across multiple criteria and aggregating across many people for public sector initiatives.

Aggregating across multiple criteria.

If the alternatives we are considering have contrasting advantages (e.g., one program is a little more effective while the other is substantially less expensive), how do we combine these differing aspects into an overall valuation? Franklin explains his method: “If I judge some two Reasons con equal to some three Reasons pro, I strike out the five.” This canceling out of pros and cons (akin to benefits and costs canceling out, referred to as a wash, in benefit-cost analysis) simplifies complexity and allows greater focus on what is left (so that “the whole lies before me”) to support overall, or aggregate, conclusions. However, there is no consensus on how this aggregation is best done (e.g., Scriven, 1993, p. 72, judges the standard numerical weight-and-sum method to be “invalid”), leaving us to “reckon how to weight opposing values” (Bardach, 2012, p. 37).

Aggregating across individuals.

As difficult as it is for individuals to balance competing criteria in judging value, it is even more difficult in valuing policies and programs that purport to serve “public interest.” In the middle of the last century economists sought to finesse this difficulty by developing a general method for valuation that would begin with the different preferences of individuals and yield a “social welfare function,” based on aggregated rankings of alternatives that provided a straightforward, linear ordering of social preferences. However, Kenneth Arrow demonstrated that even in a simple situation of three alternatives and three stakeholders, there often is no objectively preferable choice: “For any method of deriving social choices by aggregating individual preference patterns which satisfies certain natural conditions, it is possible to find individual preference patterns which give rise to a social choice patterning which is not a linear ordering” (Arrow, 1951, p. 330).

Methods of Aggregation

The recognized failure to develop a technical economic approach to aggregated public valuation has not impeded the use of other approaches. The examples provided below show that they differ both in the degree of aggregation and how it is accomplished.

Minimal aggregation.

One alternative approach to aggregation is that the “political process takes care of it” (Bardach, 2012, p. 37) so evaluators need only provide impartial background information. For example, GAO analyses generally do not aggregate findings in support of a single summative conclusion, instead often documenting performance separately for multiple groups or on multiple criteria established by regulation or legislation.

Checklist approaches to aggregation.

For situations in which summative judgments are deemed appropriate and necessary, evaluators commonly make use of checklists. The task here is to identify the most important criteria for judging value, establish standards, display performance in achieving those standards, as Consumer Reports does with its ratings on a few columns of criteria, and reach qualitative or quantitative syntheses of rankings and ratings accordingly. Stufflebeam (2001) used the checklist method to reach overall value judgments about evaluation approaches. Note, however, that checklists are limited in registering and weighting different impacts for different people.

Quantitative aggregation.

A third approach to summative conclusions is to aggregate using formal quantitative calculations. For example, Consumer Reports not only provides checklists but also generally provides an overall score that is derived from weighting the performance on the identified criteria, ranging from a perfect score of 100 to a perfectly bad score of 0. Multiattribute utility (MAU; Edwards & Newman, 1982) analysis uses stakeholder input to value alternatives on different criteria and weight the criteria to yield a stakeholder-based quantitative aggregation. However, the most dominant quantitative approach to valuing in policy evaluation is economic valuation, which represents the major benefits and costs with the use of a single metric, usually money, that allows aggregation to a single value, as in net present value (Adler & Posner, 2000). Because this approach is generally used to aggregate across many people, it highlights the problem of weighting—should everyone be treated equally so that a $1.10 benefit for a rich person justifies imposing a $1 loss on a poor person?

Social aggregation.

A stark contrast to quantitative synthesis, evaluators like Ernie House (House & Howe, 1999) and Robert Stake (Stake et al., 1997) promote a social approach to value synthesis. For House, democratic deliberation is a social process that explicates values and supports judgments. Stake makes use of triangulation and other standard qualitative methods to support disciplined intuition, noting that “we must use intuition as well as analysis to understand the larger picture” (Stake et al., 1997, p. 96). In this perspective, iterative interactions and intuition form the basis of summative judgment.

Prefatory Conclusions

This introductory chapter opened with a call for a more proactive debate in the evaluation community on balancing pressures to be more systematic in using preferred methods of valuing with the need for flexibility in using methods best in specific contexts. This final section addresses the territory of context-aligned valuing and the types of balance among approaches that this issue on valuing, as a whole, needs to support.

Aligning Assisted Valuation With Context

To help structure our thinking about how context should influence methods of valuing in evaluation, Figure 1.1 presents a flow in which (a) various contextual factors result in (b) different understandings of the problems to be addressed through evaluation and the information and process needs of the decisions to be supported. These evaluation needs, in turn, lead to (c) a particular focus for the evaluation and, hence, (d) the methods selected and employed. Elements of evaluation design are addressed elsewhere in this issue, but in reading the chapters that follow, consider the contextual factors that influence the information needs, valuative needs, and social-process needs of particular evaluations. These three categories of needs are not independent, but rather build on each other, with information needs depending on addressing the valuative needs, which in turn presuppose an understanding of the social process that evaluation is intended to serve.

Figure 1.1. Relating Evaluation Context to Evaluation Design for Valuing

Contextual influences on information needs.

Many evaluators follow Patton (2008) in planning evaluations around the information needs of selected stakeholders. Organizational settings (government versus nonprofit; executive versus legislative) and cultural differences (Kirkhart, LaFrance, & Nichols, 2011) influence the nature of these information needs, and so do stakeholder characteristics. Consider Scriven’s (1993) delineation of three types of information that an evaluator might provide: (a) reports the outcomes and causal impacts of a program (“just the facts”) and the principal stakeholders use this information to judge the value of the alternatives, (b) offers conclusions about the value of alternatives being considered, and (c) recommends specific actions (e.g., to fund or not fund the program). In Scriven’s view, the evaluator (almost) never understands enough about the context to recommend actions but, to avoid dereliction of duty, must be the “valuator” who reaches conclusions about the overall value of programs or improvements to them.