Meta-Analysis for Public Management and Policy E-Book

Evan Ringquist

• Herausgeber: John Wiley & Sons
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 965

Veröffentlichungsjahr: 2013

Beschreibung

Meta-Analysis for Public Management and Policy is a groundbreaking book that includes a proven set of tools for making sense of mountains of sometimes inconsistent conclusions from original research. The tools of meta-analysis can help to improve scholarship, ensure more accurate tests of theories, provide clearer and more authoritative advice for policy and management, and ultimately contribute to the wider knowledge base of public management and policy and the social sciences more broadly. This important resource contains an in-depth explanation of the six-stage process for conducting a meta-analysis which consists of Scoping, Literature Search, Data Coding, Calculating and Combining Effect Sizes, Explaining Differences in Effect Sizes Across Original Studies, and Identifying Areas for Further Research. The text includes detailed explanations of the statistical approaches to meta-analysis that have been found to be most useful to researchers and practitioners in public management and policy, and offers four original meta-analyses of school vouchers, performance measurement, public housing decentralization, and public service motivation. These original studies (conducted by the author and his team) offer step-by-step templates for how to conduct a meta-analysis while also contributing original research on important issues in public management and policy. Meta-Analysis for Public Management and Policy is the hands-on resource that can help students and professionals improve the quality and the relevance of research in public management and policy.

Leseprobe

Table of Contents

Title Page

Copyright

Figures, Tables, and Exhibit

Preface

The Audience for the Book

The Plan of the Book

Acknowledgments

The Authors

Introduction: Meta-Analysis for Public Management and Policy

Introducing Meta-Analysis

The Institutionalization of Meta-Analysis

A Role for Meta-Analysis in Public Management and Policy

Meta-Analysis: Understanding the Basics

Adapting Meta-Analysis for Public Management and Policy

Conclusion

Part One: Techniques of Meta-Analysis

Chapter One: Conceptualizing Research and Gathering Studies

Conceptualizing a Meta-Analysis

Conducting a Literature Search in Meta-Analysis

Conclusion

Chapter Two: Turning Studies into Data

What to Code

The Process of Coding

Conclusion

Chapter Three: Calculating and Combining Effect Sizes

Calculating Effect Sizes

Fixed Effects and Random Effects Meta-Analysis

Combining Effect Sizes

Conclusion

Chapter Four: Meta-Regression Analysis

Historical Development of Meta-Regression

Designing a Meta-Regression

Fixed Effects Meta-Regression

Random Effects Meta-Regression

Conclusion

Chapter Five: Advanced Meta-Regression for Public Management and Policy

Clustered Robust Estimation in Meta-Regression

Alternatives for Addressing Dependence in Meta-Regression

Conclusion

Chapter Six: Publication Bias

Sources and Consequences of Publication Bias

Identifying Publication Bias

Remedying and Preventing Publication Bias

Conclusion

Chapter Seven: Meta-Analysis in Economics

The FAT-MST-PET-MRA Approach to Meta-Regression

Assessing the FAT as a Test for Publication Bias

Identifying “True” Effects Using the MST

Estimating the Magnitude of “True” Effects Using the PET

Conclusion

Part Two: Meta-Analysis in Public Management and Policy Research

Chapter Eight: Evaluating the Effectiveness of Educational Vouchers

School Choice and Educational Vouchers

The Effectiveness of Educational Vouchers

Designing a Meta-Analysis of Voucher Effects

Searching the Literature

Data Analysis: Synthesis

Meta-Regression Analysis

Conclusion

Chapter Nine: Performance Management in the Public Sector

Origin of Performance Management

Performance Management in the Literature

Literature Search

Data Analysis: Synthesis

Meta-Analytic Regression Analysis

Conclusion

Chapter Ten: The Effects of Federal Poverty Deconcentration Efforts on Economic Self-Sufficiency and Problematic Behaviors

History of Poverty Deconcentration Programs in the United States

Theoretical Foundations of Poverty Deconcentration Policies

Impacts of Poverty Deconcentration Policies on Individual Economic and Behavioral Life Outcomes

Literature Search

Analysis

Meta-Regression Analysis

Conclusion

Chapter Eleven: The Relationship Between Public Service Motivation and Performance

The Importance of a Possible PSM-Performance Relationship

Theoretical Foundations of the PSM-Performance Relationship

Framing the Meta-Analysis of the PSM-Performance Relationship

Data Analysis

Meta-Regression Analysis

Conclusion

Conclusion: Meta-Analysis and the Future of Research in Public Management and Policy

Why Meta-Analysis?

Why Meta-Analysis Now?

Meta-Analysis for Public Management and Policy: An Adaptation

A World of Possibilities: The Future of Meta-Analysis in Public Management and Policy

Appendix A: Coding Sheets

References

Name Index

Subject Index

The Instructor's Guide for Meta-Analysis for Public Management and Policy includes the following:

Microsoft Excel .csv files containing the data necessary to replicate all of the example analyses in the book, including the school voucher meta-analysis in chapter 8

Stata command files that allow readers to replicate all of the example analyses in the book, including the school voucher meta-analyses in chapter 8

Sample syllabus for a graduate seminar in meta-analysis, tailored for public management, public policy, and the social sciences

PowerPoint files that can be used as is or customized to deliver classroom lectures that cover the following:

Introduction to meta-analysis and meta-analysis in public management and policy (effectively, the book introduction plus additional items from chapter 1)

Literature searches for meta-analysis (effectively, chapter 2 from the book)

The calculation and combining of effect sizes in meta-analysis (effectively, chapter 3 from the book)

The basics of meta-regression analysis (effectively, chapter 4 from the book)

Advanced meta-regression for public management and policy (effectively, chapter 5 from the book)

The Instructor's Guide is available free online. If you would like to download and print a copy of the Guide, please visit: www.wiley.com/college/ringquist

Copyright © 2013 by John Wiley & Sons, Inc. All rights reserved.

Cover design by Jeff Puda.

Published by Jossey-Bass

A Wiley Imprint

One Montgomery Street, Suite 1200, San Francisco, CA 94104-4594— www.josseybass.com

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400, fax [phone], or on the Web at www.copyright.com. Requests to the publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, 201-748-6011, fax [phone], or online at www.wiley.com/go/permissions.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. Readers should be aware that Internet Web sites offered as citations and/or sources for further information may have changed or disappeared between the time this was written and when it is read.

Jossey-Bass books and products are available through most bookstores. To contact Jossey-Bass directly call our Customer Care Department within the U.S. at 800-956-7739, outside the U.S. at 317-572-3986, or fax [phone].

Wiley also publishes its books in a variety of electronic formats and by print-on-demand. Some material included with standard print versions of this book may not be included in e-books or in print-on-demand. If the version of this book that you purchased references media such as CD or DVD that was not included in your purchase, you may download this material at http://booksupport.wiley.com. For more information about Wiley products, visit www.wiley.com.

Library of Congress Cataloging-in-Publication Data

Library of Congress Cataloging-in-Publication Data has been applied for and is on file

with the Library of Congress.

ISBN 9781118190135 (pbk.); ISBN 9781118227657 (ebk.); ISBN 9781118240496 (ebk.);

ISBN 9781118265253 (ebk.)

FIRST EDITION

Figures, Tables, and Exhibit

Figures

Preface

It took twenty-five years to write this book. I was introduced to meta-analysis in 1987 at the Inter-University Consortium for Political and Social Research Summer Program at the University of Michigan. At the time, I was intrigued by the power of the idea of meta-analysis—a suite of statistical techniques for drawing generalizable conclusions from a set of sometimes inconsistent original studies. At the same time, I was struck by the fundamental limitations of the techniques, especially when applied to the types of original studies that are the stock in trade of the social sciences. I left Ann Arbor with the impression that meta-analysis was attractive but ultimately not very useful for social scientists, although I vowed to monitor progress in the field.

I returned to meta-analysis in 2000, and have taught graduate-level courses in the field since. The past decade or so has been an exciting time for meta-analysts, as the statistical techniques of meta-analysis have developed rapidly, particularly in the area of meta-regression. These developments, coupled with recent advances in econometrics and the ever-increasing quality of quantitative research in the social sciences, have produced an environment in which now the techniques of meta-analysis can be used effectively to not only synthesize the results from original quantitative research in public management, public policy, and the social sciences, but also account for systematic variation in the conclusions from this original research.

Meta-Analysis for Public Management and Policy was written with the hope that it might encourage public management and policy scholars to rethink how their research can contribute to evidence-based management and policy, and ultimately inform more effective governance. It was also written with the hope that it might encourage scholars in the social sciences generally to think more carefully about (1) the production of cumulative knowledge in their respective disciplines and (2) the production of knowledge that is useable by practitioners. Meta-analysis offers a set of tools to help scholars meet all of these goals.

The Audience for the Book

The primary audience for the book is the next generation of researchers in public management and policy—that is, graduate students in public policy, public affairs, public management, and public administration programs. While some of the material is moderately technical—especially in chapters 5, 6, and 7 —advanced or highly skilled master's students should find the material accessible, as should all Ph.D. students. To make full use of the techniques offered here, students should have had a basic statistics course and a two-semester (or a very good one-semester) course in applied regression analysis and extensions of the general linear model. A second audience for the book is current researchers in public management and policy. This audience includes academics, researchers in government, and analysts working at policy research organizations. Any researcher interested in generating cumulative knowledge to inform decision making will benefit from using the techniques described in the text. A third audience for the book is researchers in the social sciences more generally, particularly in disciplines closely related to public management and policy: political science, economics, and sociology. Scholars in these fields devote somewhat less attention to problem-focused research and the production of knowledge useable by practitioners in government. The research methods and products produced by political scientists, economists, and sociologists, however, are often nearly identical to those of public management and policy scholars. Moreover, the ability to build cumulative knowledge from original studies is no less important for the type of theory-driven basic research that is more common in these fields. That is, meta-analysis is as useful for fostering scientific progress as it is for fostering evidence-based decision making.

The Plan of the Book

The Aim of the Book

This book is written with three aims in mind. First, I wanted to write a textbook that would introduce readers to the design and execution of meta-analysis within the framework of public management, public policy, and the social sciences. Chapters 1, 2, 3, 4, and 6 serve this aim. Apart from some differences in emphasis and presentation consistent with the substantive focus on policy and management, these chapters could appear in any standard meta-analysis textbook.

Second, I wanted to write a book that would integrate the statistics of meta-analysis with advanced techniques in econometrics that are more familiar to social scientists. The goals of this integration are twofold: to offer researchers in public management and policy a set of sophisticated techniques for conducting meta-regression analysis using effect sizes from the types of original studies common in these fields, and to systematically assess the most common approach to meta-regression in the social sciences—the FAT-MST-PET-MRA techniques from economics (Stanley 2001, 2005). Chapter 5 offers readers a set of advanced meta-regression techniques customized for public management and policy, while chapter 7 reports on extensive Monte Carlo analyses of the elements of the FAT-MST-PET-MRA approach. The conclusion from chapter 7 is that researchers should avoid the economics approach to meta-analysis in favor of the more traditional approach developed in the fields of statistics and medical statistics.

Third, I wanted to offer readers a set of original meta-analyses that employ the techniques developed in chapters 1 through 7 to answer important questions in the fields of public management and policy. In chapter 8, Tatyana Guzman, Mary Anderson, and I offer a meta-analysis summarizing the effects of educational vouchers on student academic achievement. In chapter 9, Ed Gerrish and Po-Ju Wu present a meta-analysis of the effects of performance measurement systems on the performance of public organizations. Chapter 10, coauthored by Joe Bolinger and Lanlan Xu, offers a meta-analysis summarizing evidence of the effects from the deconcentration of federal public housing on the life outcomes of recipients of federal housing assistance. In chapter 11, David Warren and Li-Ting Chen provide a meta-analysis of the relationship between public service motivation and the performance of government agencies. We believe that these are the first meta-analyses to address any of these questions.

Empirical Examples

I use two recurring examples in chapters 1 through 7 to illustrate the techniques of meta-analysis. The first is a meta-analysis of the empirical literature on environmental justice. The original article (Ringquist 2005) is one of the twenty-seven meta-analyses published in the fields of public administration and policy since 1980 (see table I.1). The central question in environmental justice research is whether actual or potential environmental risks are distributed disproportionately with respect to race and class. Typical operationalizations of these general questions examine whether polluting facilities are more likely to be located in communities with large percentages of poor or minority residents, and whether levels of pollution are higher in these communities. A spirited debate took place in the academic literature during the 1990s over whether environmental inequities existed. The sample data set used here contains 680 effect sizes from forty-eight original studies measuring race-based environmental inequities. The original article concludes that there is strong evidence of race-based environmental inequities, but that the magnitude of these inequities (that is, the conditional average effect size) is substantively small.

The second empirical example considers a contentious issue in education research and policy—the effects of educational vouchers on student academic achievement. Advocates argue that vouchers will increase student academic achievement either by allowing students to attend higher-performing private schools (the private school effect) or by forcing existing public schools to improve to avoid losing students and revenue (the competition effect). As was the case with the topic of environmental justice, a spirited debate took place in the academic literature during the 1990s and 2000s regarding the effectiveness of vouchers. Working with Mary Anderson and Her Sung Kum, I presented a conference paper in 2002 synthesizing the evidence regarding the effect of vouchers to that point (Ringquist, Anderson, and Kum 2002). At that time, however, I felt that the relevant empirical literature was neither large enough nor mature enough to support a published meta-analysis. I returned to this question in 2010, and the results are presented in chapter 8. In addition, I use the example of educational vouchers in chapters 1 through 7 to illustrate the practical application of the techniques of meta-analysis.

Computational Approach

There are a handful of very good statistical software programs specialized for meta-analysis, including Comprehensive Meta-Analysis, Review Manager (RevMan), and MetaWin. In my experience, these packages have difficulty managing the type of effect sizes and conducting the types of analysis that characterize meta-analysis in public management and policy. Therefore, all of the data management and statistical analysis in the book are conducted using the Stata statistical software package. I chose Stata for three reasons. First, Stata is one of the most common, if not the most common, of the statistical packages used by researchers in public management and policy. Readers are more likely to adopt the techniques in this book if they can be executed within a familiar computing environment. Second, while Stata does not currently include commands for meta-analysis, members of the Stata Users Group (SUG) have developed a handful of very useful meta-analysis ado files that can be downloaded from the Stata website and integrated in Stata versions 10.0 and above. Moreover, Sterne (2009) has collected excellent documentation for these user-written commands. While useful, the SUG meta-analysis ado commands cannot estimate the meta-regression models introduced in chapter 5 and used in chapters 8 through 11. This is not surprising, since these commands were written by researchers in medical statistics. Management and policy scholars work in a different statistical environment from that of most meta-analysts. Thankfully, it is a relatively simple matter to adapt the statistics of meta-analysis to make use of data coded from original studies in public management and policy. The third reason for using Stata is that it is easy to write command files for the statistical routines necessary for estimating the meta-regression models introduced in chapter 5 and the Monte Carlo simulations in chapter 7. The Stata command files and data sets necessary to replicate all of the analyses in chapters 3 through 8 are available on the companion website maintained by Jossey-Bass.

Acknowledgments

My first debt in crafting this book is to my colleague Mary Anderson. Mary believed in this project for ten years, never (publicly) giving up hope that our early work together on meta-analysis would see the light of day. In addition to contributing original material to the introduction, conclusion, and chapter 8, Mary served as counsel to the other members of the research team and edited (and re-edited) all of the chapters in the volume. Just as important, Mary served as a trusted adviser and proved an inexhaustible well of good humor as I struggled to make this book a reality.

My second debt is to the members of the research team that collaborated on the book. Joe Bollinger is a student in the Joint Ph.D. Program in Public Policy at Indiana University; Ed Gerrish, Tatyana Guzman, Dave Warren, and Lanlan Xu are students in the Public Affairs Ph.D. Program at Indiana University; and Li-Ting Chen and Po-Ju Wu are Ph.D. students in the School of Education at Indiana University. Each and every one of these students is an intellectual marathoner. Conducting an original meta-analysis is not for the faint of heart. Though I warned each member of the team that the project would require three times the effort they expected up front, in reality the project demanded even more from them. They met deadlines and produced outstanding content while never uttering (publicly) complaints about the unreasonable expectations placed upon them by the senior author. Weaker colleagues would have cracked. My team barely flinched.

My third debt is to the professionals who helped develop the ideas in the book and prepare the manuscript for publication. Alison Hankey and her team at Jossey-Bass have been absolutely first-rate. Alison believed in the project early on, allowing me to write the book I wanted to write. Her support and firm guidance were invaluable in moving the manuscript to the finish line. Fran Berry, Carolyn Heinrich, and Michael McGuire were far too patient as I droned on about the possibilities of meta-analysis in public management and policy research, serving as sounding boards and providing ideas for original meta-analyses that might be included in the book. The School of Public and Environmental Affairs at Indiana University allowed me to teach the two-semester course in meta-analysis that provided both a proving ground for the ideas in the book and a training ground for the research team. Many schools wouldn't have allowed a faculty member this type of flexibility. But SPEA is a special place. Colleagues at SPEA and elsewhere who reviewed sections of the manuscript also deserve a special note of thanks: J. S. Butler, Joshua Cowen, Brad Heim, Haeil Jung, Jim Perry, Jonathan Plucker, David Reingold, and Justin Ross.

My deepest debt, for this book as for most things in life, is to my wife, Laurie, and daughters, Rachel and Hannah. They showed levels of patience, forbearance, understanding, and support that I didn't deserve as I abandoned my roles as husband and father in favor of my roles as researcher and author. Writing a book adapting the techniques of meta-analysis for public management and policy has been a first-order professional goal, but it came at the opportunity cost of nine months of missed soccer games, dance recitals, movie nights, and bedtime stories. As a reader, I hope you conclude it was worth it.

Evan J. RingquistBloomington, IndianaNovember 2012

The Authors

Evan J. Ringquist is professor and director of the Ph.D. program in public affairs and joint Ph.D. in public policy in the School of Public and Environmental Affairs at Indiana University, where he also holds affiliate appointments in the Department of Political Science and West European Studies. Ringquist received undergraduate degrees in political science, economics, and biology from Moorhead State University, and received M.A. and Ph.D. degrees in political science and an M.S. degree in land resources from the University of Wisconsin-Madison. Ringquist served as the co-editor of the Journal of Policy Analysis and Management and on the editorial boards of Policy Studies Journal, Social Science Quarterly, and State Politics and Policy Quarterly. Ringquist has published three books and more than forty articles and book chapters examining the implementation and effectiveness of environmental policies, the distribution of environmental risks and the contributions of policy decisions to these distributions, democratic influences in policy making, and bureaucratic politics and behavior. Some of this research has been funded by the National Science Foundation, the U.S. Department of Energy, and the German Marshall Fund of the United States. Ringquist has served under contract or as a consultant to the U.S. Environmental Protection Agency, the U.S. National Park Service, and numerous state governments and nonprofit organizations.

Mary R. Anderson is an assistant professor of government and world affairs at the University of Tampa. She holds a B.A. degree in history and political science from the University of Central Florida and M.S. and Ph.D. degrees in political science from Florida State University. Her research interests focus on political decision making. More specifically, she studies political behavior, such as voting and public opinion, with an interest in how psychological factors influence such decisions. Her work has been published in the American Political Science Review, Journal of Politics, Political Behavior, and Political Psychology. She lives in Tampa, Florida, with her husband and three children.

Introduction

Meta-Analysis for Public Management and Policy

Evan J. Ringquist and Mary R. Anderson

“Meta-Analysis is not a fad. It is rooted in the fundamental values of the scientific enterprise: replicability, quantification, causal and correlational analysis. Valuable information is needlessly scattered in individual studies. The ability of social scientists to deliver generalizable answers to basic questions of policy is too serious a concern to allow us to treat research integration lightly. The potential benefits of meta-analysis methods seem enormous”

(Bangert-Drowns 1986, 398).

“We get the impression that meta-analysis is on the brink of becoming a major part of the evidence base underlying important questions about which programs or policies work (or not)”

(Cordray and Morphy 2009, 478).

There are two perspectives on how scientific research generates knowledge. The first views science as a largely solitary enterprise. Individual researchers or research teams, inspired by a brilliant insight, toil tirelessly to nurture that insight until it becomes a singularly influential article, book, or report that improves our understanding of the world. Especially valuable studies might even be seen as “critical tests” in the Poppernian sense that they distinguish between competing theories of physical or social phenomena. In the public mind, at least, our highest scientific honors go to researchers embodying this perspective (for example, the Nobel Prize). The second perspective views science as a cooperative, collective endeavor. Knowledge advances through the accumulation of countless small pieces of evidence offered by countless investigators. As eloquently articulated by Cooper, Hedges, and Valentine (2009, 4), “the moment we are introduced to science we are told it is a cooperative, cumulative enterprise. Like the artisans who construct a building from blueprints, bricks, and mortar, scientists contribute to a common edifice, called knowledge. Theorists provide our blueprints and researchers collect data that are our bricks.” These two perspectives are not mutually exclusive, or even in conflict, and we would guess that most scholars view the research enterprise through both perspectives. We would also assert, however, that most researchers view their own work through the first perspective, implicitly expecting that once new knowledge is generated, the accumulation process will take care of itself.

Vast increases in the number of researchers, the proliferation of research outlets, and the power of information-processing technology have led to exponential growth in the production and dissemination of scientific research. An inevitable consequence of this growth is that the conclusions from scientific research sometimes conflict. We find ourselves in a situation described by Morton Hunt in which “virtually every field of science is now pervaded by a relentless cross fire in which the findings of new studies not only differ from previously established truths but disagree with one another, often vehemently. Our faith that scientists are cooperatively and steadily enlarging their understanding of the world is giving way to doubt as, time and time again, new research assaults existing knowledge” (Hunt 1997, 1).