39,99 €
Mehr erfahren.
- Herausgeber: John Wiley & Sons
- Kategorie: Fachliteratur
- Serie: Measurement and Evaluation Series
- Sprache: Englisch
Data Collection Data Collection is the second of six books in the Measurement and Evaluation Series from Pfeiffer. The proven ROI Methodology--developed by the ROI Institute--provides a practical system for evaluation planning, data collection, data analysis, and reporting. All six books in the series offer the latest tools, most current research, and practical advice for measuring ROI in a variety of settings. Data Collection offers an effective process for collecting data that is essential to the implementation of the ROI Methodology. The authors outline the techniques, processes, and critical issues involved in successful data collection. The book examines the various methods of data collection, including questionnaires, interviews, focus groups, observation, action plans, performance contracts, and monitoring records. Written for evaluators, facilitators, analysts, designers, coordinators, and managers, Data Collection is a valuable guide for collecting data that are adequate in quantity and quality to produce a complete and credible analysis.
Sie lesen das E-Book in den Legimi-Apps auf:
Seitenzahl: 210
Veröffentlichungsjahr: 2016
Ähnliche
Table of Contents
Cover
About This Book
Why This Book Is Important
What This Book Achieves
How This Book Is Organized
The Measurement and Evaluation Series
Introduction to the Measurement and Evaluation Series
The Need
Audience
Flow of the Books
Terminology: Programs, Projects, Solutions
Features
About Pfeiffer
Title
Copyright
Acknowledgments from the Editors
Principles of the ROI Methodology
1 Using Questionnaires and Surveys
Types of Questions
Questionnaire Design Steps
Improving the Response Rate for Questionnaires and Surveys
Final Thoughts
2 Using Tests
Types of Tests
Simulations
Informal Tests
Final Thoughts
3 Using Interviews, Focus Groups, and Observation
Interviews
Focus Groups
Observations
Final Thoughts
4 Using Other Data Collection Methods
Business Performance Monitoring
Action Planning
Performance Contracts
Final Thoughts
5 Measuring Reaction and Planned Action
Why Measure Reaction and Planned Action?
Importance of Other Levels of Evaluation
Areas of Feedback
Data Collection Issues
Uses of Reaction Data
Final Thoughts
6 Measuring Learning and Confidence
Why Measure Learning and Confidence?
Measurement Issues
Administrative Issues
Uses of Learning Data
Final Thoughts
7 Measuring Application and Implementation
Why Measure Application and Implementation?
Challenges
Measurement Issues
Data Collection Methods
Barriers to Application
Uses of Application Data
Final Thoughts
8 Measuring Impact and Consequences
Why Measure Business Impact?
Effective Impact Measures
Data Collection Methods
Final Thoughts
9 Selecting the Proper Data Collection Method
Matching Exercise
Selecting the Appropriate Method for Each Level
Final Thoughts
Index
About the Authors
Pfeiffer Publications Guide
What will you find on pfeiffer.com?
Measurement and Evaluation Series
Special Offer from the ROI Institute
End User License Agreement
List of Tables
Introduction to the Measurement and Evaluation Series
Table I.1 Terms and Applications
2 Using Tests
Table 2.1. Reporting Format for Data from a Criterion-Referenced Test
5 Measuring Reaction and Planned Action
Table 5.1. Noncontent Issues Versus Content Issues for a Marketing Event
7 Measuring Application and Implementation
Table 7.1. Examples of Action Areas for Application of Program Learning
8 Measuring Impact and Consequences
Table 8.1. Examples of Hard Data
Table 8.2. Examples of Soft Data
Table 8.3. Typical Impact Measures in ROI Applications
List of Illustrations
Principles of the ROI Methodology
Figure P.1. The ROI Process Model
5 Measuring Reaction and Planned Action
Figure 5.1. Correlations Between Reaction and Application Data
6 Measuring Learning and Confidence
Figure 6.1. Relationship Between Validity and Reliability
Guide
Cover
Table of Contents
Begin Reading
Pages
cover
contents
i
ii
iii
iv
v
vi
vii
viii
ix
x
xi
xii
xxiii
xxiv
xxv
xxvi
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
23
24
25
26
27
28
29
30
31
32
33
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
153
154
155
157
158
159
160
161
162
163
164
About This Book
Why This Book Is Important
This second book in the M&E Series begins with a brief introduction to the ROI process model and the Twelve Guiding Principles. It goes on to explain, in detail, one of the most critical activities in the evaluation process: data collection. Data are collected during every step of the ROI Methodology, and how and when these data are collected are crucial to successful program evaluation. This book provides detailed information about data collection methods and how and when to collect data at each evaluation level.
What This Book Achieves
This guide details how to develop or use each of the following data collection tools:
Surveys
Questionnaires
Tests
Simulations
Interviews
Focus groups
Direct observation
Performance monitoring
Action plans
Performance contracts
In addition, this book explains how to select the appropriate data collection method for any situation and any level of evaluation.
How This Book Is Organized
This book begins by introducing and describing each data collection method and ends by discussing the methods that are best for each evaluation level. Chapter One details how to design effective questionnaires and surveys in order to maximize response rates. It also provides tips for presenting questionnaires to participants in ways that will encourage them to provide valuable data. Chapter Two discusses the major types of tests, including how to develop and administer them. When and how to use simulations to collect data are also explained.
Chapter Three explores how to conduct interviews and focus groups for optimal results, including question development and interview techniques. Guidelines for effective observation are also provided. Chapter Four explains how to use business performance monitoring, action planning, and performance contracts to collect data.
Chapter Five focuses on measuring reaction and planned action and the importance of gathering participant feedback. Chapter Six discusses the reasons for measuring learning data and then describes how to use the data. Measurement and administrative issues involved in data collection at this level are also explored.
Chapter Seven explains the importance of collecting application and implementation data. The challenges and issues of collecting application data are explored, and appropriate data collection methods are also discussed. Chapter Eight explores the critical reasons for collecting impact data. Effective impact measures, appropriate data collection measures, and linking the measures to business needs are all discussed in this chapter.
Finally, Chapter Nine illustrates how to select the appropriate data collection method for each program and each level of evaluation.
The Measurement and Evaluation Series
Editors
Patricia Pulliam Phillips, Ph.D.
Jack J. Phillips, Ph.D.
Introduction to the Measurement and Evaluation Series
The ROI Six Pack provides detailed information on developing ROI evaluations, implementing the ROI Methodology, and showing the value of a variety of functions and processes. With detailed examples, tools, templates, shortcuts, and checklists, this series will be a valuable reference for individuals interested in using the ROI Methodology to show the impact of their projects, programs, and processes.
The Need
Although financial ROI has been measured for over one hundred years to quantify the value of plants, equipment, and companies, the concept has only recently been applied to evaluate the impact of learning and development, human resources, technology, quality, marketing, and other support functions. In the learning and development field alone, the use of ROI has become routine in many organizations. In the past decade, hundreds of organizations have embraced the ROI process to show the impact of many different projects and programs.
Along the way, professionals and practitioners need help. They need tools, templates, and tips, along with explanations, examples, and details, to make this process work. Without this help, using the ROI Methodology to show the value of projects and programs is difficult. In short, practitioners need shortcuts and proven techniques to minimize the resources required to use this process. Practitioners’ needs have created the need for this series. This series will provide the detail necessary to make the ROI Methodology successful within an organization. For easy reference and use, the books are logically arranged to align with the steps of the ROI Methodology.
Audience
The principal audience for these books is individuals who plan to use the ROI Methodology to show the value of their projects and programs. Such individuals are specialists or managers charged with proving the value of their particular project or program. They need detailed information, know-how, and confidence.
A second audience is those who have used the ROI Methodology for some time but want a quick reference with tips and techniques to make ROI implementation more successful within their organization. This series, which explains the evaluation process in detail, will be a valuable reference set for these individuals, regardless of other ROI publications owned.
A third audience is consultants and re-searchers who want to know how to address specific evaluation issues. Three important challenges face individuals as they measure ROI and conduct ROI evaluations: (1) collecting post-program data, (2) isolating the effects of the program, and (3) converting data to monetary values. A book is devoted to each of these critical issues, allowing researchers and consultants to easily find details on each issue.
A fourth audience is those who are curious about the ROI Methodology and its use. The first book in this series focuses specifically on ROI, its use, and how to determine whether it is appropriate for an organization. When interest is piqued, the remaining books provide more detail.
Flow of the Books
The six books are presented in a logical sequence, mirroring the ROI process model. Book one, ROI Fundamentals: Why and When to Measure ROI, presents the basic ROI Methodology and makes the business case for measuring ROI as it explores the benefits and barriers to implementation. It also examines the type of organization best suited for the ROI Methodology and the best time to implement it. Planning for an ROI evaluation is also explored in this book.
Book two, Data Collection: Planning For and Collecting All Types of Data, details data collection by examining the different techniques, processes, and issues involved in this process, with an emphasis on collecting post-program data. It examines the different data collection methods: questionnaires, interviews, focus groups, observation, action plans, performance contracts, and monitoring records.
Book three, Isolation of Results: Defining the Impact of the Program, focuses on the most valuable part of the ROI Methodology and the essential step for ensuring credibility. Recognizing that factors other than the program being measured can influence results, this book shows a variety of ways in which the effects of a program can be isolated from other influences. Techniques include comparison analysis using a control group, trend line analysis and forecasting methods, and expert input from a variety of sources.
Book four, Data Conversion: Calculating the Monetary Benefits, covers perhaps the second toughest challenge of ROI evaluation: placing monetary value on program benefits. To calculate the ROI, data must be converted to money, and Data Conversion shows how this conversion has been accomplished in a variety of organizations. The good news is that standard values are available for many items. When they are not, the book shows different techniques for converting them, ranging from calculating the value from records to seeking experts and searching databases. When data cannot be converted to money credibly and with minimum resources, they are considered intangible. This book explores the range of intangible benefits and the necessary techniques for collecting, analyzing, and recording them.
Book five, Costs and ROI: Evaluating at the Ultimate Level, focuses on costs and ROI. This book shows that all costs must be captured in order to create a fully loaded cost profile. All the costs must be included in order to be conservative and to give the analysis additional credibility. Next, the actual ROI calculation is presented, showing the various assumptions and issues that must be addressed when calculating the ROI. Three different calculations are presented: the benefit-cost ratio, the ROI percentage, and the payback period. The book concludes with several cautions and concerns about the use of ROI and its meaning.
Book six, Communication and Implementation: Sustaining the Practice, explores two important issues. The first issue is reporting the results of an evaluation. This is the final part of the ROI Methodology and is necessary to ensure that audiences have the information they need so that improvement processes can be implemented. A range of techniques is available, including face-to-face meetings, brief reports, one-page summaries, routine communications, massaudience techniques, and electronic media. All are available for reporting evaluation results. The final part of the book focuses on how to sustain the ROI evaluation process: how to use it, keep it going, and make it work in the long term to add value to the organization and, often, to show the value of all the programs and projects within a function or department.
Terminology: Programs, Projects, Solutions
In this series the terms program and project are used to describe many processes that can be evaluated using the ROI Methodology. This is an important issue because readers may vary widely in their perspectives. Individuals involved in technology applications may use the terms system and technology rather than program or project. In public policy, in contrast, the word program is prominent. For a professional meetings and events planner, the word program may not be pertinent, but in human resources, program is often used. Finding one term for all these situations would be difficult. Consequently, the terms program and project are used interchangeably. Table I.1 lists these and other terms that may be used in other contexts.
Table I.1Terms and Applications
Term
Example
Program
Leadership development skills enhancement for senior executives
Project
A reengineering scheme for a plastics division
System
A fully interconnected network for all branches of a bank
Initiative
A faith-based effort to reduce recidivism
Policy
A new preschool plan for disadvantaged citizens
Procedure
A new scheduling arrangement for truck drivers
Event
A golf outing for customers
Meeting
A U.S. Coast Guard conference on innovations
Process
Quality sampling
People
Staff additions in the customer care center
Tool
A new means of selecting hotel staff
Features
Each book in the series takes a straightforward approach to make it understandable, practical, and useful. Checklists are provided, charts are included, templates are presented, and examples are explored. All are intended to show how the ROI Methodology works. The focus of these books is implementing the process and making it successful within an organization. The methodology is based on the work of hundreds of individuals who have made the ROI Methodology a successful evaluation process within their organizations.
About Pfeiffer
Pfeiffer serves the professional development and hands-on resource needs of training and human resource practitioners and gives them products to do their jobs better. We deliver proven ideas and solutions from experts in HR development and HR management, and we offer effective and customizable tools to improve workplace performance. From novice to seasoned professional, Pfeiffer is the source you can trust to make yourself and your organization more successful.
Essential Knowledge Pfeiffer produces insightful, practical, and comprehensive materials on topics that matter the most to training and HR professionals. Our Essential Knowledge resources translate the expertise of seasoned professionals into practical, how-to guidance on critical workplace issues and problems. These resources are supported by case studies, worksheets, and job aids and are frequently supplemented with CD-ROMs, Web sites, and other means of making the content easier to read, understand, and use.
Essential Tools Pfeiffer’s Essential Tools resources save time and expense by offering proven, ready-to-use materials—including exercises, activities, games, instruments, and assessments—for use during a training or team-learning event. These resources are frequently offered in looseleaf or CD-ROM format to facilitate copying and customization of the material.
Pfeiffer also recognizes the remarkable power of new technologies in expanding the reach and effectiveness of training. While e-hype has often created whizbang solutions in search of a problem, we are dedicated to bringing convenience and enhancements to proven training solutions. All our e-tools comply with rigorous functionality standards. The most appropriate technology wrapped around essential content yields the perfect solution for today’s on-the-go trainers and human resource professionals.
Essential resources for training and HR professionals
Data Collection
Planning For and Collecting All Types of Data
Patricia Pulliam Phillips, Ph.D.
Cathy A. Stawarski, Ph.D.
Copyright © 2008 by John Wiley & Sons, Inc. All rights reserved.
Published by PfeifferAn Imprint of Wiley989 Market Street, San Francisco, CA 94103-1741www.pfeiffer.com
No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400, fax 978-646-8600, or on the web at www.copyright.com. Requests to the publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, 201-748-6011, fax 201-748-6008, or online at http://www.wiley.com/go/permissions.
Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.
Readers should be aware that Internet Web sites offered as citations and/or sources for further information may have changed or disappeared between the time this book was written and when it is read.
For additional copies/bulk purchases of this book in the U.S. please contact 800-274-4434.
Pfeiffer books and products are available through most bookstores. To contact Pfeiffer directly call our Customer Care Department within the U.S. at 800-274-4434, outside the U.S. at 317-572-3985, fax 317-572-4002, or visit www.pfeiffer.com.
Pfeiffer also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books.
Library of Congress Cataloging-in-Publication Data
Phillips, Patricia Pulliam.Data collection: planning for and collecting all types of data / Patricia Pulliam Phillips, Ph.D., and Cathy A. Stawarski, Ph.D.p. cm.—(The measurement and evaluation series)Includes bibliographical references and index.ISBN: 978-0-7879-8718-3 (pbk.)1. Project management—Evaluation. 2. Rate of return. 3. Employees—Rating of. 4. Organizational learning—Evaluation. 5. Social sciences—Research—Methodology. I. Stawarski, Cathy A. II. Title.HD69.P75P496 2008001.4’33—dc222007035758
Production Editor: Michael Kay
Editor: Matthew Davis
Editorial Assistant: Julie Rodriguez
Manufacturing Supervisor: Becky Morgan
Acknowledgments from the Editors
From Patti
No project, regardless of its size or scope, is completed without the help and support of others. My sincere thanks go to the staff at Pfeiffer. Their support for this project has been relentless. Matt Davis has been the greatest! It is our pleasure and privilege to work with such a professional and creative group of people.
Thanks also go to my husband, Jack. His unwavering support of my work is always evident. His idea for the series was to provide readers with a practical understanding of the various components of a comprehensive measurement and evaluation process. Thank you, Jack, for another fun opportunity!
From Jack
Many thanks go to the staff who helped make this series a reality. Lori Ditoro did an excellent job of meeting a very tight deadline and delivering a quality manuscript.
Much admiration and thanks go to Patti. She is an astute observer of the ROI Methodology, having observed and learned from hundreds of presentations, consulting assignments, and engagements. In addition, she is an excellent researcher and student of the process, studying how it is developed and how it works. She has become an ROI expert in her own right. Thanks, Patti, for your many contributions. You are a great partner, friend, and spouse.
Principles of the ROI Methodology
The ROI Methodology is a step-by-step tool for evaluating any program, project, or initiative in any organization. Figure P.1 illustrates the ROI process model, which makes a potentially complicated process simple by breaking it into sequential steps. The ROI process model provides a systematic, step-by-step approach to ROI evaluations that helps keep the process manageable, allowing users to address one issue at a time. The model also emphasizes that the ROI Methodology is a logical, systematic process that flows from one step to another and provides a way for evaluators to collect and analyze six types of data.
Applying the model consistently from one program to another is essential for successful evaluation. To aid consistent application of the model, the ROI Methodology is based on twelve Guiding Principles. These principles are necessary for a credible, conservative approach to evaluation through the different levels.
When conducting a higher-level evaluation, collect data at lower levels.
When planning a higher-level evaluation, the previous level of evaluation is not required to be comprehensive.
When collecting and analyzing data, use only the most credible sources.
When analyzing data, select the most conservative alternative for calculations.
Use at least one method to isolate the effects of a project.
If no improvement data are available for a population or from a specific source, assume that little or no improvement has occurred.
Adjust estimates of improvement for potential errors of estimation.
Avoid use of extreme data items and unsupported claims when calculating ROI.
Use only the first year of annual benefits in ROI analysis of short-term solutions.
Fully load all costs of a solution, project, or program when analyzing ROI.
Intangible measures are defined as measures that are purposely not converted to monetary values.
Communicate the results of the ROI Methodology to all key stakeholders.
Figure P.1. The ROI Process Model
1Using Questionnaires and Surveys
Data collection is the first operational part of the ROI process model. Data are collected in different time frames and from different sources. This is the first of four chapters on data collection methods. Collectively, these four chapters will provide a variety of ways to meet any application, budget, or time constraint.
Probably the most common data collection method is the questionnaire (Alreck and Settle, 1995). Ranging from short reaction forms to detailed follow-up tools, questionnaires can be used both to obtain subjective information about participants and to document objective, measurable impact results for an ROI analysis. Because of this versatility, the questionnaire is the preferred method for capturing data at Levels 1, 2, 3, and 4 in some organizations.
A survey is a specific type of questionnaire with several applications in measuring program success. Surveys are used in situations in which only attitudes, beliefs, and opinions are captured; questionnaires are much more flexible, capturing a wide range of data from attitudes to specific improvement statistics. The principles of survey construction and design are similar to those of questionnaire design. This chapter explains how to develop both types of instruments.
Types of Questions
In addition to the types of data sought, the types of questions distinguish surveys from questionnaires. Surveys may elicit yes-or-no responses, if absolute agreement or disagreement is required, or they may solicit a range of responses, often on a five-point scale from “strongly agree” to “strongly disagree.”
A questionnaire may contain any or all of these types of questions:
Open-ended questions
allow unlimited answers. Questions are followed by ample blank space for the responses.
Checklists
provide a list of items, and the participant is asked to check those that apply in the situation.
Two-way questions
limit answers to a pair of alternative responses, such as yes and no.
Multiple-choice questions
