Designing and Constructing Instruments for Social Research and Evaluation E-Book

David Colton

• Herausgeber: John Wiley & Sons
• Kategorie: Fachliteratur
• Serie: Research Methods for the Social Sciences
• Sprache: Englisch

Seitenzahl: 733

Veröffentlichungsjahr: 2015

Beschreibung

Written in easy-to-understand language, this important text provides a systematic and commonsense approach to developing instruments for data collection and analysis. This book can be used by both those who are developing instruments for the first time and those who want to hone their skills, including students, agency personnel, program managers, and researchers.

This book provides a thorough presentation of instrument construction, from conception to development and pre-testing of items, formatting the instrument, administration, and, finally, data management and presentation of the findings. Throughout the book, the authors emphasize how to create an   instrument that will produce trustworthy and accurate data. To that end they have included guidelines for reviewing and revising the questionnaire to enhance validity and reliability. They also show how to work effectively with stakeholders such as instruments designers, decision-makers, agency personnel, clients, and raters or respondents.

Leseprobe

Table of Contents

COVER

TITLE

COPYRIGHT

FIGURES, EXHIBITS, TABLES, AND INSTRUMENTS

Figures

Exhibits

Tables

Instruments

PREFACE: ASKING AND ANSWERING

Feedback

Acknowledgments

THE AUTHORS

PART ONE: CONCEPTS

CHAPTER ONE: INTRODUCTION

Instrumentation

Components of an Instrument

Selecting an Appropriate Instrument

The Process of Instrument Construction

Summary

Instrument 1.A: Illustrating the Parts of a Questionnaire

Instrument 1.B: Medical History Questionnaire

Instrument 1.C: Example of a Checklist

Endnotes

CHAPTER TWO: INSTRUMENTS AND SOCIAL INQUIRY

Instruments and Questionnaires in the Context of Social Science Research

Methods of Inquiry

Methodology and Instruments

The Art of Instrument Construction

Instrument 2.A: A Political Poll

Instrument 2.B: Mental Health Screening Form

CHAPTER THREE: MEASUREMENT

Levels of Measurement

Level of Measurement and Item Construction

Summary

Instrument 3.A: Data Extraction Form

Endnotes

CHAPTER FOUR: INSTRUMENT CONSTRUCTION, VALIDITY, AND RELIABILITY

Validity

Demonstrating Evidence for Validity

Reliability

Methods for Establishing Evidence for Reliability

Interrater and Intrarater Reliability

Validity, Reliability, and Decision Making

Summary

Instrument 4.A: Performance Appraisal

Instrument 4.B: Instructor Evaluation

Endnotes

PART TWO: APPLICATION

CHAPTER FIVE: PURPOSEFUL CREATIVITY

Articulating the Purpose of the Study and the Focus of the Instrument

First Steps in Creating Questionnaire Items

Summary

Instrument 5.A: Employee Questionnaire

Endnotes

CHAPTER SIX: PRETESTING

Where Problems Are Likely to Occur

Initial Pretesting: Focusing the Study

Pretesting During Development

Pilot-Testing the Instrument

Summary

Instrument 6.A: Failure to Pretest

CHAPTER SEVEN: THE STRUCTURE AND FORMAT OF SELECTION ITEMS

Response Alternatives, or Scales

Numerical Scales

Graphic Scales

Likert Response Scale

Summary

Instrument 7.A: A Large-Scale Survey Using Rating Scales

Instrument 7.B: An Assessment Instrument Using Graphic Scales

Endnotes

CHAPTER EIGHT: GUIDELINES FOR WRITING SELECTION ITEMS

Writing Items: Preliminary Considerations

Multicultural Considerations

Guidelines for Rating Scale Development

Summary

Instrument 8.A: Writing Sensitive Questions

Instrument 8.B: Biased Language

Endnotes

CHAPTER NINE: SELECTION ITEMS

Alternative Response Scales

Check-All-That-Apply Response Sets

Dichotomous Response Sets

Rank-Ordered Response Sets

Summary

Instrument 9.A: Records Audit Checklist

Instrument 9.B: A Marketing Survey

CHAPTER TEN: SUPPLY ITEMS

Guidelines for Constructing Supply Items

Describe the Units of Interest Where Necessary

Making Sense of Qualitative Data

Summary

Instrument 10.A: Open-Ended Item Examples and Commentary

Instrument 10.B: Behavioral Assessment

Five Essential Characteristics of Multi-Item Scales

CHAPTER ELEVEN: GUIDELINES FOR CONSTRUCTING MULTI-ITEM SCALES

Scale Construction

Summary

Instrument 11.A: Summative Scale

Endnotes

PART THREE: ORGANIZATION AND ADMINISTRATION

CHAPTER TWELVE: ORGANIZING THE INSTRUMENT

Title

Introductory Statement

Directions

Demographic Section

Organization and Format of the Instrument

Typography and Instrument Design

Summary

Instrument 12.A: Scoring by Scanning

Instrument 12.B: Word Processing Software

Instrument 12.C: Conflict Resolution Skills Assessment

Endnotes

CHAPTER THIRTEEN: ADMINISTERING THE INSTRUMENT

Administering Instruments Completed by a Rater

Administering Self-Report Instruments

Summary

Instrument 13.A: Behavioral Rating Scale

Endnote

CHAPTER FOURTEEN: COMPUTERS AND INSTRUMENT CONSTRUCTION

Item Construction and Questionnaire Organization

Computers and Questionnaire Administration

Summary

Instrument 14.A: Example of a Web Survey

CHAPTER FIFTEEN: MANAGING THE DATA AND REPORTING THE RESULTS

Data Management

Who Owns and Has Access to the Instrument and the Data?

Reporting Your Results

Summary

REFERENCES

INDEX

End User License Agreement

Guide

Cover

Table of Contents

Begin Reading

List of Illustrations

CHAPTER ONE: INTRODUCTION

FIGURE 1.1: CATEGORIES OF SOCIAL SCIENCE INSTRUMENTS

FIGURE 1.2: STEPS IN THE INSTRUMENT CONSTRUCTION PROCESS

CHAPTER EIGHT: GUIDELINES FOR WRITING SELECTION ITEMS

FIGURE 8.1: EXAMPLES OF RESPONSE SETS WRITTEN IN THE SAME DIRECTION

FIGURE 8.2: MATRIX LAYOUT FOR A RATING SCALE

FIGURE 8.3: DIFFICULTIES ASSOCIATED WITH USING ABSTRACT TERMS FOR RESPONSE CHOICES

CHAPTER FIFTEEN: MANAGING THE DATA AND REPORTING THE RESULTS

FIGURE 15.1: EXAMPLES OF DATA ENTRY ERRORS BY RESPONDENTS

List of Tables

CHAPTER TWO: INSTRUMENTS AND SOCIAL INQUIRY

TABLE 2.1: STUDY PLANNING GRID

CHAPTER THREE: MEASUREMENT

TABLE 3.1: LEVELS OF MEASUREMENT

CHAPTER FIVE: PURPOSEFUL CREATIVITY

TABLE 5.1: PROCESSES AND OUTCOMES

TABLE 5.2: Q-SORT DISTRIBUTION

TABLE 5.3: TABLE OF SPECIFICATIONS

CHAPTER ELEVEN: GUIDELINES FOR CONSTRUCTING MULTI-ITEM SCALES

TABLE 11.1: GOAL ATTAINMENT SCALE

TABLE 11.2: GOAL ATTAINMENT SCALE CONVERSION TABLE: CONVERTS GAS SCORES TO STANDARD SCORES

TABLE 11.3: ITEM ANALYSIS

Pages

Cover

Contents

iii

iv

vii

viii

ix

x

xi

xii

xiii

xiv

xv

xvi

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

DESIGNING AND CONSTRUCTING INSTRUMENTS FOR SOCIAL RESEARCH AND EVALUATION

David Colton

Robert W. Covert

Copyright © 2007 by John Wiley & Sons, Inc. All rights reserved.

Published by Jossey-BassA Wiley Imprint989 Market Street, San Francisco, CA 94103–1741—www.josseybass.com

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, 978–750–8400, fax [phone], or on the Web at www.copyright.com. Requests to the publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, 201–748–6011, fax [phone], or online at www.wiley.com/go/permissions.

Readers should be aware that Internet Web sites offered as citations and/or sources for further information may have changed or disappeared between the time this was written and when it is read.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

Jossey-Bass books and products are available through most bookstores. To contact Jossey-Bass directly call our Customer Care Department within the U.S. at 800–956–7739, outside the U.S. at 317–572–3986, or fax [phone].

Jossey-Bass also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books.

Library of Congress Cataloging-in-Publication Data

Colton, David, 1948-

Designing and constructing instruments for social research and evaluation/David Colton and Robert W. Covert.

p. cm.

Includes bibliographical references and index.

ISBN-13: 978-0-7879-8784-8 (cloth)

1. Social sciences—Research—Methodology. 2. Evaluation—Methodology. I. Covert, Robert W., 1943- II. Title.

H62.C583 2007

300.72—dc22

2007026748

FIGURES, EXHIBITS, TABLES, AND INSTRUMENTS

Figures

1.1 Categories of Social Science Instruments

1.2 Steps in the Instrument Construction Process

8.1 Examples of Response Sets Written in the Same Direction

8.2 Matrix Layout for a Rating Scale

8.3 Difficulties Associated with Using Abstract Terms for Response Choices

15.1 Examples of Data Entry Errors by Respondents

Exhibits

5.1 Statement of Purpose

6.1 Questions to Address When Pilot-Testing the Questionnaire

7.1 Response Set Alternatives for Rating Scales

7.2 Juster Purchase Probability Scale

12.1 Organizing and Formatting Checklist

Tables

2.1 Study Planning Grid

3.1 Levels of Measurement

5.1 Processes and Outcomes

5.2 Q-Sort Distribution

5.3 Table of Specifications

11.1 Goal Attainment Scale

11.2 Goal Attainment Scale Conversion Table: Converts GAS Scores to Standard Scores

11.3 Item Analysis

Instruments

1.A Workshop Evaluation

1.B Sample Medical History

1.C Research Evaluation Checklist

2.A Political Opinion Poll

2.B Mental Health Screening Form-III

3.A Data Extraction Form

4.A Samples of Employee Evaluation Form Items

4.B Instructor Evaluation

5.A Employee Questionnaire

6.A Checklist for a Medical Record Audit

7.A Large-Scale Employee Satisfaction Survey

7.B Brief Situational Confidence Questionnaire

8.A Youth Risk Behavior Survey (Sample Items)

8.B Results of the 1998 Congressional Questionnaire

9.A Medical Record Audit Checklist

9.B Marketing Survey

10.A Open-Ended Item Examples and Commentary

10.B Behavioral Assessment

11.A Rosenberg Self-Esteem Scale

12.A Course Survey

12.B Training Needs Assessment

12.C Conflict Resolution Skills Assessment

13.A Behavioral Rating Scale

14.A Web Questionnaire

PREFACE: ASKING AND ANSWERING

When word of the people’s discontent reached the grand vizier, he had trouble sleeping, for, you see, he was a conscientious ruler who had the welfare of his people at heart. He called upon the captain of guards and directed him to question members of the populace.

Dutifully, the captain of the guards rounded up one hundred people, men and women, and asked them a series of questions to discover the cause of their discontent. Now the captain was an imposing fellow, some six feet four inches tall and muscular. He wore a scimitar in a scabbard, and the medals he had won for his bravery in battle adorned his waistcoat. When he addressed the citizens, his voice had a hard and commanding tone, and he looked at them with the steely glare of a man ready for combat, which of course was how he approached all tasks.

“The grand vizier has commanded that you provide information about the quality of life in our kingdom. First, I want to know if you believe that taxes are too high?”

In unison all one hundred citizens responded, “No,” although records of this session suggest that some citizens responded less enthusiastically than others.

“Ah yes, good.” replied the captain of the guards. “Now tell me, do you think that the laws of the land are administered fairly?”

In unison all one hundred citizens responded, “Yes,” although it was noted that one citizen had recently lost property in a dispute with the vizier’s second cousin’s nephew.

“Splendid,” said the captain of the guards, and he exhaled a sigh of relief, for he did not relish bringing bad news to the grand vizier. “And do you citizens agree with the new law prohibiting mules in the marketplace?”

In unison all one hundred citizens responded, “Yes,” although several were merchants whose businesses were hurt by the new law.

And so the questioning went, and the captain of the guard was pleased that each time the citizens replied in unison and each time there was complete agreement to his question. Armed with this information (for the captain of the guards never went anywhere without being fully armed), he returned to the grand vizier. “Good news, my ruler,” he declared. “Although there is some grumbling among the populace, the people I spoke to, who were a diverse lot, all had good things to say about you and your administration. Sleep soundly now, for truly the kingdom is in good hands.”

The grand vizier did indeed sleep soundly, at least for a day or two, until word again reached him of the people’s discontent. Now this sorely vexed the vizier, for he had the best interests of his people at heart. But he was also confused, because what the people had told the captain of the guards was clearly at odds with what he continued to hear. For that reason he sent for Halcolm, a teacher and counselor, and asked him to go forth among the people to determine the source of their displeasure. Halcolm agreed to this task with the understanding that he would speak to the people in confidence, so that they could speak freely without fear of reprisal.

Now Halcolm was a man of simple and plain means, very undiscerning in appearance. His voice was calm and reassuring, and people tended to feel comfortable in his presence. While buying some fruit in the marketplace, he asked the vendor, “I’ve been thinking of opening a stand in the marketplace to sell candlesticks and other brass goods. Do you think I could make a profit at that?” The vendor smiled and replied, “Yes and no. Surely you should have a good market for your wares. But as to making a profit, it will be hard given the current tax rate. When the vizier decided to build an addition to the palace, he upped our taxes by 20 percent! I’ll be happy when that project is over, and the taxes are reduced.”

Further down the marketplace, Halcolm spoke with a rug vendor and asked the same question. “Yes,” the rug vendor replied, “you will certainly have a market for brass goods. However, the laws of the land are often applied unevenly, and they create a lot of headaches for us. For example, you might be fined if your cart is too close to the street. Also, since the vizier passed the rule outlawing mules in the marketplace, it is becoming increasingly difficult to carry our wares into town to sell. If changes aren’t soon made, many of us will go out of business. In the end, we, the vizier, and ultimately the entire kingdom will suffer under these rules. Now don’t get me wrong. I have great respect for the grand vizier. He has been a good and just ruler in the past. It is just that recently I think he has lost touch with the people.”

Halcolm continued his project and met with a number of people in the marketplace, at religious shrines, and at the theater. After several days of asking his questions he returned to the grand vizier and shared his findings. “Perhaps I have been out of touch,” sighed the vizier. “I will immediately revise my plans for constructing and financing the palace addition. Also, I will establish a committee to examine our laws and rewrite them so that they are fair and just. Thank you, Halcolm, for finding out what really troubled our citizens. I am pleased that the people were so honest in their responses. If I had depended on what people told the captain of the guards, my kingdom might have gone down in ruin,” and as he spoke those words, he glared at the captain of the guards, who was also present at this meeting.

“Thank you for your kind words,” responded Halcolm, “but I would like to add that as a teacher and counselor, it is my job to know how to ask the right questions and how to ask questions right. You would not, however, want me to lead your army into battle. For that, you would want your captain of the guards, who is an excellent tactician and great warrior.” With that, the grand vizier smiled, the captain of the guards smiled, and Halcolm smiled, for in addition to knowing how to ask a good question, Halcolm knew how to phrase a good answer.

(With special thanks to Michael Quinn Patton, for providing the evaluation profession a role model in the guise of Halcolm.)

Increasingly, individuals and organizations are being asked to collect, manage, and use information for decision making, particularly to improve the quality of services and products. Rather than being based on intuition or hunches, decision making is viewed as being a data-driven process, one that is systematic and produces trustworthy information.

The purpose of this book is to provide the reader with a systematic, nontechnical and commonsense approach to developing instruments for data collection and analysis. We have written the book as a guide for both those who are using or developing instruments for the first time and those with experience who want to hone their skills, people ranging from students to agency personnel to program managers to researchers. The book does not require any technical expertise and is written for all levels of readers.

Throughout the text we use the term instrument generically to describe any format for collecting data, such as attitudinal questionnaires, checklists, and political polls. And as we note in the text, the process for developing an instrument is the same whether that instrument will be used for self-rating (by a respondent) or whether another person (an observer) will fill it out.

There are literally thousands of instruments that have been developed and marketed to fulfill distinct needs for information. However, we have found that informational needs are often unique to a particular person, organization, situation, time, or event. Consequently, these needs can be met only by designing and developing a questionnaire for that specific purpose and situation. Additionally, finding a previously developed instrument may be time consuming, and the cost of purchasing the instrument may prove to be greater than the cost of developing a new instrument that uniquely fits your needs.

The approach presented here is based on the underlying assumption that the process of constructing an instrument is both a creative and a technical venture. It involves not only being very familiar with the content or substance of the topic of interest but also developing good questions or items and presenting them in a format accessible to the people who will have to complete the instrument. Consequently, this book is designed to help you create an instrument that will obtain the information you seek.

Throughout we emphasize the need to ensure that an instrument will produce trustworthy and accurate data. To that end, we provide guidelines for reviewing and revising to enhance data validity and reliability. Additionally, we stress the importance of involving, throughout the process of instrument construction, the different groups of people who will be affected by the data generated. These groups, referred to as stakeholders, include the instrument designer(s), decision makers (such as administrators, policymakers, and funding agencies), agency personnel, clients, and raters or respondents.

The ideas in this book are based on the authors’ experience in several hundred evaluation projects over the past twenty-five years. In addition, they have been presented in a variety of settings, from the university classroom to workshops on questionnaire construction. Moreover, in writing this text, we have addressed you, the reader, directly, as if we were present to advise you through this process. We hope this familiarity will make you more comfortable and less intimidated as you undertake the challenge of designing an instrument.

This book is organized around the process of instrument construction and takes the reader through each of the steps. The chapters in Part One present the conceptual basis for designing and constructing instruments for data collection and analysis. We describe how instruments fit into the process of social inquiry and how different types support specific informational needs. Before you decide on the type of instrument to construct, it is important to understand the variety of approaches available for gathering information about a particular research or evaluation question. These chapters describe the various types of instruments as well as the components of an effective instrument. We also introduce such concepts as validity and reliability.

The chapters in Part Two offer guidance in constructing questionnaires and other forms of instrumentation, helping you to define the purpose of your study, to understand and choose among the different ways to format items, and to pretest and construct items that will meet your informational needs.

Part Three provides guidelines for organizing the instrument, administering it, and reporting the results to stakeholders and decision makers. This is to ensure that the effort you put into obtaining reliable data, through a well-designed instrument, is not compromised when you actually carry out the measurement process.

To give you opportunities for applying the information presented in this text, the chapters conclude with examples of instruments. These instruments have been developed to meet different informational needs, and many come from the public domain. Each one is discussed and critiqued, and reviewing these samples can help you hone your skills both as a user of instruments and as a developer and designer of your own questionnaires.

Feedback

A central theme of this book is that instrument construction is a process of continual development and refinement. We welcome feedback and the sharing of information that can improve both the substance and presentation of our material. Please contact us through the publisher.

Acknowledgments

Our view that instrument construction involves constant revision has certainly held true during the development of this book. We want to express our appreciation to Daniel M. Stuhlsatz, of Mary Baldwin College, Staunton, Virginia, who reviewed an early version of the manuscript, and to Gary Skolits, of the University of Tennessee, Knoxville, Tennessee, who reviewed a revised version. Their feedback at both stages of development was incorporated into successive drafts.

We also want to acknowledge the students who have taken the instrument construction class we teach at the Curry School of Education, University of Virginia, as this book was in part written to address the topics and questions that typically arise there. Student projects have also provided useful resources, and one group in particular, the class of 2003, also reviewed and provided feedback for an early version of the manuscript. We offer special thanks to Tracey Armstrong, Ronda Bryant, Dan Bublitz, Holly Conti, Antoinette Ewell, Alan Fortescure, Jeanne Hineline, Sa Rah Ho, Amelia Hunt, Catherine Johnson, Jae Hyun Jun, Jennifer Mabry, Ana Paula Loucao Martins, Shizuka Modica, Yi Ni, Ana Palla, Margaret Peak. Nikkia Sheppard, Doug Toti, Dave Wolcott, Trimika Yates, Hyunsil Yoo, and Yubo Zhang. The multicultural makeup of this class led to very interesting discussions on translating instruments into different languages.

We also offer a special thank-you to our families for their support and encouragement during this drawn-out but fulfilling writing project. Finally, we thank the editorial staff at Jossey-Bass, in particular Elspeth MacHattie and Rachel Anderson, for their assistance in helping us through the publication process.

June 2007

David Colton

Waynesboro, Virginia

Robert W. Covert

Faber, Virginia

THE AUTHORS

David Colton comes to the area of instrument construction as both a practitioner in the field and an educator and evaluator. He received his BA degree in secondary education and MEd degree in reading education from Salisbury University, in Salisbury, Maryland, an MPA degree from James Madison University, in Harrisonburg, Virginia, and an PhD degree, with a major in evaluation, from the University of Virginia at Charlottesville. Over the past thirty years he has worked in a variety of management and program development positions with the Maryland and Virginia public mental health systems. His current position involves evaluation, performance measurement, and quality improvement for an inpatient treatment facility serving children and adolescents. He has published a number of articles on quality improvement and cultural change in human service agencies, and his Checklist for Assessing Organizational Readiness for Reducing Seclusion and Restraint is used by mental health organizations throughout the United States, Canada, and Europe. For the past seventeen years, Colton has served as an adjunct instructor with the health care administration program at Mary Baldwin College, in Staunton, Virginia, and for the past five years he has taught the course on instrument construction at the Curry School of Education, University of Virginia.

Robert W. Covert is associate professor in the program area of research, statistics and evaluation at the Curry School of Education, University of Virginia. He received his BS degree in math from Grove City College, in Grove City, Pennsylvania, his MEd degree in math education, and his PhD degree in educational psychology, with a specialization in research and statistics, from Temple University, in Philadelphia, Pennsylvania. An active member of the evaluation community, he served as the second president of the American Evaluation Association and was responsible for that group’s membership services for over ten years. He is a recipient of the Robert Ingle Service Award, presented by the American Evaluation Association for continuous contribution to its leadership and mission. He has conducted over 500 evaluations, primarily in the education and social services sectors. As a faculty member at the University of Virginia he has designed and taught a variety of methodological courses, including instrument construction, computer statistics lab, and introduction to qualitative research methods. In addition he has designed and currently teaches a popular course in multicultural education, and he has conducted AEA presessions in instrument construction and multicultural issues for evaluators.

PART ONECONCEPTS

CHAPTER ONEINTRODUCTION

In this chapter we will

Explain the purpose and function of a social science instrument.

Describe nomenclature used to describe instruments.

List and describe the components of an instrument.

Outline the steps in the instrument construction process.

We are living in a time characterized as the information age, and we encounter data-gathering instruments in all facets of our lives. For example, we are familiar with polls that gather information about political preferences and voting behaviors. Surveys of potential voters try to predict who will be elected or what proposition will pass. Media commentators remind us of the margin of error associated with a survey or note that an election is still too close to call.

The proliferation of instruments to provide data and information for decision making is not unique to political polls. Survey questionnaires can be used to obtain factual information and to assess attitudes and beliefs across a variety of topics and groups. For example, surveys can assess consumer behaviors, client satisfaction with services, employee attitudes, and the general public’s values and beliefs. The federal government is perhaps the greatest consumer and user of survey questionnaires, as these instruments are used to collect data about such topics as criminal activity, educational needs, services to the mentally ill, and health care utilization, not to mention the data for the U.S. Census.

You may be familiar with other forms of measurement instruments as well. For example, many organizations conduct annual evaluations of employee work performance. Often these evaluation instruments use scales to rate job performance on a number of attributes, such as attendance, ability to work with others, or the ability to complete work tasks in a timely manner. Although the intended purpose of a checklist or rating instrument such as an employee evaluation is different from the purpose of a survey questionnaire, the same principles are used in constructing these instruments.