The Wiley Handbook of Cognition and Assessment E-Book

• Herausgeber: John Wiley & Sons
• Kategorie: Bildung
• Serie: Wiley Handbooks in Education
• Sprache: Englisch

Seitenzahl: 1437

Veröffentlichungsjahr: 2016

Beschreibung

This state-of-the-art resource brings together the most innovative scholars and thinkers in the field of testing to capture the changing conceptual, methodological, and applied landscape of cognitively-grounded educational assessments. * Offers a methodologically-rigorous review of cognitive and learning sciences models for testing purposes, as well as the latest statistical and technological know-how for designing, scoring, and interpreting results * Written by an international team of contributors at the cutting-edge of cognitive psychology and educational measurement under the editorship of a research director at the Educational Testing Service and an esteemed professor of educational psychology at the University of Alberta as well as supported by an expert advisory board * Covers conceptual frameworks, modern methodologies, and applied topics, in a style and at a level of technical detail that will appeal to a wide range of readers from both applied and scientific backgrounds * Considers emerging topics in cognitively-grounded assessment, including applications of emerging socio-cognitive models, cognitive models for human and automated scoring, and various innovative virtual performance assessments

Leseprobe

The Handbook of Cognition and Assessment

Frameworks, Methodologies, and Applications

Edited by

André A. Rupp and Jacqueline P. Leighton

This edition first published 2017© 2017 John Wiley & Sons, Inc

Registered OfficeJohn Wiley & Sons, Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, UK

Editorial Offices350 Main Street, Malden, MA 02148‐5020, USA9600 Garsington Road, Oxford, OX4 2DQ, UKThe Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, UK

For details of our global editorial offices, for customer services, and for information about how to apply for permission to reuse the copyright material in this book please see our website at www.wiley.com/wiley‐blackwell.

The right of André A. Rupp and Jacqueline P. Leighton to be identified as the authors of the editorial material in this work has been asserted in accordance with the UK Copyright, Designs and Patents Act 1988.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by the UK Copyright, Designs and Patents Act 1988, without the prior permission of the publisher.

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books.

Designations used by companies to distinguish their products are often claimed as trademarks. All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners. The publisher is not associated with any product or vendor mentioned in this book.

Limit of Liability/Disclaimer of Warranty: While the publisher and authors have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. It is sold on the understanding that the publisher is not engaged in rendering professional services and neither the publisher nor the author shall be liable for damages arising herefrom. If professional advice or other expert assistance is required, the services of a competent professional should be sought.

Library of Congress Catalog Number: 2016036147

ISBN Hardback: 9781118956571

A catalogue record for this book is available from the British Library.

Cover image: Claire Sower, Pool of Dreams (2016) / Claire Sower, Falling in Love (2016)

To Brooke, my only possible soulmate and the most loving, inspiring, and simply fabulous partner I could ever hope for as well as to Jean‐Marie, my truly amazing, compassionate, and delightful son ‐ you two are my family and will always be immensely loved!

André A. Rupp

To my husband and best friend, Greg.

Jacqueline P. Leighton

The Wiley Handbooks in Education offer a capacious and comprehensive overview of higher education in a global context. These state‐of‐the‐art volumes offer a magisterial overview of every sector, sub‐field and facet of the discipline – from reform and foundations to K–12 learning and literacy. The Handbooks also engage with topics and themes dominating today’s educational agenda – mentoring, technology, adult and continuing education, college access, race and educational attainment. Showcasing the very best scholarship that the discipline has to offer, The Wiley Handbooks in Education will set the intellectual agenda for scholars, students, researchers for years to come.

The Wiley Handbook of Learning TechnologyEdited by Nick Rushby and Daniel W. Surry

The Handbook of Cognition and AssessmentEdited by André A. Rupp and Jacqueline P. Leighton

Notes on Contributors

Jessica J. Andrews is an Associate Research Scientist in the Computational Psychometrics Research Center at Educational Testing Service (ETS) in Princeton, NJ. She received her Ph.D. in Learning Sciences at Northwestern University. Her research examines the cognitive processes underlying collaborative learning, and the use of technological environments (e.g., simulations, learning management systems) in supporting student learning and assessing individuals’ cognitive and noncognitive (e.g., collaborative) skills.

Ryan S. Baker is Associate Professor of Cognitive Studies at Teachers College, Columbia University, and Program Coordinator of TC’s Masters of Learning Analytics. He earned his Ph.D in Human‐Computer Interaction from Carnegie Mellon University. Dr. Baker was previously Assistant Professor of Psychology and the Learning Sciences at Worcester Polytechnic Institute, and served as the first Technical Director of the Pittsburgh Science of Learning Center DataShop, the largest public repository for data on the interaction between learners and educational software. He was the founding president of the International Educational Data Mining Society, and is currently Associate Editor of the Journal of Educational Data Mining. He has taught two MOOCs, Big Data and Education (twice), and (co‐taught) Data, Analytics, and Learning. His research combines educational data mining and quantitative field observation methods to better understand how students respond to educational software, and how these responses impact their learning. He studies these issues within intelligent tutors, simulations, multi‐user virtual environments, MOOCs, and educational games.

John T. Behrens is Vice President, Advanced Computing & Data Science Lab at Pearson and Adjunct Assistant Research Professor in the Department of Psychology at the University of Notre Dame. He develops and studies learning and assessment systems that integrate advances in the learning, computing, and data sciences. He has written extensively about the use of evidence‐centered design to guide development of complex educational systems as well as about the foundational logics of data analysis/data science and the methodological impacts of the digital revolution.

Isaac I. Bejar holds the title of Principal Research Scientist with Educational Testing Service (ETS) in Princeton, NJ. He is interested in improving methods of testing by incorporating advances in psychometric theory, cognitive psychology, natural language processing, and computer technology. He was a member of the editorial board and advisory board of Applied Psychological Measurement from 1981 to 1989, and was awarded the ETS Research Scientist Award in 2000. He published Cognitive and Psychometric Analysis of Analogical Problem Solving and co‐edited Automated Scoring of Complex Tasks in Computer‐Based Testing.

Laine Bradshaw is an Assistant Professor of Quantitative Methodology in the Educational Psychology Department in the College of Education at the University of Georgia (UGA). Her primary research focuses on advancing multidimensional psychometric methodology to support the diagnostic assessment of complex knowledge structures for educational purposes. With a Master’s degree in Mathematics Education, she is also active in collaborations on interdisciplinary assessment development projects that require tailoring psychometrics to cognitive theories. Her work has been published in journals such as Psychometrika and Educational Measurement: Issues and Practice. Her early career program of research was recently recognized by the National Council of Measurement in Education’s Jason Millman Award.

Gabrielle Cayton‐Hodges is a Research Scientist in the Learning Sciences Group at Educational Testing Service (ETS) in Princeton, NJ. She earned her BS degree in Brain and Cognitive Sciences from MIT and her PhD in Mathematics, Science, Technology, and Engineering Education from Tufts University. Gabrielle’s specialty is mathematical cognition and elementary mathematics education, focusing on the application of cognitive and learning sciences to mathematics assessment and the use of technology to support innovative approaches to gathering evidence about what students know and can do. She has a specific expertise in student understandings of numerical concepts such as place value and the use of multiple representations in mathematics and has also spent several years studying early algebra and learning progressions in the understanding of area and volume.

Sun‐Joo Cho is an Assistant Professor at Peabody College, Vanderbilt University. Her research topics include generalized latent variable modeling and its parameter estimation, with a focus on item response modeling.

Ying Cui is an Associate Professor at the University of Alberta. Her research interests include cognitive diagnostic assessment, person fit analysis, and applied statistical methods.

Paul De Boeck is Professor of Quantitative Psychology at The Ohio State University and emeritus from the KU Leuven (Belgium). He is especially interested in how psychometric models can be redefined as explanatory models or supplemented with explanatory components for applications in psychology and education.

Kristen E. DiCerbo’s research program centers on digital technologies in learning and assessment, particularly on the use of data generated from interactions to inform instructional decisions. She is the Vice President of Education Research at Pearson and has conducted qualitative and quantitative investigations of games and simulations, particularly focusing on the identification and accumulation of evidence. She previously worked as an educational researcher at Cisco and as a school psychologist. She holds doctorate and master’s degrees in Educational Psychology from Arizona State University.

Susan Embretson is Professor of Psychology at the Georgia Institute of Technology. Previously, she was Professor at the University of Kansas. Her research concerns integrating cognitive theory into psychometric item response theory models and into the design of measurement tasks. She has been recognized for this research, including the Career Contribution Award (2013) and the Technical and Scientific Contribution Award (1994–1997) from the National Council on Measurement and Education; the Distinguished Lifetime Achievement Award (2011) from the American Educational Research Association: Assessment and Cognition; and the Distinguished Scientist Award from American Psychological Association Division (5) for Measurement, Evaluation and Statistics for research and theory on item generation from cognitive theory. Embretson has also served as president for three societies in her area of specialization.

Gary Feng is a Research Scientist in the Research and Development division at Educational Testing Service (ETS) in Princeton, NJ. He works in the Cognitive, Accessibility, and Technology Sciences Center. He received his PhD in Developmental Psychology and MS in Statistics from the University of Illinois at Champaign‐Urbana. Before joining ETS, he was a faculty member at Duke University and held visiting and research positions at the University of Michigan and the University of Potsdam, Germany. He is broadly interested in the acquisition of reading skills and neurocognitive processes in reading. His past work uses eye‐tracking to examine cognitive processes of skilled and developing readers across different cultures. Gary contributes to the development of innovative literacy assessments.

Steve Ferrara was Vice President for Performance Assessment and led the Center for Next Generation Learning and Performance in Pearson’s Research and Innovation Network. Steve conducts psychometric research and designs large scale and formative assessments and automated language learning systems. He specializes in principled design, development, implementation, and validation of performance assessments and in research content, cognitive, and linguistic response demands placed on examinees and predicts technical characteristics of items. Steve earned an MEd in Special Education from Boston State College and an EdS in Program Evaluation and a PhD in Educational Psychology and measurement from Stanford University.

Mark J. Gierl is Professor of Educational Psychology and the Director of the Centre for Research in Applied Measurement and Evaluation (CRAME) at the University of Alberta. His specialization is educational and psychological testing, with an emphasis on the application of cognitive principles to assessment practices. Professor Gierl’s current research is focused on automatic item generation and automated essay scoring. His research is funded by the Medical Council of Canada, Elsevier, ACT Inc., and the Social Sciences and Humanities Research Council of Canada. He holds the Tier I Canada Research Chair in Educational Measurement.

Janice D. Gobert is a Professor of Learning Sciences and Educational Psychology at Rutgers. Formerly, she was the Co‐director of the Learning Sciences and Technologies Program at Worcester polytechnic Institute. Her specialty is in technology‐based with visualizations and simulations in scientific domains; her research areas are: intelligent tutoring systems for science, skill acquisition, performance assessment via log files, learning with visualizations, learner characteristics, and epistemology. She is also the Founding CEO of a start‐up company named Apprendis (www.apprendis.com), whose flagship products are Inq‐ITS and Inq‐Blotter, both described in the chapter.

José P. González‐Brenes is a Research Scientist in the Center for Digital Data, Analytics & Adaptive Learning at Pearson. He investigates methods of machine learning to make education faster, better, and less expensive. His work has been nominated for best paper awards in the International Educational Data Mining and the Special Interest Group of Dialogue Systems conferences. He is the happy first‐prize winner of international data mining competition involving over 350 teams. His postgraduate training includes a PhD in Computer Science from Carnegie Mellon University and an IMBA in Technology Management from National Tsing Hua University in Taiwan.

Joanna S. Gorin is Vice President of Research at Educational Testing Service (ETS) in Princeton, NJ. As Vice President for Research, she is responsible for a comprehensive research agenda to support current and future educational assessments for K–12, higher education, global, and workforce settings. Her research has focused on the integration of cognitive theory and psychometric theory as applied to principled assessment design and analysis. Prior to joining ETS, Joanna was an Associate Professor at Arizona State University where her research focused on the application of cognitive theories and methods to the design and validation of tests of spatial reasoning, quantitative reasoning, and verbal reasoning. She received her PhD in Quantitative Psychology (minor: Cognitive Psychology) from the University of Kansas.

Qi Guo is a PhD student at the University of Alberta. His research interests include cognitive diagnostic assessment, test reliability, and structural equation modeling.

Jeffrey R. Harring is an Associate Professor of Measurement, Statistics and Evaluation in the Department of Human Development and Quantitative Methodology at the University of Maryland. Generally, his research focuses on the development and evaluation of statistical models and methods used in education, social and behavioral science research. His current research centers on methodological issues surrounding linear, generalized linear, and nonlinear latent variable models for longitudinal data. Other threads of his research focus on finite mixture models and nonlinear structural equation models.

Ari Houser is a doctoral candidate in the Measurement, Statistics and Evaluation program within the Department of Human Development and Quantitative Methodology at the University of Maryland. He works concurrently as a Senior Methods Advisor in the AARP Public Policy Institute. His main research interests are on longitudinal models for discrete‐valued latent variables.

Kristen Huff received her EdD in Measurement, Research and Evaluation Methods from the University of Massachusetts Amherst in 2003, and her MEd in Educational Research, Measurement, and Evaluation from the University of North Carolina at Greensboro in 1996. Her work focuses on ensuring the coherence of assessment design, interpretation, use, and policy to advance equity and high‐quality education for all students. Currently, Kristen serves as Vice President, Research Strategy and Implementation at ACT.

G. Tanner Jackson is a Research Scientist at Educational Testing Service (ETS) in Princeton, NJ. His work focuses on innovative assessments and student process data, including the development and evaluation of conversation‐based assessments (through ETS strategic initiatives) and game‐based assessments (working in collaboration with academic and industry partners). Additionally, Tanner is interested in how users interact with complex systems and he leverages these environments to examine and interpret continuous and live data streams, including user interactions across time within educational environments.

Irvin R. Katz is Senior Director of the Cognitive, Accessibility, and Technology Sciences Center at Educational Testing Service (ETS) in Princeton, NJ. He received his PhD in Cognitive Psychology from Carnegie Mellon University. Throughout his 25 year career at ETS, he has conducted research at the intersection of cognitive psychology, psychometrics, and technology, such as developing methods for applying cognitive theory to the design of assessments, building cognitive models to guide interpretation of test‐takers’ performance, and investigating the cognitive and psychometric implications of highly interactive digital performance assessments. Irv is also a human‐computer interaction practitioner with more than 30 years of experience in designing, building, and evaluating software for research, industry, and government.

Madeleine Keehner is a Managing Senior Research Scientist in the Cognitive, Accessibility, and Technology Sciences Center at Educational Testing Service (ETS) in Princeton, NJ. She received her PhD in experimental psychology from the University of Bristol and her BS degree (honors) in psychology from the University of London, Goldsmiths College. She also received a Certificate in Education from the University of Greenwich. Maddy has studied individual differences in spatial and general reasoning in medicine and the STEM disciplines. She is also interested in what we can infer from process data captured by new technologies such as interactive virtual models and simulations. Her current work focuses on understanding cognition in various domains within the NAEP program and is exploring cognitive processes related to interactive computer‐based or tablet‐based assessments.

Deirdre Kerr is an Associate Research Scientist in the Computational Psychometrics Research Center at Educational Testing Service (ETS) in Princeton, NJ. Her research focuses on determining methods of extracting information about student understanding and performance from low‐level log data from educational video games and simulations. Publications include Identifying Key Features of Student Performance in Educational Video Games and Simulations through Cluster Analysis, Identifying Learning Trajectories in an Educational Video Game, and Automatically Scoring Short Essays for Content.

Leanne R. Ketterlin‐Geller is a Professor in Education Policy and Leadership at Southern Methodist University in Dallas, TX. Her research focuses on the development and validation of formative assessment systems in mathematics to support instructional decision making. She investigates the application of test accommodations and principles of universal design for improving accessibility of educational assessments for all students.

Jennifer L. Kobrin is Director of Institutional Research and Effectiveness at the Graduate Center, City University of New York. Her current research focuses on higher education assessment and institutional effectiveness. Her previous research focused on the promise of learning progressions for improving assessment, instruction, and teacher development. She holds a doctorate in Educational Statistics and Measurement from Rutgers University and a Masters in Educational Research, Measurement, and Evaluation from Boston College.

James Koepfler is a Senior Analytical Consultant at SAS. His areas of interest include operational diagnostic assessments, large‐scale assessment implementation, IRT, vertical scaling, and applied statistics. He holds a PhD in Assessment and Measurement and a Masters in Psychological Sciences from James Madison University.

Patrick C. Kyllonen is Senior Research Director of the Center for Academic and Workforce Readiness and Success at Educational Testing Service (ETS) in Princeton, NJ. Center scientists conduct innovative research on (a) higher education assessment; (b) workforce readiness; (c) international large scale assessment (e.g., Program for International Student Assessment; PISA); and (d) twenty‐first‐century skills assessment, such as creativity, collaborative problem solving, and situational interviews. He received his BA from St. John's University and PhD from Stanford University and is author of Generating Items for Cognitive Tests (with S. Irvine, 2001); Learning and Individual Differences (with P. L. Ackerman & R. D. Roberts, 1999); Extending Intelligence: Enhancement and New Constructs (with R. Roberts and L. Stankov, 2008), and Innovative Assessment of Collaboration (with A. von Davier and M. Zhu, forthcoming). He is a fellow of the American Psychological Association and the American Educational Research Association, recipient of The Technical Cooperation Program Achievement Award for the “design, development, and evaluation of the Trait‐Self Description (TSD) Personality Inventory,” and was a coauthor of the National Academy of Sciences 2012 report, Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century.

Emily Lai is Director of Formative Assessment and Feedback in the Efficacy and Research organization at Pearson. Emily's areas of interest include principled assessment design approaches, performance assessment, assessment for learning, and assessment of twenty‐first‐century competencies. Her most recent research includes co‐developing a learning progression and online performance assessments to teach and assess concepts related to geometric measurement of area. Emily holds a PhD in Educational Measurement & Statistics from the University of Iowa, a Masters in Library and Information Science from the University of Iowa, and a Masters in Political Science from Emory University.

Hollis Lai is Assistant Professor of Dentistry and the Director of Assessment for Undergraduate Medical Education program at the University of Alberta. His specialization is educational and psychological testing, with an emphasis on assessment designs in medical education, curriculum mapping, educational data mining, and item generation.

Roy Levy is an Associate Professor of Measurement and Statistical Analysis in the T. Denny Sanford School of Social & Family Dynamics at Arizona State University. His primary research interests include methodological developments and applications of psychometrics and statistical modeling in item response theory, Bayesian networks, and structural equation modeling, with applications in assessment, education, and the social sciences. He recently published Bayesian Psychometric Modeling (with Robert J. Mislevy).

Jacqueline P. Leighton is Professor and Chair of Educational Psychology and past Director of the Centre for Research in Applied Measurement and Evaluation (CRAME), a centre that is part of the Department she oversees at the University of Alberta. As a registered psychologist with the College of Alberta Psychologists, her research is focused on measuring the cognitive and socio‐emotional processes underlying learning and assessment outcomes, including cognitive diagnostic assessment and feedback delivery and uptake. Funded by NSERC and SSHRC, she completed her graduate and postdoctoral studies at the University of Alberta, and Yale University, respectively. She has published in a variety of educational measurement journals, is past editor of Educational Measurement: Issues and Practice, and has published 3 books with Cambridge University Press.

Richard M. Luecht is a Professor of Educational Research Methodology at the University of North Carolina at Greensboro. His research interests include developing computer‐based testing models and software, large‐scale computerized assessment systems design, standard setting, innovative item design, item response theory parameter estimation, scaling linking and equating, automated test design algorithms and heuristics, and the application of design engineering principles to assessment.

Taylor Martin is an Associate Professor in Instructional Technology and Learning Sciences at Utah State University, where she is a principal investigator of the Active Learning Lab. Her work focuses on how learning, instruction, and practice come together in authentic contexts for Science, Technology, Engineering, and Mathematics education, focusing on topics ranging from how children learn fractions to how engineers refine their problem‐solving skills. Her findings demonstrate that active learning strategies can improve motivation, encourage innovative thinking, and match traditional strategies on developing core content knowledge. In addition, she employs data science methods to understand how these strategies impact important outcomes. She is currently on assignment at the National Science Foundation, focusing on a variety of efforts to understand how Big Data is impacting research in Education and across the STEM disciplines. Previously, she was at the Department of Curriculum and Instruction The University of Texas at Austin.

Robert J. Mislevy is the Frederic M. Lord Chair in Measurement and Statistics at Educational Testing Service (ETS) and Emeritus Professor at the University of Maryland. His research applies developments in technology, statistics, and cognitive science to practical problems in assessment. His work includes collaborating with Cisco Systems on simulation‐based assessment of network engineering and developing an evidence‐centered assessment design framework. Publications include Bayesian Networks in Educational Assessment, Bayesian Psychometric Modelling, and the Cognitive Psychology chapter in Educational Measurement.

Paul D. Nichols is a Senior Director in Research at ACT where Paul supports assessment and product design, the development of validity arguments and the use of qualitative methods. Paul’s current research focuses on applying the theories and methods from the learning sciences to a broad range of activities in educational measurement. Paul holds a PhD and a Masters in educational psychologyfrom the University of Iowa.

Andreas Oranje is a Principal Research Director in the Research department of Educational Testing Service (ETS) in Princeton, NJ. He oversees various research centers focused on the development and validation of generalizable assessment capabilities including automated scoring evaluation, natural language and speech processing, dialogic and multimodal assessment, cognitive science, assessment and assistive technologies, and psychometric research related to group‐score assessments. He serves as Project Director for Design, Analysis, and Reporting of the National Assessment of Educational Progress (NAEP 2013–2017). His research interests include designs for large scale (adaptive) assessments, psychometric research, and game‐ and scenario‐based assessment.

William R. Penuel is a Professor of Learning Sciences and Human Development in the School of Education at the University of Colorado Boulder. His research focuses on the design, implementation, and evaluation of innovations in science and mathematics education. He has designed a number of innovations focused on improving classroom assessment in science and was a member of the committee that developed the consensus report, Developing Assessments for the Next Generation Science Standards (2014).

Hilary Persky is a Principal Assessment Designer in the Assessment Development division of Educational Testing Service (ETS) in Princeton, NJ. She has focused largely on forms of performance assessment in various subject areas, ranging from visual arts and theatre to science, writing, and most recently, reading tasks incorporating avatars. Her work is concerned with how to introduce meaningful innovation into large‐scale, on‐demand assessment while retaining reliable measurement. She is also interested in ways of enriching assessment reporting with process data, in particular in the area of writing.

Amy Reilly is Director of Research Support with Pearson’s Research and Innovation Network. Her previous work experience includes serving as the Pearson program manager for statewide assessment programs including Tennessee, Arkansas, and Utah and as a test development manager and lead content specialist in English/Language Arts. She also was a Texas public school teacher, specializing in reading interventions for special education students. Amy holds a BS in Interdisciplinary Studies from Texas A&M University and an MBA from St. Edwards University.

Lisa M. Rossi worked as a Research Analyst in the Educational Psychology Laboratory at Worcester Polytechnic Institute. She holds a Master’s degree in Human‐Computer Interaction from Georgia Institute of Technology and a Bachelor’s degree in Psychological Science from Worcester Polytechnic Institute. Currently, she works as a UX Architect for State Farm in Atlanta, Georgia.

André A. Rupp is a Research Director at Educational Testing Service (ETS) in Princeton, NJ, where he works with teams that conduct comprehensive evaluation work for mature and emerging automated systems. His research has focused on applications of principled assessment design frameworks in innovative assessment contexts as well as translating the statistical complexities of diagnostic measurement models into practical guidelines for applied specialists. Through dissemination and professional development efforts he is deeply dedicated to help interdisciplinary teams navigate the complicated trade‐offs between scientific, financial, educational, and political drivers of decision making in order to help shape best methodological practices.

Michael A. Sao Pedro gained his PhD under Janice Gobert’s supervision while at Worcester Polytechnic Institute. He is a Co‐Founder and the Chief Technology Officer of Apprendis. He specializes in the development of digital assessments for science using Educational Data Mining. Formerly, he was a Senior Software Engineer at BAE Systems (formerly ALPHATECH, Inc.). There, he led several artificial intelligence‐inspired software efforts on several Phase I/II SBIR and DARPA projects.

Jason Schweid is a former classroom educator who received his EdD in Measurement, Research and Evaluation Methods in 2011 and his MEd in Counseling in 2008, both from the University of Massachusetts Amherst. His work focuses on assessment design, development, validation and education policy. Currently, Jason serves as a Fellow for Assessment at the USNY Regents Research Fund, where he advises the NY State Department of Education on assessment design and policy.

Lorrie A. Shepard is Dean and Distinguished Professor of Research and Evaluation Methodology in the School of Education at the University of Colorado Boulder. Her early research focused on test validity, contingent on the contexts of test use. Her current research focuses on classroom assessment. Drawing on cognitive research and sociocultural theory, she examines ways that assessment can be used as an integral part of instruction to help students learn.

Valerie Shute is the Mack & Effie Campbell Tyner Endowed Professor in Education in the Department of Educational Psychology and Learning Systems at Florida State University. Her current research involves using games with stealth assessment to support learning – of cognitive and noncognitive knowledge, skills, and dispositions. Val’s research has resulted in numerous grants, a patent, and publications (e.g., Measuring and supporting learning in games: Stealth assessment, with Matthew Ventura).

Lubin Wang is a doctoral candidate of Instructional Systems and Learning Technologies program at Florida State University. Her research interests include game‐based learning and assessment. She is particularly interested in the assessment and improvement of problem‐solving skills as well as identifying gaming‐the‐system behaviors during gameplay. She has participated in various funded research projects led by Dr. Shute, and coauthored or is coauthoring several papers and chapters with her.

Zachary Warner is a former high school math teacher who received his PhD in Educational Psychology from the University of Albany, SUNY in 2013. His research focuses on large‐scale assessments and how results can best inform educational goals at school, district, and state levels. Zach has published research on using computer‐based formative assessment tools and rubric‐referenced student self‐assessment. He currently serves as a state psychometrician for the New York State Education Department.

Mark Wilson is the Director of the Berkeley Evaluation and Assessment Research (BEAR) Centre, and a professor at the University of California, Berkeley, USA, and also is a professor in Education (Assessment) in the Assessment Research Centre at The University of Melbourne, Australia. He is an internationally recognized specialist in psychometrics and educational assessment, and is currently president of the National Council for Measurement in Education (NCME). His work spans assessment topics such as mathematics and science assessment, cognitive modeling, learning progressions, school‐based assessment, and interactive online assessment of 21st century skills.

Diego Zapata‐Rivera is a Senior Research Scientist at Educational Testing Service (ETS) at Princeton, NJ. His research focuses on innovations in score reporting and technology‐enhanced assessment, including work on assessment‐based learning environments and game‐based assessments. He has published numerous articles and has been a committee member and organizer of conferences in his research areas. He is a member of the Editorial Board of the User Modeling and User‐Adapted Interaction journal and an Associate Editor of IEEE Transactions on Learning Technologies.

Mo Zhang is a Research Scientist in the Research and Development Division at Educational Testing Service (ETS) in Princeton, NJ. Her research interests lie in the methodology of measurement and validation for automated and human scoring.

Foreword

“I don't give a hoot about cognitive psychology!”

This statement (but using much saltier language) was said to me by a senior colleague sometime in my first year working at Educational Testing Service (ETS). As possibly the first cognitive psychologist on staff at ETS, I expected some culture clash when I arrived in 1990, and I was not disappointed. My research training on detailed investigations of human problem solving in academic domains differed greatly in terms of methodologies, perspectives, and typical Ns (tens versus thousands) from the psychometric tradition. For example, in 1990, few of my psychometrician colleagues had heard of think‐aloud studies in which students talk concurrently as they solved problems, let alone saw their value for real‐world educational measurement. Sometimes I struggled to convince people that someone like me, with no formal measurement training, had something useful to contribute.

On the other hand, my assessment development colleagues showed a great interest in cognition. They wanted to know, for example, why their test questions weren’t working as intended, such as being too difficult or too easy. As more than one assessment developer put it, “what were those test takers thinking?” Ah ha! Here was a clear use for cognitive psychology and think‐aloud studies (sometimes called “cognitive labs”): provide insights to the people who write test items. I had found a niche, but felt dissatisfied that I had failed to bridge the gap between cognition and psychometrics.

Thankfully, in the past quarter century, psychometricians increasingly have been the bridge builders, taking up the challenge of accommodating measurement models and assessment practices to theories of cognition. No doubt growing measurement challenges pushed things along, such as the educational community’s desire for assessments that reflect real‐world situations, provide more detailed information than a scaled unidimensional score, and target knowledge and skills beyond those that had been traditionally investigated by educational measures.

With this Handbook, André and Jackie have brought together a truly multidisciplinary group: classically trained psychometricians who have developed methods to incorporate cognitive models into measurement models as well as cognitive psychologists who have put their minds and theories to the problem of measurement. This is a special group of people, dedicated not only to rigorous science (you see their names regularly in major measurement journals), but also to bringing measurement science into real‐world practice.

Given the multidisciplinary group of authors, it’s no surprise that the Handbook should appeal to multiple audiences:

Psychometricians

who already use a psychometric modeling or assessment design framework approach involving cognitive models, but who want to learn about other methods and how they contrast with the ones with which they are familiar.

Educational measurement researchers