Multilevel Modeling Techniques and Applications in Institutional Research E-Book

• Herausgeber: John Wiley & Sons
• Kategorie: Bildung
• Serie: J-B IR Single Issue Institutional Research
• Sprache: Englisch

Seitenzahl: 238

Veröffentlichungsjahr: 2013

Beschreibung

Multilevel modeling is an increasingly popular multivariatetechnique that is widely applied in the social sciences.Increasingly, practitioners are making instructional decisionsbased on results from their multivariate analyses, which often comefrom nested data that lend themselves to multilevel modelingtechniques. As data-driven decision making becomes more critical tocolleges and universities, multilevel modeling is a tool that willlead to more efficient estimates and enhance understanding ofcomplex relationships. This volume illustrates both the theoretical underpinnings andpractical applications of multilevel modeling in IR. It introducesthe fundamental concepts of multilevel modeling techniques in aconceptual and technical manner. Providing a range of examples ofnested models that are based on linear and categoricaloutcomes, it then offers important suggestions aboutpresenting results of multilevel models through charts andgraphs. This is the 154th volume of this Jossey-Bassquarterly report series. Always timely and comprehensive, NewDirections for Institutional Research provides planners andadministrators in all types of academic institutions withguidelines in such areas as resource coordination, informationanalysis, program evaluation, and institutional management.

Leseprobe

Table of Contents

Cover

Title page

Copyright page

About AIR

Editors’ Notes

Chapter 1: Hierarchical Data Structures, Institutional Research, and Multilevel Modeling

Introduction

Historical Approaches to Analyzing Data with a Multilevel Structure

Importance of Advancements in Statistical Methods in Institutional Research

Research Questions in Institutional Research

Introducing Multilevel Models

Designing Research within Institutional Research Settings

Summary

Chapter 2: Introduction to Estimation Issues in Multilevel Modeling

Conceptual Introduction to Maximum

Reliability in MLM

Empirical Bayes Estimation

Deviance

Standard Error Estimation

Hypothesis Testing

Considerations When Choosing and Coding Variables for HLM

Conclusion

Chapter 3: Using Existing Data Sources/Programs and Multilevel Modeling Techniques for Questions in Institutional Research

Data Sources for Multilevel Modeling

Considerations when Applying MLM to Large-Scale Data Sets

Software to Estimate Multilevel Models

Programs to Assist in Analysis

Dealing with Missing Data

Final Thoughts

Chapter 4: Multilevel Models for Binary Data

Modeling Categorical Outcomes

Overview of Common Models for Binary and Binomial Responses

Multilevel Models for Categorical Outcomes

Models for Clustered Binary Data

Summary

Chapter 5: Cross-Classified Random Effects Models in Institutional Research

Using CCREM to Understand Faculty Gender Pay and Inform Institutional Policy

Methods

Data Source and Description of Study Sample

Results

Conclusion

Chapter 6: Multilevel Modeling: Applications to Research on the Assessment of Student Learning, Engagement, and Developmental Outcomes

Purpose of Study and Research Questions

Literature Review

Methods

Results

Interpreting and Presenting the Results

Conclusion

Chapter 7: Multilevel Modeling: Presenting and Publishing the Results for Internal and External Constituents

Example 1: Predicting First-Year Grades

Example 2: Critical Thinking Outcomes and Academic Disciplines

Principles of Effective Reporting

Index

OTHER TITLES AVAILABLE IN THE NEW DIRECTIONS FOR INSTITUTIONAL RESEARCH SERIES

MULTILEVEL MODELING TECHNIQUES AND APPLICATIONS IN INSTITUTIONAL RESEARCH

Joe L. Lott, II and James S. Antony (eds.)

New Directions for Institutional Research, no. 154

Paul D. Umbach, Editor-in-Chief

Copyright © 2012 Wiley Periodicals, Inc., A Wiley Company

All rights reserved. No part of this publication may be reproduced in any form or by any means, except as permitted under section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the publisher or authorization through the Copyright Clear- ance Center, 222 Rosewood Drive, Danvers, MA 01923; [phone]; fax [phone]. The code and copyright notice appearing at the bottom of the first page of an article in this journal indicate the copyright holder’s consent that copies may be made for personal or internal use, or for personal or internal use of specific clients, on the condition that the copier pay for copying beyond that permitted by law. This consent does not extend to other kinds of copying, such as copying for general distri-bution, for advertising or promotional purposes, for creating collective works, or for resale. Such permission requests and other permission inquiries should be addressed to the Permissions Department, c/o John Wiley & Sons, Inc., 111 River St., Hoboken, NJ 07030; [phone], fax [phone], http://www.wiley.com/go/permissions.

NEW DIRECTIONS FOR INSTITUTIONAL RESEARCH (ISSN 0271-0579, electronic ISSN 1536-075X) is part of The Jossey-Bass Higher and Adult Education Series and is published quarterly by Wiley Subscription Services, Inc., A Wiley Company, at Jossey-Bass, One Montgomery Street, Suite 1200, San Francisco, California 94104-4594 (publication number USPS 098-830). Periodicals Postage Paid at San Francisco, California, and at additional mailing offices. POSTMASTER: Send address changes to New Directions for Institutional Research, Jossey-Bass, One Montgomery Street, Suite 1200, San Francisco, California 94104-4594.

SUBSCRIPTIONS cost $109 for individuals and $297 for institutions, agen- cies, and libraries in the United States. See order form at end of book.

EDITORIAL CORRESPONDENCE should be sent to Paul D. Umbach, Leadership, Policy and Adult and Higher Education, North Carolina State University, Poe 300, Box 7801, Raleigh, NC 27695-7801.

New Directions for Institutional Research is indexed in Academic Search (EBSCO), Academic Search Elite (EBSCO), Academic Search Premier (EBSCO), CIJE: Current Index to Journals in Education (ERIC), Contents Pages in Education (T&F), EBSCO Professional Development Collection (EBSCO), Educational Research Abstracts Online (T&F), ERIC Database (Education Resources Information Center), Higher Education Abstracts (Claremont Graduate University), Multicultural Education Abstracts (T&F), Sociology of Education Abstracts (T&F).

Microfilm copies of issues and chapters are available in 16mm and 35mm, as well as microfiche in 105mm, through University Microfilms, Inc., 300 North Zeeb Road, Ann Arbor, Michigan 48106-1346.

www.josseybass.com

ISBN: 9781118444009

ISBN: 9781118457337 (epdf)

ISBN: 9781118457344 (epub)

ISBN: 9781118457351 (mobi)

THE ASSOCIATION FOR INSTITUTIONAL RESEARCH was created in 1966 to benefit, assist, and advance research leading to improved understanding, planning, and operation of institutions of higher education. Publication policy is set by its Publications Committee.

PUBLICATIONS COMMITTEE

Gary R. Pike (Chair)Indiana University–Purdue University IndianapolisGloria Crisp University of Texas at San AntonioPaul DubyNorthern Michigan UniversityJames HearnUniversity of GeorgiaTerry T. IshitaniUniversity of MemphisJan W. LyddonSan Jacinto Community CollegeJohn R. RyanThe Ohio State University

EX-OFFICIO MEMBERS OF THE PUBLICATIONS COMMITTEE

John Muffo (Editor, Assessment in the Disciplines), Ohio Board of Regents

John C. Smart (Editor, Research in Higher Education), University of Memphis

Richard D. Howard (Editor, Resources in Institutional Research), University of Minnesota

Paul D. Umbach (Editor, New Directions for Institutional Research), North Carolina State University

Marne K. Einarson (Editor, AIR Electronic Newsletter), Cornell University

Gerald W. McLaughlin (Editor, AIR Professional File/IR Applications), DePaul University

Richard J. Kroc II (Chair, Forum Publications Committee), University of Arizona

Sharron L. Ronco (Chair, Best Visual Presentation Committee), Florida Atlantic University

Randy Swing (Staff Liaison)

For information about the Association for Institutional Research, write to the following address:

AIR Executive Office

1435 E. Piedmont Drive

Suite 211

Tallahassee, FL 32308-7955

[phone]

[email protected]

http://airweb.org

Editors’ Notes

Many colleges and universities currently face mounting pressure to shrink their budgets and maximize resources on one hand, and they are expected to maintain and even increase their institutional profile on the other hand. Institutional research (IR) offices play central roles in addressing this rather dubious task through their multidimensional roles associated with institutional planning and decision making. Terenzini (1999) describes these roles in three tiers that promote the position of IR as organizational intelligence: (1) technical and analytical intelligence—factual knowledge or information, and analytical and methodological skills; (2) issues intelligence—substantive problems on which technical and analytical intelligence is brought to bear; and (3) contextual intelligence—understanding the culture of higher education and the particular campus where the researcher works. The functions associated with these IR roles are carried out by forging connections among institutional research, planning and budgeting, and assessment, effectiveness, and accreditation; or, as Volkwein (2008) calls them, the golden triangle of IR. The technical and analytical tier, which requires the most basic knowledge of statistics foundational to all practitioners in IR, is one of the most important functions.

Technical and analytical tools are increasing at a rapid pace, and a better understanding of them allows the IR practitioner to conduct research and shape institutional policy using cutting-edge technologies and methodologies. Delaney’s (1997) study of 243 New England colleges and universities found that many IR practitioners have limited methodological and technical expertise. Volkwein (2008) reminds us that “the primary role of IR has changed over time from emphasizing and requiring primarily descriptive statistics, fact books, and reporting to more analysis and evaluation, both quantitative and qualitative” (p. 8). It is important for IR practitioners to be on the cutting edge of research methodologies, particularly those that inform multivariate models. Multivariate models allow the IR practitioner to understand how the outcome of interest is affected by more than one variable.

Multilevel modeling is an increasingly popular multivariate technique. It is also referred to as hierarchical linear modeling (HLM), mixed-effects modeling, random-effects modeling, and covariance component modeling (Raudenbush and Bryk, 2002). Increasingly, IR practitioners are informing institutional decisions based on results from their multivariate analyses, which often come from nested data. Nested data are cases where lower-level units are located within one or more higher-level units. Common examples from data for many IR practitioners include students nested within institutions, students nested within majors or classes, and faculty nested within departments. HLM provides a class of models that take into account the hierarchical, or nested, structure of data and makes it possible to incorporate variables from all levels and examine how the variability in the outcome can be explained within and between nests, or clusters (Raudenbush and Bryk, 2002).

Countless studies have used institutional data to estimate multivariate models with student and faculty outcomes. Regarding student outcomes, most multi-institutional college impact studies have relied on ordinary least squares (OLS) regression (Astin and Denson, 2009). Since the 1960s, many studies have relied on OLS regression because it became increasingly sophisticated and complex, and developments in SPSS made it more efficient by giving users options that showed direct and indirect paths to the dependent variable and how each indirect path had been mediated by the action of other intervening variables (Astin and Denson, 2009). The advent and popularity of multilevel techniques provided insight into the limitations of OLS approaches; mainly in the presence of nested data, OLS techniques violate the assumption of independence that leads to imprecise parameter estimates and loss of statistical power, and increases the likelihood of rejecting a true null hypothesis (Snijders and Bosker, 1999; Raudenbush and Bryk, 2002). In addition, Astin and Denson (2009) observed that “increasing numbers of editorial reviewers for scholar journals are now routinely recommending that HLM rather than OLS regression be used whenever a study using individual and institutional data is submitted” (p. 365). Although they note the circumstances under which OLS could be applicable, they underscore the value of using HLM approaches where applicable. It is imperative that IR practitioners understand the nature of multimodeling techniques so they can properly model their nested and sometimes dependent data structures. Many times institutional decisions are based on feedback and analyses from IR practitioners, and multilevel modeling is one tool that will lead to more efficient estimations and enhance the ability to better understand complex relationships with nested data.

Chapters in this volume illustrate both the theoretical and practical information about using multilevel modeling in IR. Chapter 1 introduces the fundamental concepts of hierarchy and its statistical treatment in institutional research settings. This chapter also provides an overview and highlights the advantages of HLM and sets the stage for chapters that follow. By briefly discussing historical approaches to analyzing data with multilevel structures, giving examples of types of research questions in IR that can be addressed through HLM, providing examples as the authors in this volume explain the nature of HLM and IR, and explaining the meanings of variances and error structures associated with HLM, the chapter also provides a foundation to understand the nature and the kinds of variability present in data.

Chapter 2 builds on Chapter 1 by providing a conceptual, nontechnical overview of estimation and model fit issues in multilevel modeling. Being very didactic in their approach, the authors discuss the rationale for using maximum likelihood estimation in multilevel modeling and explain the approach for understanding the reliability of parameter estimates with empirical Bayes estimation. This chapter also introduces hypothesis testing in multilevel modeling and several model index comparison approaches to evaluate the goodness of fit where the researcher compares competing models and the estimation procedures to consider in these model comparisons. Although the chapter is more conceptual than technical, the authors provide many resources for follow-up discussions on concepts illuminated in the chapter.

Chapter 3 highlights the various data sources that lend themselves to multilevel modeling and IR, including a discussion about public, private, and institutional data. This chapter also provides information about some matters that should be taken into account when working with large-scale data sets. Importantly, the authors discuss the benefits and challenges of some software packages commonly used to estimate multilevel models. They also provide a brief word on missing data and how the software packages accommodate missing data.

Chapter 4 discusses multilevel models for binary outcomes, which is becoming increasingly utilized in social science research. This chapter focuses on methods for binary and binomial data and models for categorical data analysis that are adapted to both multilevel and mixed modeling frameworks using generalized linear mixed modeling. To make the text more concrete, the author provides two examples of multilevel modeling with categorical outcomes. One study evaluates the probability of placement into postdoctoral training based on a data set consisting of 40 Ph.D.-granting institutions. The second example uses data from 3,600 graduate students who entered various Ph.D. programs to examine the probability of program dropout for students who entered science, technology, engineering, and math programs.

Chapter 5 provides one example of how cross-classified random effects modeling can be used to assess faculty gender pay differentials in higher education and how those results can be used to help inform policy at the institutional level. Using data collected from the 2004 National Study of Postsecondary Faculty, this chapter provides a good example of an HLM study that has a complex data structure. Results from this chapter provide guidance for understanding macro-level issues and trends and examining institutional policies using HLM techniques.

Chapter 6 provides an example of a study that uses institutional data and multilevel modeling to examine the effects that high school academic performance and first-year college students’ academic persistence have on the likelihood to participate in learning communities. This chapter takes the reader through the multilevel model building process that evolved from preliminary binomial logistic regression models. It also gives an example about how results from the statistical models can have implications for how resources can be distributed and policies created based on how the results are interpreted and presented to audiences that do not necessarily know the multilevel lingo.

The final chapter, Chapter 7, provides a fuller conversation about presenting results. This chapter is an appropriate end to the volume because it provides strategies for presenting complex multilevel data and statistical results to institutional and higher education decision makers. Using two examples of research studies, one predicting first-year college grade point average and the other perception of gains in critical thinking of college seniors, this chapter shows how to communicate results to audiences that may not have prior exposure to statistical models and results from multilevel data. In addition, the chapter gives examples of how charts and graphs can be useful in communicating results.

Joe L. Lott, IIJames S. AntonyEditors

References

Astin, A. W., and Denson, N. “Multi-Campus Studies of College Impact: Which Statistical Method Is Appropriate.” Research in Higher Education, 2009, 50, 354–367.

Delaney, A. M. “The Role of Institutional Research in Higher Education: Enabling Researchers to Meet New Challenges.” Research in Higher Education, 1997, 38(1), 1–16.

Raudenbush, S. W., and Bryk, A. S. Hierarchical Linear Models: Applications and Data Analysts. (2nd ed.). Thousand Oaks, Calif.: Sage Publications, 2002.

Snijders, T., and Bosker, R. Multilevel Analysis. Thousand Oaks, Calif.: Sage Publications, 1999.

Terenzini, P. T. “On the Nature of Institutional Research and the Knowledge and Skills It Requires.” In J. F. Volkwein (ed.), What Is Institutional Research All About? A Critical and Comprehensive Assessment of the Profession. New Directions for Institutional Research, no. 104. San Francisco: Jossey-Bass, 1999.

Volkwein, J. F. “The Foundations and Evolution of Institutional Research.” In D. G. Terkla (ed.), More Than Just Data. New Directions for Higher Education, no. 141. San Francisco: Jossey-Bass, 2008.

1

Hierarchical Data Structures, Institutional Research, and Multilevel Modeling

Ann A. O’Connell, Sandra J. Reed

This chapter provides an introduction to multilevel modeling, including the impact of clustering and the intraclass correlation coefficient. Prototypical research questions in institutional research are examined, and an example is provided to illustrate the application and interpretation of multilevel models.

Introduction

Multilevel modeling (MLM), also referred to as hierarchical linear modeling (HLM) or mixed models, provides a powerful analytical framework through which to study colleges and universities and their impact on students. Due to the natural hierarchical structure of data obtained from students or faculty in colleges and universities, MLM offers many advantages to analysts and policy makers involved in institutional research (IR). This chapter introduces fundamental concepts of hierarchy and its statistical treatment specifically for data structures occurring in IR settings. Our goal is to provide an overview of HLM and set the stage for the chapters that follow as well as highlight the particular advantages of HLM for those involved in IR.

IR professionals routinely encounter the kinds of clustered or nested data structures for which HLM is uniquely suited. Cross-sectional studies of students nested within classes or courses, classes nested within departments or schools, faculty within departments, athletes within sport designations within departments or schools—each of these settings describes lower-level individuals (that is, students or faculty) nested or clustered within one or more higher-level contexts or groups (that is, within classes or within departments). In such cases, the variability in lower-level outcomes (student retention, faculty satisfaction) might be due in part to differences among higher-level groups or contexts (class size, department size, and so on). Analyses of these data using ordinary linear regression methods are problematic, as the underlying structure of the data often leads to violations of the assumptions of independence intrinsic to these models. Through HLM, we are able to model these dependencies and to examine how differential characteristics in the higher-level contexts help to explain variation in individual or lower-level outcomes. An HLM approach can also be used in place of repeated-measures analysis of variance in longitudinal studies. By viewing a series of repeated observations as lower-level outcomes nested within the individual, researchers are able to explore the effects of higher-level individual characteristics (gender, age) on the patterns of change in the lower-level outcomes over time.

Figure 1.1 represents a prototype situation for nested or clustered cross-sectional data from a single institution. In this figure, potential data of interest such as student persistence, gender, or first-year grade point average (GPA) reside at level one, the lowest level of the hierarchy. These level-one characteristics vary across individuals within the same department as well as between departments. Students are nested within different departments, and these departments may vary in terms of supports in place for mentoring new students or size of faculty in that department. These level-two characteristics vary between departments, but they do not vary between students within the same department. Finally, data representing the institution, such as total endowment or selectivity of undergraduate admissions, are common to all departments and all students within departments at that institution; there is no variability at the institutional level for the prototype model shown. Thus, while there seems to be three levels to the hierarchy, the analysis of outcomes at the student level would be examined through a two-level HLM: students nested within departments. More complex structures are easily accommodated in the HLM environment in both cross-sectional and longitudinal studies. If this cross-sectional data collection scheme were implemented at multiple institutions (see Hox, 1998, and Maas and Hox, 2005, for discussion of factors related to sample sizes at different levels), variability in institutional-level characteristics can be measured and examined, and the influence of institutional context as well as departmental context on student outcomes can be examined through a three-level HLM. If repeated observations of the outcome are collected on all students over a four-year period, such as end-of-year GPA, these repeated measurements reside at the lowest level of data collection; they are nested within students, who are nested within departments, and as a result would add a third level to the analytical design.

Figure 1.1.Prototype of Nested or Clustered Data: Students Nested Within Departments

Whether the data of interest are longitudinal or cross-sectional, multilevel analyses are concerned with the study of variation. Just as in standard (single-level) regression, the goal of multilevel analysis is to attempt to explain variability, which implies that the outcome of interest can be reliably modeled through a well-chosen or predefined set of predictors, covariates, or explanatory variables. As the multilevel example illustrates, variability exists at each level of a multilevel analysis, and predictors or explanatory variables can exist at different levels as well. Overall, the primary motivation for employing multilevel analysis is to examine and understand the nature of the many different kinds of variability present in the data (Gelman and Hill, 2007). In doing so, we attempt to model the outcomes of interest by examining how group-level or individual-level characteristics are related to lowest-level outcomes.

An assumption in standard regression is that the observations or data subjected to analysis are statistically independent. With nested data, this assumption is clearly violated. Research has consistently shown that for clustered data, observations obtained from persons within the same cluster tend to exhibit more similarity to each other than to observations from different clusters. This similarity leads to underestimation of the standard errors for regression parameter estimates and inflates Type I error even when the similarity is mild (Donner, Birkett, and Buck, 1981; Sudman, 1985; Kenny and Judd, 1986; Murray and Hannon, 1990; Kish, 1995; Fowler, 2001). Cluster homogeneity is commonly measured through the intraclass correlation coefficient (ICC), which can be interpreted as the familiar Pearson correlation between two observations from the same cluster (Donner and Klar, 2000). In a two-level design, the ICC represents the proportion of total variance in the outcome that is captured by differences between the clusters or groups. When no variability is present between the clusters or groups, the value of the ICC is zero, and the assumption of independence among all individuals in the sample is justified. However, in the presence of between-cluster variability, the value of the ICC is positive, indicating a lack of independence, which invalidates standard regression models where clustering is ignored. The presence of ICC supports the adoption of a multilevel approach to analyzing the data, incorporating critical features of the hierarchical structure of the data into the analysis.

Clustered data may arise due to the existence of intact groups within an institution or by design if, for example, first-year students are randomly assigned to small-group mentoring activities to boost student engagement. Another situation in which IR researchers may be presented with clustered data is through sampling convenience. For example, requesting that all students within randomly selected intact courses complete a survey on first-year experiences would yield a more practical and feasible design relative to a sample based on a random selection of students across the entire college or university. However, such clustered samples have limitations as well as strengths that can affect how data may be interpreted. Whether naturally occurring or by intent, the structure of clustered data involves collecting information from clusters or groups of individuals experiencing a common phenomenon or event. In an IR setting, these common phenomena could arise from attending the same class or being in the same degree program. Regardless of the nature of the cluster, the ICC is found through decomposition of total variance in an outcome of interest into its within-group and between-group components; the ICC represents the proportion of variance that is between groups (Raudenbush and Bryk, 2002).

Historical Approaches to Analyzing Data with a Multilevel Structure

Prior to the advent of specialized software devoted to multilevel data, researchers often used two approaches when confronted with clustered or nested data: aggregation and disaggregation. Although multilevel models may eliminate the particular kinds of bias prevalent in these earlier approaches, we review them here to underscore the need for researchers to avoid the kinds of fallacies that earlier methods may have encouraged and to focus instead on methodologically appropriate and ethical practices for multilevel data (American Statistical Association, 1999; Goldstein, 2011).

In an aggregation approach, researchers sometimes averaged the lower-level data within a cluster or group and then used these averages as outcomes or predictor variables in a single-level analysis model. W. S. Robinson’s seminal 1950 article on ecological correlations (reprinted in the International Journal of Epidemiology, 2009) describes an ecological correlation as the statistical correlation among groups of individuals. It was fairly common at that time to use ecological correlations as if they represented the correlations among the underlying individual-level data; the ecological fallacy refers to the inferential problems inherent in using group-level data to generalize to individual-level relationships. Robinson used 1930 census data to describe correlations among county- or region-level illiteracy rates and the percentage of African Americans in that region. His data showed the ecological correlation to be .946, while the individual-level correlation between illiteracy and race was .203. Since the publication of Robinson’s work, researchers have continued to examine and caution against the ecological fallacy, with implications for the importance of context in multilevel studies (for example, Schwartz, 1994; Susser, 1994; Diez-Roux, 1998; Oakes, 2009; Goldstein, 2011).

In a disaggregation approach, researchers disregarded the tendency for data from persons within distinct groups or geographical regions to be correlated and ignored the multilevel structure of the data completely. Thus, all data was analyzed as if it arose at the individual level, and group-relevant variables would retain the same value for all persons within the same group. Such an approach clearly violates the traditional assumption of independence necessary for valid statistical tests. Consequently, standard errors are underestimated and probability values for statistical tests are too small, leading to the potential for overstating statistical significance of the resulting research findings. These issues have been well documented by sampling methodologists and multilevel researchers (for example, Kish, 1995; Murray, 1998; Raudenbush and Bryk, 2002).

Much of the literature on levels-of-analysis problems has focused on the ecological fallacy, but researchers are also cautioned against the atomistic fallacy, which occurs while drawing inferences based on individual-level data and generalizing these inferences to group-level associations (Diez-Roux, 1998). Both kinds of fallacies can be avoided by careful consideration of the level at which data are collected (individual versus group) and by consistent representation of these levels in the statistical model. Hierarchically structured data, such as those that occur with most institutional research data, are uniquely represented through multilevel models.

Importance of Advancements in Statistical Methods in Institutional Research

Pascarella and Terenzini (1991, 2005) are renowned for their emphasis on methodological rigor in understanding how colleges affect students. Their work documents the importance of remaining current in statistical and research methods for those conducting institutional research. In addition to advancing theories and models for student change, theory development and research must be matched by advances in statistical methodologies. For example, two decades ago, Pascarella and Terenzini’s 1991 volume discussed, in part, the strengths and limitations of the use of meta-analysis as an approach to aggregating and comparing results across research studies. In their 2005 volume, while still characterizing limitations to meta-analysis in their updated literature for the new edition, they specifically recognize the profound advances in statistical methods that have occurred over the past 20 years or so, including MLM. The capacity for multilevel models to strengthen our understanding of how colleges and universities affect students cannot be overemphasized. In the next section, we highlight some of the ways in which multilevel models may be used in institutional research, before turning to our introduction of model notation and interpretation.

Research Questions in Institutional Research

Colleges and universities are complex organizations involving countless interactions among students, faculty, staff, and administration. These interactions occur among organizational entities made up of departments, schools, and colleges, each with unique policies, practices, and values. Observations of student achievement, faculty productivity, and other important performance indicators may be affected by group-level similarities based on these organizational structures. In addition, increased reliance on institutional data for strategic planning, accreditation, accountability, and performance improvement presents a significant challenge for IR professionals (Brittingham, O’Brien, and Alig, 2008; Voorhees, 2008). Similarly, the increasing demand for comparative analysis across institutions for the purpose of performance benchmarking requires that analytical models be developed that accommodate potential heterogeneity across institutions, states, and regions (Yorke, 2010). To effectively assess institutional performance, IR professionals require analytical tools that facilitate comparative analysis across these heterogeneous groups and permit the evaluation of group effects on individual-level performance. By learning and employing multilevel techniques to provide actionable information based in this broad institutional perspective, IR offices can position themselves as key partners in organizational dialogue and decision making (Parmley, 2009).