Total Survey Error in Practice E-Book

• Herausgeber: John Wiley & Sons
• Kategorie: Wissenschaft und neue Technologien
• Serie: Wiley Series in Survey Methodology
• Sprache: Englisch

Seitenzahl: 1448

Veröffentlichungsjahr: 2017

Beschreibung

Featuring a timely presentation of total survey error (TSE), this edited volume introduces valuable tools for understanding and improving survey data quality in the context of evolving large-scale data sets This book provides an overview of the TSE framework and current TSE research as related to survey design, data collection, estimation, and analysis. It recognizes that survey data affects many public policy and business decisions and thus focuses on the framework for understanding and improving survey data quality. The book also addresses issues with data quality in official statistics and in social, opinion, and market research as these fields continue to evolve, leading to larger and messier data sets. This perspective challenges survey organizations to find ways to collect and process data more efficiently without sacrificing quality. The volume consists of the most up-to-date research and reporting from over 70 contributors representing the best academics and researchers from a range of fields. The chapters are broken out into five main sections: The Concept of TSE and the TSE Paradigm, Implications for Survey Design, Data Collection and Data Processing Applications, Evaluation and Improvement, and Estimation and Analysis. Each chapter introduces and examines multiple error sources, such as sampling error, measurement error, and nonresponse error, which often offer the greatest risks to data quality, while also encouraging readers not to lose sight of the less commonly studied error sources, such as coverage error, processing error, and specification error. The book also notes the relationships between errors and the ways in which efforts to reduce one type can increase another, resulting in an estimate with larger total error. This book: * Features various error sources, and the complex relationships between them, in 25 high-quality chapters on the most up-to-date research in the field of TSE * Provides comprehensive reviews of the literature on error sources as well as data collection approaches and estimation methods to reduce their effects * Presents examples of recent international events that demonstrate the effects of data error, the importance of survey data quality, and the real-world issues that arise from these errors * Spans the four pillars of the total survey error paradigm (design, data collection, evaluation and analysis) to address key data quality issues in official statistics and survey research Total Survey Error in Practice is a reference for survey researchers and data scientists in research areas that include social science, public opinion, public policy, and business. It can also be used as a textbook or supplementary material for a graduate-level course in survey research methods.

Leseprobe

Table of Contents

Cover

Title Page

Notes on Contributors

Preface

Section 1: The Concept of TSE and the TSE Paradigm

1 The Roots and Evolution of the Total Survey Error Concept

1.1 Introduction and Historical Backdrop

1.2 Specific Error Sources and Their Control or Evaluation

1.3 Survey Models and Total Survey Design

1.4 The Advent of More Systematic Approaches Toward Survey Quality

1.5 What the Future Will Bring

References

2 Total Twitter Error

2.1 Introduction

2.2 Social Media: An Evolving Online Public Sphere

2.3 Components of Twitter Error

2.4 Studying Public Opinion on the Twittersphere and the Potential Error Sources of Twitter Data: Two Case Studies

2.5 Discussion

2.6 Conclusion

References

3 Big Data

3.1 Introduction

3.2 Definitions

3.3 The Analytic Challenge: From Database Marketing to Big Data and Data Science

3.4 Assessing Data Quality

3.5 Applications in Market, Opinion, and Social Research

3.6 The Ethics of Research Using Big Data

3.7 The Future of Surveys in a Data‐Rich Environment

References

4 The Role of Statistical Disclosure Limitation in Total Survey Error

4.1 Introduction

4.2 Primer on SDL

4.3 TSE‐Aware SDL

4.4 Edit‐Respecting SDL

4.5 SDL‐Aware TSE

4.6 Full Unification of Edit, Imputation, and SDL

4.7 “Big Data” Issues

4.8 Conclusion

Acknowledgments

References

Section 2: Implications for Survey Design

5 The Undercoverage–Nonresponse Tradeoff

5.1 Introduction

5.2 Examples of the Tradeoff

5.3 Simple Demonstration of the Tradeoff

5.4 Coverage and Response Propensities and Bias

5.5 Simulation Study of Rates and Bias

5.6 Costs

5.7 Lessons for Survey Practice

References

6 Mixing Modes

6.1 Introduction

6.2 The Effect of Offering a Choice of Modes

6.3 Getting People to Respond Online

6.4 Sequencing Different Modes of Data Collection

6.5 Separating the Effects of Mode on Selection and Reporting

6.6 Maximizing Comparability Versus Minimizing Error

6.7 Conclusions

References

7 Mobile Web Surveys

7.1 Introduction

7.2 Coverage

7.3 Nonresponse

7.4 Measurement Error

7.5 Links Between Different Error Sources

7.6 The Future of Mobile web Surveys

References

8 The Effects of a Mid‐Data Collection Change in Financial Incentives on Total Survey Error in the National Survey of Family Growth

8.1 Introduction

8.2 Literature Review: Incentives in Face‐to‐Face Surveys

8.3 Data and Methods

8.4 Results

8.5 Conclusion

References

9 A Total Survey Error Perspective on Surveys in Multinational, Multiregional, and Multicultural Contexts

9.1 Introduction

9.2 TSE in Multinational, Multiregional, and Multicultural Surveys

9.3 Challenges Related to Representation and Measurement Error Components in Comparative Surveys

9.4 QA and QC in 3MC Surveys

References

10 Smartphone Participation in Web Surveys

10.1 Introduction

10.2 Prevalence of Smartphone Participation in Web Surveys

10.3 Smartphone Participation Choices

10.4 Instrument Design Choices

10.5 Device and Design Treatment Choices

10.6 Conclusion

10.7 Future Challenges and Research Needs

Appendix 10.A: Data Sources

Appendix 10.B: Smartphone Prevalence in Web Surveys

Appendix 10.C: Screen Captures from Peterson et al. (2013) Experiment

Appendix 10.D: Survey Questions Used in the Analysis of the Peterson et al. (2013) Experiment

References

11 Survey Research and the Quality of Survey Data Among Ethnic Minorities

11.1 Introduction

11.2 On the Use of the Terms

Ethnicity

and

Ethnic Minorities

11.3 On the Representation of Ethnic Minorities in Surveys

11.4 Measurement Issues

11.5 Comparability, Timeliness, and Cost Concerns

11.6 Conclusion

References

Section 3: Data Collection and Data Processing Applications

12 Measurement Error in Survey Operations Management

12.1 TSE Background on Survey Operations

12.2 Better and Better: Using Behavior Coding (CARIcode) and Paradata to Evaluate and Improve Question (Specification) Error and Interviewer Error

12.3 Field‐Centered Design: Mobile App for Rapid Reporting and Management

12.4 Faster and Cheaper: Detecting Falsification With GIS Tools

12.5 Putting It All Together: Field Supervisor Dashboards

12.6 Discussion

References

13 Total Survey Error for Longitudinal Surveys

13.1 Introduction

13.2 Distinctive Aspects of Longitudinal Surveys

13.3 TSE Components in Longitudinal Surveys

13.4 Design of Longitudinal Surveys from a TSE Perspective

13.5 Examples of Tradeoffs in Three Longitudinal Surveys

13.6 Discussion

References

14 Text Interviews on Mobile Devices

14.1 Texting as a Way of Interacting

14.2 Contacting and Inviting Potential Respondents through Text

14.3 Texting as an Interview Mode

14.4 Costs and Efficiency of Text Interviewing

14.5 Discussion

References

15 Quantifying Measurement Errors in Partially Edited Business Survey Data

15.1 Introduction

15.2 Selective Editing

15.3 Effects of Errors Remaining After SE

15.4 Case Study: Foreign Trade in Goods Within the European Union

15.5 Editing Big Data

15.6 Conclusions

References

Section 4: Evaluation and Improvement

16 Estimating Error Rates in an Administrative Register and Survey Questions Using a Latent Class Model

16.1 Introduction

16.2 Administrative and Survey Measures of Neighborhood

16.3 A Latent Class Model for Neighborhood of Residence

16.4 Results

16.5 Discussion and Conclusion

Appendix 16.A: Program Input and Data

Acknowledgments

References

17 ASPIRE

17.1 Introduction and Background

17.2 Overview of ASPIRE

17.3 The ASPIRE Model

17.4 Evaluation of Registers

17.5 National Accounts

17.6 A Sensitivity Analysis of GDP Error Sources

17.7 Concluding Remarks

Appendix 17.A: Accuracy Dimension Checklist

References

18 Classification Error in Crime Victimization Surveys

18.1 Introduction

18.2 Background

18.3 Analytic Approach

18.4 Model Selection

18.5 Results

18.6 Discussion and Summary of Findings

18.7 Conclusions

Appendix 18.A: Derivation of the Composite False‐Negative Rate

Appendix 18.B: Derivation of the Lower Bound for False‐Negative Rates from a Composite Measure

Appendix 18.C: Examples of Latent GOLD Syntax

References

19 Using Doorstep Concerns Data to Evaluate and Correct for Nonresponse Error in a Longitudinal Survey

19.1 Introduction

19.2 Data and Methods

19.3 Results

19.4 Discussion

Acknowledgment

References

20 Total Survey Error Assessment for Sociodemographic Subgroups in the 2012 U.S. National Immunization Survey

20.1 Introduction

20.2 TSE Model Framework

20.3 Overview of the National Immunization Survey

20.4 National Immunization Survey: Inputs for TSE Model

20.5 National Immunization Survey TSE Analysis

20.6 Summary

References

21 Establishing Infrastructure for the Use of Big Data to Understand Total Survey Error

Overview

Part 1 Big Data Infrastructure at the Institute for Employment Research (IAB)

References

Part 2 Using Administrative Records Data at the U.S. Census Bureau: Lessons Learned from Two Research Projects Evaluating Survey Data

References

Part 3 Statistics New Zealand’s Approach to Making Use of Alternative Data Sources in a New Era of Integrated Data

References

Part 4 Big Data Serving Survey Research: Experiences at the University of Michigan Survey Research Center

References

Section 5: Estimation and Analysis

22 Analytic Error as an Important Component of Total Survey Error

22.1 Overview

22.2 Analytic Error as a Component of TSE

22.3 Appropriate Analytic Methods for Survey Data

22.4 Methods

22.5 Results

22.6 Discussion

Acknowledgments

References

23 Mixed‐Mode Research

23.1 Introduction

23.2 Designing Mixed‐Mode Surveys

23.3 Literature Overview

23.4 Diagnosing Sources of Error in Mixed‐Mode Surveys

23.5 Adjusting for Mode Measurement Effects

23.6 Conclusion

References

24 The Effect of Nonresponse and Measurement Error on Wage Regression across Survey Modes

24.1 Introduction

24.2 Nonresponse and Response Bias in Survey Statistics

24.3 Data and Methods

24.4 Results

24.5 Summary and Conclusion

Acknowledgments

Appendix 24.A

Appendix 24.B

References

25 Errors in Linking Survey and Administrative Data

25.1 Introduction

25.2 Conceptual Framework of Linkage and Error Sources

25.3 Errors Due to Linkage Consent

25.4 Erroneous Linkage with Unique Identifiers

25.5 Erroneous Linkage with Nonunique Identifiers

25.6 Applications and Practical Guidance

25.7 Conclusions and Take‐Home Points

References

Wiley Series in Survey Methodology

Index

End User License Agreement

List of Tables

Chapter 02

Table 2.1 Characteristics of Twitter data error.

Chapter 04

Table 4.1 WSSM‐calculated Kullback–Leibler divergences of sample (with no measurement error), unit respondents (with no measurement error), final responses (with measurement error), and post‐SDL data to the original population.

Table 4.2 Number of records that violate edit rules across the 20 replications (or single realizations for Mic and MMic) after implementing SDL.

Table 4.3 Left: original tabular dataset

O

. Right: masked tabular dataset

M

, after primary and secondary cell suppression.

Chapter 05

Table 5.1 Coverage rates and response rates for two survey approaches.

Table 5.2 Parameters varied in simulations.

Table 5.3 Description of scenarios.

Chapter 06

Table 6.1 Overall and Internet response rates, by type of area and experimental condition.

Table 6.2 Overall response rates, by mode sequence and study.

Table 6.3 Estimated percentage of voters in 2006 election, by experimental variables.

Chapter 07

Table 7.1 Response rates for PC and mobile web.

Table 7.2 Breakoff rates (%) for PC and mobile web users.

Table 7.3 Measurement error in mobile web surveys: main findings and possible directions for further research.

Chapter 08

Table 8.1 Phase‐One weighted screener, main, and combined response rates.

Table 8.2 Phase‐One weighted main interview response rates by demographic subgroup.

Table 8.3 Phase Two and final‐weighted screener, main, and combined response rates by Phase‐One incentive.

Table 8.4 Final‐weighted main interview response rates.

Table 8.5 Selected key statistics by sex.

Table 8.6 Unweighted case counts by Phase-One incentive.

Table 8.7 Selected effort indicators.

Table 8.8 Reporting discrepancies between CAPI and ACASI by incentive treatment.

Table 8.9 $60 incentive predicting ordinal discrepancy outcome with and without covariates.

Table 8.10

p

‐Values for quintiles of the main interview response propensity as a predictor of the ordinal discrepancy variable in the ordinal logistic regression model.

Chapter 10

Table 10.1 Prevalence (%) of survey starts by device and treatment.

Table 10.2 Prevalence (%) of survey completions by switching behavior and treatment.

Table 10.3 Description of experimental conditions.

Table 10.4 Estimated odds ratios for logistic regression models predicting top‐box or top‐two‐box score.

Table 10.B.1 Smartphone prevalence in web surveys—customer list samples.

Table 10.B.2 Smartphone prevalence in web surveys—institutional rosters.

Table 10.B.3 Smartphone prevalence in web surveys—probability‐based panels.

Table 10.B.4 Smartphone prevalence in web surveys—probability–based panels.

Chapter 13

Table 13.1 Nonresponse and coverage error over the course of the LISS panel.

Table 13.2 Flow through employment section of the 1996 SIPP questionnaire.

Chapter 14

Table 14.1 Properties of mobile voice and text communication, originally appeared in Schober et al. (2015).

Table 14.2 Absolute coverage bias estimates of texter population relative to U.S. mobile phone owners (2013) for six sociodemographic statistics.

Chapter 15

Table 15.1 Sum of absolute errors in observations of invoiced values found by SE and estimated sum of absolute errors in remaining observations (million SEK, standard errors in parenthesis).

Table 15.2 Net change by SE and estimated remaining error in total estimates of invoiced values (million SEK, standard errors in parenthesis).

Chapter 16

Table 16.1 Some official statistics on neighborhoods of Amsterdam (2011–2014).

Table 16.2 Cross tabulation of the observed survey measures with the administrative measure.

Table 16.3 Model fit measures for different specifications of the latent class model.

Table 16.4 Classification rates for the three observed measures (Model 6).

Chapter 17

Table 17.1 Sources of error considered by product for ASPIRE at Statistics Sweden.

Table 17.2 Ratings matrix for the TPR.

Table 17.3 Ratings matrix for the quarterly GDP.

Table 17.4 The final supply and use of product (CPA62) in 1,000,000 SEK for the years 2008–2012.

Table 17.5 The impact on GDP if the bias in the price of product CPA62 is −5, −10, +5, or +10% for all components in Table 17.4.

Table 17.6 The final supply and use of the product CPA291.

Table 17.7 The impact on GDP if the bias in the price of product CPA291 is −2.5, −5, +2.5, or +5% for production (S1) only in Table 17.4.

Chapter 18

Table 18.1 Fit statistics for MLC models of less serious crimes against an individual (models by sets of waves included).

Table 18.2 Fit statistics for MLC models of crimes against a household (models by sets of waves included).

Table 18.3 Estimated classification errors by wave and sets of waves from MLC models for less serious crimes against an individual.

Table 18.4 Estimated classification errors by wave and sets of waves from MLC models for crimes against a household.

Table 18.5 Evidence rating of classification error differences for demographic variables.

Table 18.6 Average estimated classification error rates for demographic variables with a strong evidence rating.

Chapter 19

Table 19.1 Concerns groups assignment with sample sizes and proxy measures of reluctance, using Wave 2 data.

Table 19.2 Unit response rates and efforts by concerns group.

Table 19.3 Mean number of “Don’t Know” and “Refused” answers by concerns group.

Table 19.4 Total and component nonresponse biases for five expenditure estimates.

Table 19.5 Total and component nonresponse biases in housing and education expenditure estimates.

Table 19.6 Total and component nonresponse biases estimates for five expenditures.

Table 19.7 Expenditure estimates in dollars (and standard errors) with and without weighting adjustments for nonresponse using doorstep concerns data.

Chapter 20

Table 20.1 Vaccines monitored by the National Immunization Survey (NIS), recommended doses, and 2012 national‐level vaccination coverage rates.

Table 20.2 2012 national‐level vaccination coverage rates by domains defined by important sociodemographic characteristics of the child collected in the household interview (2012 National Immunization Survey (NIS)).

Table 20.3 Total survey error model inputs by stages (2012 National Immunization Survey).

Table 20.4 Estimated mean TSE and 95% credible interval for TSE using the three‐stage model for national‐level vaccination coverage rates corresponding to 4 : 3 : 1 : 3 : 3 : 1, 4+DTaP, and 1+MMR (2012 National Immunization Survey).

Table 20.5 Means and 95% credible intervals for four component errors in national‐level vaccination coverage rates (2012 National Immunization Survey).

Chapter 21

Table 21.4.1 Proportion of housing units (HU) by commercial list substratum with one or more persons and actual number of persons in age ranges 18–44 and 65+ based on NSFG screener responses.

Table 21.4.2 Proportion of teachers identified as eligible by commercial educational database compared to MTTS screener results for schools with matched NCES IDs.

Chapter 22

Table 22.1 Variables coded for each of the 100 peer‐reviewed articles in the study.

Table 22.2 Chi‐square tests for bivariate associations of each of the estimation errors with whether or not the journal has a statistical consultant on the editorial board.

Table 22.3 Chi‐square tests for bivariate associations of software errors with whether or not the journal has a statistical consultant on the editorial board.

Chapter 23

Table 23.1 Multinomial regression predicting mode (face‐to‐face is reference category).

Table 23.2 Results of measurement equivalence testing, without and with adjustment.

Table 23.3 Estimates of means and variances of the Parenthood factor, unweighted and weighted data (scalar model).

Table 23.4 Means and variances of observed Parenthood scores with and without adjustment.

Table 23.5 Estimated means and variances of latent Parenthood factor unadjusted and with adjustment for mode measurement effect.

Table 23.6 Means and variances of observed Parenthood scores unadjusted and with adjustment for mode measurement effect.

Table 23.7 Means and variances of Parenthood score unadjusted and with adjustment for mode measurement effect.

Table 23.8 Estimated means and variances of latent Parenthood factor unadjusted and with adjustment for mode measurement effect.

Chapter 24

Table 24.1 Response rates across modes of data collection.

Table 24.A.1 Wage regression: coefficients, absolute and relative deviations (coverage web sample).

Table 24.A.2 Wage regression: coefficients, absolute and relative deviations (coverage tel. sample).

Table 24.A.3 Wage regression: coefficients, absolute and relative deviations (coverage pooled sample).

Table 24.B.1 Wage regression: coefficients, absolute and relative deviations (web sample).

Table 24.B.2 Weighted wage regression: coefficients, absolute and relative deviations (web sample).

Table 24.B.3 Wage regression: coefficients, absolute and relative deviations (telephone sample).

Table 24.B.4 Weighted wage regression: coefficients, absolute and relative deviations (telephone sample).

Table 24.B.5 Wage regression: coefficients, absolute and relative deviations (pooled sample, unweighted).

Chapter 25

Table 25.1 A selection of large‐scale surveys that link survey records to administrative records.

List of Illustrations

Chapter 01

Figure 1.1 The generic statistical business process model.

Figure 1.2 Subjective sample of events in the evolution of the concept of TSE.

Chapter 02

Figure 2.1 Theoretical spaces of Twitter data error.

Figure 2.2 Percent of online U.S. adults who use Twitter as of 2014‥

Figure 2.3 Marijuana legalization tweet volume by month, worldwide, U.S.A. and Colorado, basic query (2011–2014).

Figure 2.4 Pro‐choice/life tweet volume by month, worldwide, U.S. and Colorado, basic query (2011–2014).

Figure 2.5 Marijuana legalization tweet volume by month, worldwide, basic query, including and excluding RTs, 2011–2014.

Figure 2.6 Pro‐choice/life tweet volume by month, worldwide, basic query, including and excluding RTs, 2011–2014.

Figure 2.7 Marijuana legalization tweet volume by month, worldwide, basic vs. expanded query (2011–2014).

Figure 2.8 Pro‐choice/life tweet volume by month, worldwide, basic vs. expanded query (2011–2014).

Figure 2.9 Marijuana legalization Twitter and Gallup sentiment by month, worldwide, basic query, 2011–2014.

Figure 2.10 Pro‐choice/life Twitter and Gallup sentiment by month, worldwide, basic query, 2011–2014.

Chapter 03

Figure 3.1 Word cloud based on Dutcher (2014).

Figure 3.2 Infographic from James (2014).

Figure 3.3 Sample data visualization‥

Figure 3.4 Simplified view of data linkage.

Chapter 04

Figure 4.1 The role of SDL within the TSE paradigm, modeled on Groves (1989).

Figure 4.2 Forms of dissemination of confidential official statistics data.

Figure 4.3 Schematic representation of risk‐utility paradigms. Candidate releases on the efficient frontier are connected by dotted lines. The large point is optimal for the value function shown.

Figure 4.4 Risk‐distortion scatterplots for 108 candidate releases from the CPS‐8D database. Three swap rates (1%—circles, 2%—triangles, and 10%—plus signs) are shown, and for each, there are 36 candidate releases representing all choices of one or two swap variables.

Figure 4.5 Edit violations created by four SDL methods. Two variables are depicted: SL = number of hours of skilled labor and SW = wages paid to skilled labor. The dotted lines represent range and ratio edit constraints. Top left: original data, from the 1991 Columbian manufacturing survey. Top right: SDL method = additive noise. Bottom left: SDL method = rank swapping. Bottom right: SDL method = microaggregation followed by additive noise.

Figure 4.6 Risk‐utility map for the Colombian manufacturing survey data‐based simulation. The solid line indicates the risk‐utility frontier. The open symbols represent edit‐after‐SDL approaches, and the solid symbols represent edit‐preserving SDL approaches. Smaller values of PL1 and

U

prop

represent the higher levels of data protection and data utility. PL1 is the risk arising from linkage to the “nearest” record being correct;

U

prop

is the propensity score utility.

Figure 4.7 For simulated data, scatterplot of correlations among nine log‐transformed variables following edit–imputation against correlations in true data. Abbreviations: AAI, all active items; BE, Bayes edit; BE‐min, Bayes edit applied to minimal number of fields; FH, Fellegi–Holt.

Figure 4.8 For the CM data, scatterplot of correlations among 27 variables following edit–imputation against correlations in edit passing records. Abbreviations: BE, Bayes edit; BE‐min, Bayes edit applied to minimal number of fields; FC, final released data.

Figure 4.9 The effect on indexed microaggregation on stratum‐weight relationships. Left: beforehand, weight discloses stratum completely. Right: afterward, weight only places stratum in one of two classes.

Chapter 05

Figure 5.1 Diagram of relationship between CPs and RPs and survey variable

Y

.

Figure 5.2 Simulation results: coverage and response rates.

Figure 5.3 Simulation results: coverage rates and bias. See Table 5.3 for descriptions of the scenarios.

Chapter 06

Figure 6.1 Costs per household (in 2010 dollars) and mail return rates for the U.S. Decennial Census since 1970. The dashed line represents costs; the solid line, the return rate‥

Chapter 08

Figure 8.1 The ratio of standard errors under the $40 and $60 treatments across 16 key statistics for males, females, and age, race, and ethnicity subgroups.

Chapter 09

Figure 9.1 TSE (a) representation and (b) measurement in a cross‐cultural context.

Figure 9.2 The translation; review; adjudication; pretesting; and documentation model.

Chapter 10

Figure 10.1 Device and design choices and associated survey errors.

Figure 10.2 Prevalence of smartphone starts from the Decipher platform’s highest volume accounts.

Figure 10.B.1 Prevalence of smartphone starts ordered by prevalence within year: customer list samples.

Figure 10.B.2 Prevalence of smartphone starts: institutional roster samples.

Figure 10.B.3 Prevalence of smartphone starts: probability panel samples.

Figure 10.B.4 Prevalence of smartphone starts: opt‐in panel samples.

Figure 10.C.1 Smartphone/Legacy (screen captures from an iPhone 5).

Figure 10.C.2 Smartphone/New (screen captures from Blackberry Torch).

Figure 10.C.3 Smartphone/Drop‐down (screen captures from iPhone 4s).

Figure 10.C.4 Smartphone/Numeric (screen captures from iPhone 4s, Blackberry Torch).

Figure 10.C.5 Smartphone/Slider (screen captures from iPhone 5).

Figure 10.C.6 PC/Legacy (screen capture Windows 7 PC).

Figure 10.C.7 PC/New (screen capture Windows 7 PC).

Chapter 12

Exhibit 12.1 Example of a question‐level CARI coding screen.

Exhibit 12.2 Fictionalized screenshots for MyCases in the mFOS app.

Figure 12.1 Differences between recorded time and actual time on laptops.

Figure 12.2 Time of electronic record of contacts (EROC) entry by device.

Exhibit 12.3 Visualization of GIS data to detect falsification.

Exhibit 12.4 Sample of dashboard alerts.

Exhibit 12.5 Annotated example of field supervisor dashboard.

Exhibit 12.6 Example of a CARI portlet on the dashboard.

Chapter 13

Figure 13.1 Proportion of “don’t know” and “refusal” answers to personal gross income question in last five waves before attrition in BHPS.

Chapter 14

Figure 14.1 Texting rates among U.S. adults based on Pew telephone surveys from December 2007 through April 2013.

Figure 14.2 Data quality in text versus voice and human versus automated interviews; (a) rounding, (b) straightlining, and (c) disclosure.

Figure 14.3 Interview duration (median duration of question–answer sequences in minutes) and median number of turns per survey question in text and voice interviews.

Chapter 15

Figure 15.1 Cumulative distribution of score

g

k

, normalized to between 0 and 1. The graph is limited to the 10,000 highest scores monthly in 2014. Some 500 highest scores are excluded for scalability. Each isarithm line corresponds to 50 units.

Figure 15.2 Plot of log of sum of absolute value of errors against log of sum of scores. Each sum of 100 units classified by order of the score. Crosses, sample exports; dots, SE imports; pyramids, sample imports; stars, SE exports.

Chapter 16

Figure 16.1 Neighborhoods of residence (“stadsdeel”) in Amsterdam, the Netherlands.

Figure 16.2 Formulation of the 2013 Sportmonitor survey questions on neighborhood of residence. The postcode (ZIP code) is asked at the most detailed level possible.

Figure 16.3 Latent class measurement model for three measurements of neighborhood of residence. Latent variables shown in shaded ovals, observed variables as rectangles.

Figure 16.4 Model‐based estimates of the proportion of residents within each (true) neighborhood who received one of the survey modes (relationship between

x

and

S

).

Figure 16.5 Estimated correct classification probabilities (excluding missing) by survey mode for the two survey measures.

Figure 16.6 Estimated conditional probabilities of choosing each category within method classes (profile plot). Method class 1 is estimated to contain 9%, and Method class 2 is estimated to contain 91% of respondents.

Figure 16.7 Scatterplot of a constructed neighborhood SES score (see Table 16.2) and the log odds ratio (logistic regression coefficient) of choosing each neighborhood when the Method class equals 1. A, Centrum; E, West; F, Nieuw–West; K, Zuid; M, Oost; N, Noord; T, Zuidoost.

Chapter 17

Figure 17.1 Process for estimating GDP by current and constant price approaches.

Chapter 18

Figure 18.1 Illustration of relationship between NCVS interview waves and analysis groups.

Chapter 19

Figure 19.1 Screenshot of CHI used for the Consumer Expenditure Interview Survey.

Chapter 20

Figure 20.1 Distribution of total survey error (TSE) for the 4 : 3 : 1 : 3 : 3 : 1

a

vaccination coverage rate (2012 National Immunization Survey). (Note:

a

4 or more doses of diphtheria and tetanus toxoids and acellular pertussis vaccine (DTaP/DTP/DT); 3 or more doses of polio vaccine; 1 or more doses of measles‐containing vaccine, 3 or more doses of

Haemophilus influenzae

Type b (Hib)‐containing vaccine, 3 or more doses of Hepatitis B vaccine, and 1 or more doses of varicella vaccine (at or after 12 months of age).)

Figure 20.2 Mean TSE and 95% credible intervals using the two‐stage model (incorporating sample‐frame coverage and nonresponse error only) for national‐level vaccination coverage rates corresponding to 4 : 3 : 1 : 3 : 3 : 1

a

, 4+DTaP

b

, and 1+MMR

c

(2009–2012 National Immunization Survey). (Note:

a

4 or more doses of diphtheria and tetanus toxoids and acellular pertussis vaccine (DTaP/DTP/DT); 3 or more doses of polio vaccine; 1 or more doses of measles‐containing vaccine, 3 or more doses of

Haemophilus influenzae

Type b (Hib)‐containing vaccine, 3 or more doses of Hepatitis B vaccine, and 1 or more doses of varicella vaccine (at or after 12 months of age).

b

4 or more doses of diphtheria and tetanus toxoids and acellular pertussis vaccine (DTaP/DTP/DT).

c

1 or more doses of measles/mumps/rubella vaccine (MMR).)

Figure 20.3 Mean TSE using the two‐ and three‐stage models for national‐level vaccination coverage rates corresponding to 4 : 3 : 1 : 3 : 3 : 1

a

, 4+DTaP

b

and 1+MMR

c

(2012 National Immunization Survey). (Note: Mean TSE estimate for the overall group area based solely on 2012 data.

a

4 or more doses of diphtheria and tetanus toxoids and acellular pertussis vaccine (DTaP/DTP/DT); 3 or more doses of polio vaccine; 1 or more doses of measles‐containing vaccine, 3 or more doses of

Haemophilus influenzae

Type b (Hib)‐containing vaccine, 3 or more doses of Hepatitis B vaccine, and 1 or more doses of varicella vaccine (at or after 12 months of age).

b

4 or more doses of diphtheria and tetanus toxoids and acellular pertussis vaccine (DTaP/DTP/DT).

c

1 or more doses of measles/mumps/rubella vaccine (MMR).)

Figure 20.4 Mean TSE and 95% credible intervals using the three‐stage model for vaccination coverage rates corresponding to 4 : 3 : 1 : 3 : 3 : 1

a

, 4+DTaP

b

, and 1+MMR

c

by child’s race/ethnicity (2011–2012 National Immunization Survey). (Note: Mean TSE estimate for the overall group area based solely on 2012 data.

a

4 or more doses of diphtheria and tetanus toxoids and acellular pertussis vaccine (DTaP/DTP/DT); 3 or more doses of polio vaccine; 1 or more doses of measles‐containing vaccine, 3 or more doses of

Haemophilus influenzae

Type b (Hib)‐containing vaccine, 3 or more doses of Hepatitis B vaccine, and 1 or more doses of varicella vaccine (at or after 12 months of age).

b

4 or more doses of diphtheria and tetanus toxoids and acellular pertussis vaccine (DTaP/DTP/DT).

c

1 or more doses of measles/mumps/rubella vaccine (MMR).)

Figure 20.5 Mean TSE and 95% credible intervals using the three‐stage model for vaccination coverage rates corresponding to 4 : 3 : 1 : 3 : 3 : 1

a

, 4+DTaP

b

and 1+MMR

c

by child’s mother’s education (2011–2012 National Immunization Survey). (Note: Mean TSE estimate for the overall group area based solely on 2012 data.

a

4 or more doses of diphtheria and tetanus toxoids and acellular pertussis vaccine (DTaP/DTP/DT); 3 or more doses of polio vaccine; 1 or more doses of measles‐containing vaccine, 3 or more doses of

Haemophilus influenzae

Type b (Hib)‐containing vaccine, 3 or more doses of Hepatitis B vaccine, and 1 or more doses of varicella vaccine (at or after 12 months of age).

b

4 or more doses of diphtheria and tetanus toxoids and acellular pertussis vaccine (DTaP/DTP/DT).

c

1 or more doses of measles/mumps/rubella vaccine (MMR).)

Figure 20.6 Mean TSE and 95% credible interval using the three‐stage model for vaccination coverage rates corresponding to 4 : 3 : 1 : 3 : 3 : 1

a

, 4+DTaP

b

and 1+MMR

c

by child’s sex (2011–2012 National Immunization Survey). (Note: Mean TSE estimate for the overall group area based solely on 2012 data.

a

4 or more doses of diphtheria and tetanus toxoids and acellular pertussis vaccine (DTaP/DTP/DT); 3 or more doses of polio vaccine; 1 or more doses of measles‐containing vaccine, 3 or more doses of

Haemophilus influenzae

Type b (Hib)‐containing vaccine, 3 or more doses of Hepatitis B vaccine, and 1 or more doses of varicella vaccine (at or after 12 months of age).

b

4 or more doses of diphtheria and tetanus toxoids and acellular pertussis vaccine (DTaP/DTP/DT).

c

1 or more doses of measles/mumps/rubella vaccine (MMR).)

Figure 20.7 Mean TSE and 95% credible intervals using the three‐stage model for vaccination coverage rates corresponding to 4 : 3 : 1 : 3 : 3 : 1

a

, 4+DTaP

b

and 1+MMR

c

by child’s MSA status (2011–2012 National Immunization Survey). (Note: Mean TSE estimate for the overall group area based solely on 2012 data.

a

4 or more doses of diphtheria and tetanus toxoids and acellular pertussis vaccine (DTaP/DTP/DT); 3 or more doses of polio vaccine; 1 or more doses of measles‐containing vaccine, 3 or more doses of

Haemophilus influenzae

Type b (Hib)‐containing vaccine, 3 or more doses of Hepatitis B vaccine, and 1 or more doses of varicella vaccine (at or after 12 months of age).

b

4 or more doses of diphtheria and tetanus toxoids and acellular pertussis vaccine (DTaP/DTP/DT).

c

1 or more doses of measles/mumps/rubella vaccine (MMR).)

Chapter 21

Figure 21.1.1 Comparison of different wage records.

Chapter 22

Figure 22.1 Percentages of all potential errors identified in the 100 articles, by type of error.

Figure 22.2 Pearson correlations between the number of errors of a particular type and the impact factor of the journal (

*

p

 < 0.10).

Figure 22.3 Pearson correlations between indicators of particular types of errors and journal impact factors (see Table 22.1 for indicator names;

**

p

 < 0.05,

*

p

 < 0.10).

Figure 22.4 Proportions of articles with particular types of estimation errors over time.

Figure 22.5 Proportions of articles with particular types of quality errors over time.

Figure 22.6 Proportions of articles with particular types of software errors over time.

Figure 22.7 Proportions of articles with particular types of language/presentation errors over time.

Figure 22.8 Percentages of all possible errors identified in the articles in each time period, for each type of error.

Chapter 23

Figure 23.1 Covariate adjustment in a CFA. Note that

b1–b4

indicate equality constraints on regression coefficients across groups, and

c

indicates an equality constraint on the covariance.

Figure 23.2 The mixture model for the extended mixed‐mode survey.

Figure 23.3 Path diagram for adjustment model, eta is the latent Parenting factor.

Chapter 24

Figure 24.1 Wage regression coefficients (web sample).

Figure 24.2 Wage regression coefficients (telephone sample).

Figure 24.3 Weighted wage regression coefficients (web sample).

Figure 24.4 Weighted wage regression coefficients (telephone sample).

Figure 24.5 Weighted wage regression coefficients (pooled sample, unweighted).

Chapter 25

Figure 25.1 Conceptual framework of the record linkage process.

Guide

Cover

Table of Contents

Begin Reading

Pages

ii

iii

iv

xix

xx

xxi

xxii

xxiii

xxiv

xxv

xxvi

xxvii

1

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

123

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

281

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

367

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

467

465

466

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

486

485

515

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

603

604

605

WILEY SERIES IN SURVEY METHODOLOGY

Established in Part by Walter A. Shewhart and Samuel S. Wilks

Editors: Mick P. Couper, Graham Kalton, Lars Lyberg, J. N. K. Rao, Norbert Schwarz, Christopher Skinner

Editor Emeritus: Robert M. Groves

A complete list of the titles in this series appears at the end of this volume.

Total Survey Error in Practice

Edited by

Paul P. Biemer

RTI International and University of North Carolina

Edith de Leeuw

Utrecht University

Stephanie Eckman

RTI International

Brad Edwards

Westat

Frauke Kreuter

Joint Program in Survey Methodology, University of Mannheim, Institute for Employment Research (Germany)

Lars E. Lyberg

Inizio

N. Clyde Tucker

American Institutes for Research

Brady T. West

University of Michigan and Joint Program in Survey Methodology

Copyright © 2017 by John Wiley & Sons, Inc. All rights reserved

Published by John Wiley & Sons, Inc., Hoboken, New JerseyPublished simultaneously in Canada

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, [phone], fax [phone], or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, [phone], fax [phone], or online at http://www.wiley.com/go/permissions.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at [phone], outside the United States at [phone] or fax [phone].

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com.

Library of Congress Cataloging-in-Publication Data

Names: Biemer, Paul P., editor. | Leeuw, Edith de, editor. | Eckman, Stephanie, editor. | Edwards, Brad, editor. | Kreuter, Frauke, editor. | Lyberg, Lars E., editor. | Tucker, N. Clyde, editor. | West, Brady T., editor.Title: Total survey error in practice / edited by Paul P. Biemer, Edith de Leeuw, Stephanie Eckman, Brad Edwards, Frauke Kreuter, Lars E. Lyberg, N. Clyde Tucker, Brady T. West.Description: Hoboken, New Jersey : John Wiley & Sons, 2017. | Includes index.Identifiers: LCCN 2016031564 | ISBN 9781119041672 (cloth) | ISBN 9781119041696 (epub)Subjects: LCSH: Error analysis (Mathematics) | Surveys.Classification: LCC QA275 .T685 2016 | DDC 001.4/33–dc23LC record available at https://lccn.loc.gov/2016031564

Notes on Contributors

Manfred AntoniResearch Data Centre (FDZ)Institute for Employment Research (IAB)NurembergGermany

Christopher AntounCenter for Survey MeasurementU.S. Census BureauSuitland, MDUSA

Reg BakerMarketing Research Institute InternationalAnn Arbor, MIUSA

Stefan BenderResearch Data and Service CentreDeutsche BundesbankFrankfurt am MainGermany

Grant BensonSurvey Research CenterUniversity of MichiganAnn Arbor, MIUSA

Heather BergdahlProcess DepartmentStatistics SwedenStockholmSweden

Marcus E. BerzofskyDivision for Statistics and Data ScienceRTI InternationalResearch Triangle Park, NCUSA

Paul P. BiemerSocial, Statistical, and Environmental SciencesRTI InternationalResearch Triangle Park, NCOdum Institute for Research in Social ScienceUniversity of North CarolinaChapel Hill, NCUSA

Paul BurtonSurvey Research CenterUniversity of MichiganAnn Arbor, MIUSA

Christine BycroftStatistics New ZealandWellingtonNew Zealand

Jennifer Hunter ChildsResearch and Methodology DirectorateU.S. Census BureauWashington, DCUSA

Sue ConnorWestatRockville, MDUSA

Frederick G. ConradSurvey Research CenterUniversity of MichiganAnn Arbor, MIJoint Program in Survey MethodologyUniversity of MarylandCollege Park, MDUSA

Mick P. CouperSurvey Research CenterUniversity of MichiganAnn Arbor, MIJoint Program in Survey MethodologyUniversity of MarylandCollege Park, MDUSA

Edith de LeeuwDepartment of Methodology and StatisticsUtrecht UniversityUtrechtThe Netherlands

Stephanie EckmanSurvey Research DivisionRTI InternationalWashington, DCUSA

Brad EdwardsWestatRockville, MDUSA

Barbara FeldererCollaborative Research Center SBF 884“Political Economy of Reforms”University of MannheimMannheimGermany

Jamie GriffinSurvey Research CenterUniversity of MichiganAnn Arbor, MIUSA

Heidi GuyerSurvey Research CenterUniversity of MichiganAnn Arbor, MIUSA

Kristen Cibelli HibbenSurvey Research CenterUniversity of MichiganAnn Arbor, MIUSA

Daniela HochfellnerCenter for Urban Science and ProgressNew York UniversityNew York, NYUSA

Anders HolmbergStatistics NorwayOsloNorway

Joop HoxDepartment of Methodology and StatisticsUtrecht UniversityUtrechtThe Netherlands

Yuli Patrick HsiehSurvey Research DivisionRTI InternationalChicago, ILUSA

Frost HubbardSurvey Solutions DivisionIMPAQ InternationalColumbia, MDUSA

Andrew L. HuppSurvey Research CenterUniversity of MichiganAnn Arbor, MIUSA

Joost KappelhofDepartment of Education, Minorities, and MethodologyInstitute for Social Research/SCPThe HagueThe Netherlands

Alan F. KarrCenter of Excellence for Complex Data AnalysisRTI InternationalResearch Triangle Park, NCUSA

Jennifer KelleyInstitute for Social and Economic ResearchUniversity of EssexColchesterUK

Meena KhareNational Center for Health StatisticsCenters for Disease Control and PreventionHyattsville, MDUSA

Yumi KimDepartment of Research MethodsMarket Strategies InternationalLivonia, MIUSA

Antje KirchnerDepartment of SociologyUniversity of Nebraska‐LincolnLincoln, NESurvey Research DivisionRTI InternationalResearch Triangle Park, NCUSA

Thomas KlauschDepartment for Epidemiology and BiostatisticsVU University Medical CenterAmsterdamThe Netherlands

Frauke KreuterJoint Program in Survey MethodologyUniversity of MarylandCollege Park, MDUSADepartment of SociologyUniversity of MannheimMannheimStatistical Methods GroupInstitute for Employment Research (IAB)NurembergGermany

John LaFranceMarket Strategies InternationalLivonia, MIUSA

Thomas LaitilaDepartment of Research and DevelopmentStatistics SwedenDepartment of StatisticsÖrebro University School of BusinessÖrebroSweden

JiaoJiao LiMarket Strategies InternationalLivonia, MIUSA

Karin LindgrenProcess DepartmentStatistics SwedenStockholmSweden

Peter J. LugtigInstitute for Social and Economic ResearchUniversity of EssexColchesterUKDepartment of Methodology and StatisticsUtrecht UniversityUtrechtThe Netherlands

Lars E. LybergInizioStockholmSweden

Peter LynnInstitute for Social and Economic ResearchUniversity of EssexColchesterUK

Aaron MaitlandWestatRockville, MDUSA

Aigul MavletovaDepartment of SociologyNational Research University Higher School of EconomicsMoscowRussia

Peter Ph. MohlerUniversity of MannheimMannheimGermany

William D. MosherBloomberg School of Public HealthJohns Hopkins UniversityBaltimore, MDUSA

Mary H. MulryResearch and Methodology DirectorateU.S. Census BureauWashington, DCUSA

Joe MurphySurvey Research DivisionRTI InternationalChicago, ILUSA

Elizabeth M. NicholsResearch and Methodology DirectorateU.S. Census BureauWashington, DCUSA

Anders NorbergProcess DepartmentStatistics SwedenStockholmSweden

Daniel L. OberskiDepartment of Methodology and StatisticsUtrecht UniversityUtrechtThe Netherlands

Beth‐Ellen PennellSurvey Research CenterUniversity of MichiganAnn Arbor, MIUSA

Gregg PetersonSurvey Research CenterUniversity of MichiganAnn Arbor, MIUSA

Vicki J. PineauNORC at the University of ChicagoChicago, ILUSA

Joseph W. SakshaugCathie Marsh Institute for Social ResearchUniversity of ManchesterManchesterUKDepartment of Statistical MethodsInstitute for Employment Research (IAB)NurembergGermany

Michael F. SchoberDepartment of PsychologyNew School for Social ResearchNew York, NYUSA

James A. SingletonNational Center for Immunization and Respiratory DiseasesCenters for Disease Control and PreventionAtlanta, GAUSA

Benjamin SkallandNORC at the University of ChicagoChicago, ILUSA

Philip J. SmithNational Center for Immunization and Respiratory DiseasesCenters for Disease Control and PreventionAtlanta, GAUSA

Diana Maria StukelFHI 360Washington, DCUSA

Can TongurProcess DepartmentStatistics SwedenStockholmSweden

Roger TourangeauWestatRockville, MDUSA

Dennis TrewinFormer Australian StatisticianAustralian Bureau of StatisticsCanberraAustralia

James WagnerSurvey Research CenterUniversity of MichiganAnn Arbor, MIJoint Program in Survey MethodologyUniversity of MarylandCollege Park, MDUSA

Brady T. WestSurvey Research CenterUniversity of MichiganAnn Arbor, MIJoint Program in Survey MethodologyUniversity of MarylandCollege Park, MDUSA

Kirk M. WolterNORC at the University of ChicagoChicago, ILUSA

Gelaye WorkuDepartment of StatisticsStockholm UniversityStockholmSweden

Yingfu XieProcess DepartmentStatistics SwedenStockholmSweden

H. Yanna YanSurvey Research CenterUniversity of MichiganAnn Arbor, MIUSA

Ting YanMethodology UnitWestatRockville, MDUSA

David YankeyNational Center for Immunization and Respiratory DiseasesCenters for Disease Control and PreventionAtlanta, GAUSA

Wei ZengNORC at the University of ChicagoChicago, ILUSA

Zhen ZhaoNational Center for Immunization and Respiratory DiseasesCenters for Disease Control and PreventionAtlanta, GAUSA

Preface

Total survey error (TSE) refers to the accumulation of all errors that may arise in the design, collection, processing, and analysis of survey data. In this context, a survey error can be defined as any error contributing to the deviation of an estimate from its true parameter value. Survey errors arise from misspecification of concepts, sample frame deficiencies, sampling, questionnaire design, mode of administration, interviewers, respondents, data capture, missing data, coding, and editing. Each of these error sources can diminish the accuracy of inferences derived from the survey data. A survey estimate will be more accurate when bias and variance are minimized, which occurs only if the influence of TSE on the estimate is also minimized. In addition, if major error sources are not taken into account, various measures of margins of error are understated, which is a major problem for the survey industry and the users of survey data.

Because survey data underlie many public policy and business decisions, a thorough understanding of the effects of TSE on data quality is needed. The TSE framework, the focus of this book, is a valuable tool for understanding and improving survey data quality. The TSE approach summarizes the ways in which a survey estimate may deviate from the corresponding parameter value. Sampling error, measurement error, and nonresponse error are the most recognized sources of survey error, but the TSE framework also encourages researchers not to lose sight of the less commonly studied error sources, such as coverage error, processing error, and specification error. It also highlights the relationships between errors and the ways in which efforts to reduce one type of error can increase another, resulting in an estimate with more total error. For example, efforts to reduce nonresponse error may unintentionally lead to measurement errors, or efforts to increase frame coverage may lead to greater nonresponse.

This book is written to provide a review of the current state of the field in TSE research. It was stimulated by the first international conference on TSE that was held in Baltimore, Maryland, in September 2015 (http://www.TSE15.org). Dubbed TSE15, the conference had as its theme, “Improving Data Quality in the Era of Big Data.” About 140 papers were presented at the conference which was attended by approximately 300 persons. The conference itself was the culmination of a series of annual workshops on TSE called the International TSE Workshops (ITSEWs) which began in 2005 and still continue to this day. This book is an edited volume of 25 invited papers presented at the 2015 conference spanning a wide range of topics in TSE research and applications.

TSE15 was sponsored by a consortium of professional organizations interested in statistical surveys—the American Association of Public Opinion Research (AAPOR), three sections of the American Statistical Association (Survey Research Methods, Social Statistics, and Government Statistics), the European Survey Research Association (ESRA), and the World Association of Public Opinion Research (WAPOR). In addition, a number of organizations offered financial support for the conference and this book. There were four levels of contributions. Gallup, Inc. and AC Nielsen contributed at the highest level. At the next highest level, the contributors were NORC, RTI International, Westat, and the University of Michigan (Survey Research Center). At the third level were Mathematica Policy Research, the National Institute of Statistical Sciences (NISS), and Iowa State University. Finally, the Council of Professional Associations on Federal Statistics (COPAFS) and ESOMAR World Research offered in‐kind support. We are deeply appreciative of the sponsorship and support of these organizations which made the conference and this book possible.

Stephanie Eckman (RTI International) and Brad Edwards (Westat) cochaired the conference and the organizing committee, which included Paul P. Biemer (RTI International), Edith de Leeuw (Utrecht University), Frauke Kreuter (University of Maryland), Lars E. Lyberg (Inizio), N. Clyde Tucker (American Institutes for Research), and Brady T. West (University of Michigan). The organizing committee also did double duty as coeditors of this volume. Paul P. Biemer led the editorial committee.

This book is divided into five sections, each edited, primarily, by three members of the editorial team. These teams worked with the authors over the course of about a year and were primarily responsible for the quality and clarity of the chapters. The sections and their editorial teams were the following.

Section 1: The Concept of TSE and the TSE Paradigm (Editors: Biemer, Edwards, and Lyberg). This section, which includes Chapters 1 through 4, provides conceptual frameworks useful for understanding the TSE approach to design, implementation, evaluation, and analysis and how the framework can be extended to encompass new types of data and their inherent quality challenges.

Section 2: Implications for Survey Design (Editors: De Leeuw, Kreuter, and Eckman). This section includes Chapters 5 through 11 and provides methods and practical applications of the TSE framework to multiple‐mode survey designs potentially involving modern data collection technologies and multinational and multicultural survey considerations.

Section 3: Data Collection and Data Processing Applications (Editors: Edwards, Eckman, and de Leeuw). This section includes Chapters 12 through 15 and focuses on issues associated with applying the TSE framework to control costs and errors during data collection activities.

Section 4: Evaluation and Improvement (Editors: West, Biemer, and Tucker). This section includes Chapters 16 through 21 and describes a range of statistical methods and other approaches for simultaneously evaluating multiple error sources in survey data and mitigating their effects.

Section 5: Estimation and Analysis (Editors: Kreuter, Tucker, and West). This section includes Chapters 22 through 25 which deal with issues such as the appropriate analysis of survey data subject to sampling and nonsampling errors, potential differential biases associated with data collected by mixed modes and errors in linking records, and reducing these errors in modeling, estimation, and statistical inferences.

The edited volume is written for survey professionals at all levels, from graduate students in survey methodology to experienced survey practitioners wanting to imbue cutting‐edge principles and practices of the TSE paradigm in their work. The book highlights use of the TSE framework to understand and address issues of data quality in official statistics and in social, opinion, and market research. The field of statistics is undergoing a revolution as data sets get bigger (and messier), and understanding the potential for data errors and the various means to control and prevent them is more important than ever. At the same time, survey organizations are challenged to collect data more efficiently without sacrificing quality.

Finally, we, the editors, would like to thank the authors of the chapters herein for their diligence and support of the goal of providing this current overview of a dynamic field of research. We hope that the significant contributions they have made in these chapters will be multiplied many times over by the contributions of readers and other methodologists as they leverage and expand on their ideas.

Paul P. BiemerEdith de LeeuwStephanie EckmanBrad EdwardsFrauke KreuterLars E. LybergN. Clyde TuckerBrady T. West

Section 1The Concept of TSE and the TSE Paradigm

1The Roots and Evolution of the Total Survey Error Concept

Lars E. Lyberg1 and Diana Maria Stukel2

1 Inizio, Stockholm, Sweden

2 FHI 360, Washington, DC, USA

1.1 Introduction and Historical Backdrop

Sir Ronald Fisher

Jerzy Neyman

In this chapter, we discuss the concept of total survey error (TSE), how it originated and developed both as a mindset for survey researchers and as a criterion for designing surveys. The interest in TSE has fluctuated over the years. When Jerzy Neyman published the basic sampling theory and some of its associated sampling schemes in 1934 onward, it constituted the first building block of a theory and methodology for surveys. However, the idea that a sample could be used to represent an entire population was not new. The oldest known reference to estimating a finite population total on the basis of a sample dates back to 1000 BC and is found in the Indian epic Mahabharata (Hacking, 1975; Rao, 2005). Crude attempts at measuring parts of a population rather than the whole had been used in England and some other European countries quite extensively between 1650 and 1800. The methods on which these attempts were based were referred to as political arithmetic (Fienberg and Tanur, 2001), and they resembled ratio estimation using information of birth rates, family size, and average number of persons living in selected buildings and other observations. In 1895, at an International Statistical Institute meeting, Kiaer argued for developing a representative or partial investigation method (Kiaer, 1897). The representative method aimed at creating a sample that would reflect the composition of the population of interest. This could be achieved by using balanced sampling through purposive selection or various forms of random sampling. During the period 1900–1920, the representative method was used extensively, at least in Russia and the U.S.A. In 1925, the International Statistical Institute released a report on various aspects of random sampling (Rao, 2005, 2013; Rao and Fuller, 2015). The main consideration regarding sampling was likely monetary, given that it was resource‐intensive and time‐consuming to collect data from an entire population. Statistical information compiled using a representative sample was an enormous breakthrough. But it would be almost 40 years after Kiaer’s proposal before Neyman published his landmark paper from 1934 “On the Two Different Aspects of the Representative Method: The Method of Stratified Sampling and the Method of Purposive Selection.” At this time, there existed some earlier work by the Russian statistician Tschuprow (1923a, b) on stratified sampling and optimal allocation. It is not clear whether Neyman was aware of this work when he started to develop the sampling theory in the 1920s (Fienberg and Tanur, 1996) since he did not mention Tschuprow’s work when discussing optimal allocation. Neyman definitely had access to Ronald Fisher’s (1925) ideas on randomization (as opposed to various kinds of purposive selection) and their importance for the design and analysis of experiments, and also to Bowley’s (1926) work on stratified random sampling.

Prasanta Mahalanobis

Morris Hansen

Edwards Deming

The sampling methods proposed by Neyman were soon implemented in agencies such as the Indian Statistical Institute and the U.S. Bureau of the Census (currently named the U.S. Census Bureau). Prasanta Mahalanobis, the founder of the Indian Statistical Institute, and Morris Hansen and colleagues at the U.S. Census Bureau, became the main proponents of scientific sampling in a number of surveys in the 1940s. The development was spurred on by Literary Digest’s disastrously inaccurate prediction in the 1936 U.S. presidential election poll that was based on a seriously deficient sampling frame. However, Neyman’s sampling theory did not take into account nonsampling errors and relied on the assumption that sampling was the only major error source that affected estimates of population parameters and associated calculations of confidence intervals or margins of error. However, Neyman and his peers understood that this was indeed an unrealistic assumption that might lead to understated margins of error. The effect of nonsampling errors on censuses was acknowledged and discussed in a German textbook on census methodology relatively early on (Zizek, 1921). The author discussed what he called control of contents and coverage