Online Panel Research E-Book

• Herausgeber: John Wiley & Sons
• Kategorie: Wissenschaft und neue Technologien
• Serie: Wiley Series in Survey Methodology
• Sprache: Englisch

Seitenzahl: 1115

Veröffentlichungsjahr: 2014

Beschreibung

Provides new insights into the accuracy and value of online panels for completing surveys

Over the last decade, there has been a major global shift in survey and market research towards data collection, using samples selected from online panels. Yet despite their widespread use, remarkably little is known about the quality of the resulting data.

This edited volume is one of the first attempts to carefully examine the quality of the survey data being generated by online samples. It describes some of the best empirically-based research on what has become a very important yet controversial method of collecting data. Online Panel Research presents 19 chapters of previously unpublished work addressing a wide range of topics, including coverage bias, nonresponse, measurement error, adjustment techniques, the relationship between nonresponse and measurement error, impact of smartphone adoption on data collection, Internet rating panels, and operational issues.

The datasets used to prepare the analyses reported in the chapters are available on the accompanying website: www.wiley.com/go/online_panel

• Covers controversial topics such as professional respondents, speeders, and respondent validation.
• Addresses cutting-edge topics such as the challenge of smartphone survey completion, software to manage online panels, and Internet and mobile ratings panels.
• Discusses and provides examples of comparison studies between online panels and other surveys or benchmarks.
• Describes adjustment techniques to improve sample representativeness.
• Addresses coverage, nonresponse, attrition, and the relationship between nonresponse and measurement error with examples using data from the United States and Europe.
• Addresses practical questions such as motivations for joining an online panel and best practices for managing communications with panelists.
• Presents a meta-analysis of determinants of response quantity.
• Features contributions from 50 international authors with a wide variety of backgrounds and expertise.

This book will be an invaluable resource for opinion and market researchers, academic researchers relying on web-based data collection, governmental researchers, statisticians, psychologists, sociologists, and other research practitioners.

Leseprobe

Table of Contents

Cover

Wiley Series in Survey Methodology

Title Page

Copyright

Preface

Content of the book

References

Acknowledgments

Companion datasets and book format

References

About the Editors

About the Contributors

Chapter 1: Online panel research: History, concepts, applications and a look at the future

1.1 Introduction

1.2 Internet penetration and online panels

1.3 Definitions and terminology

1.4 A brief history of online panels

1.5 Development and maintenance of online panels

1.6 Types of studies for which online panels are used

1.7 Industry standards, professional associations' guidelines, and advisory groups

1.8 Data quality issues

1.9 Looking ahead to the future of online panels

References

Chapter 2: A critical review of studies investigating the quality of data obtained with online panels based on probability and nonprobability samples

2.1 Introduction

2.2 Taxonomy of comparison studies

2.3 Accuracy metrics

2.4 Large-scale experiments on point estimates

2.5 Weighting adjustments

2.6 Predictive relationship studies

2.7 Experiment replicability studies

2.8 The special case of pre-election polls

2.9 Completion rates and accuracy

2.10 Multiple panel membership

2.11 Online panel studies when the offline population is less of a concern

2.12 Life of an online panel member

2.13 Summary and conclusion

References

Part I: Coverage

Introduction to Part I

I.1 Coverage bias in online panels

I.2 The chapters in Part I

References

Chapter 3: Assessing representativeness of a probability-based online panel in Germany

3.1 Probability-based online panels

3.2 Description of the GESIS Online Panel Pilot

3.3 Assessing recruitment of the Online Panel Pilot

3.4 Assessing data quality: Comparison with external data

3.5 Results

3.6 Discussion and conclusion

References

Appendix 3.A

Chapter 4: Online panels and validity: Representativeness and attrition in the Finnish eOpinion panel

4.1 Introduction

4.2 Online panels: Overview of methodological considerations

4.3 Design and research questions

4.4 Data and methods

4.5 Findings

4.6 Conclusion

References

Chapter 5: The untold story of multi-mode (online and mail) consumer panels: From optimal recruitment to retention and attrition

5.1 Introduction

5.2 Literature review

5.3 Methods

5.4 Results

5.5 Discussion and conclusion

References

Part II: Nonresponse

Introduction to Part II

II.1 The nonresponse problem

II.2 The nonresponse bias

II.3 Exploring nonresponse

II.4 The chapters in Part II

References

Chapter 6: Nonresponse and attrition in a probability-based online panel for the general population

6.1 Introduction

6.2 Attrition in online panels versus offline panels

6.3 The LISS panel

6.4 Attrition modeling and results

6.5 Comparison of attrition and nonresponse bias

6.6 Discussion and conclusion

References

Chapter 7: Determinants of the starting rate and the completion rate in online panel studies

7.1 Introduction

7.2 Dependent variables

7.3 Independent variables

7.4 Hypotheses

7.5 Method

7.6 Results

7.7 Discussion and conclusion

References

Chapter 8: Motives for joining nonprobability online panels and their association with survey participation behavior

8.1 Introduction

8.2 Motives for survey participation and panel enrollment

8.3 Present study

8.4 Results

8.5 Conclusion

References

Appendix 8.A

Chapter 9: Informing panel members about study results: Effects of traditional and innovative forms of feedback on participation

9.1 Introduction

9.2 Background

9.3 Method

9.4 Results

9.5 Discussion and conclusion

References

Appendix 9.A

Part III: Measurement Error

Introduction to Part III

Measurement Error

References

Chapter 10: Professional respondents in nonprobability online panels

10.1 Introduction

10.2 Background

10.3 Professional respondents and data quality

10.4 Approaches to handling professional respondents

10.5 Research hypotheses

10.6 Data and methods

10.7 Results

10.8 Satisficing behavior

10.9 Discussion

References

Appendix 10.A

Chapter 11: The impact of speeding on data quality in nonprobability and freshly recruited probability-based online panels

11.1 Introduction

11.2 Theoretical framework

11.3 Data and methodology

11.4 Response time as indicator of data quality

11.5 How to measure “speeding”?

11.6 Does speeding matter?

11.7 Conclusion

References

Part IV: Weighting Adjustments

Introduction to Part IV

IV.1 Panel problems

IV.2 Weighting adjustments

IV.3 Effective weighting

IV.4 Weighting adjustment techniques

IV.5 Generalized regression estimation

IV.6 Raking ratio estimation

IV.7 Weighting adjustment with a reference survey

IV.8 Propensity weighting

IV.9 The chapters in Part IV

References

Chapter 12: Improving web survey quality: Potentials and constraints of propensity score adjustments

Introduction

12.2 Survey quality and sources of error in nonprobability web surveys

12.3 Data, bias description, and PSA

12.4 Results

12.5 Potentials and constraints of PSA to improve nonprobability web survey quality: Conclusion

References

Appendix 12.A

Chapter 13: Estimating the effects of nonresponses in online panels through imputation

13.1 Introduction

13.2 Method

13.3 Measurements

13.4 Findings

13.5 Discussion and conclusion

Acknowledgement

References

Part V: Nonresponse and Measurement Error

Introduction to Part V

Nonresponse and measurement error

Chapter 14: The relationship between nonresponse strategies and measurement error: Comparing online panel surveys to traditional surveys

14.1 Introduction

14.2 Previous research and theoretical overview

14.3 Does interview mode moderate the relationship between nonresponse strategies and data quality?

14.4 Data

14.5 Measures

14.6 Results

14.7 Discussion and conclusion

References

Chapter 15: Nonresponse and measurement error in an online panel: Does additional effort to recruit reluctant respondents result in poorer quality data?

15.1 Introduction

15.2 Understanding the relation between nonresponse and measurement error

15.3 Response propensity and measurement error in panel surveys

15.4 The present study

15.5 Data

15.6 Analytical strategy

15.7 Results

15.8 Discussion and conclusion

References

Part VI: Special Domains

Introduction to Part VI

Special domains

References

Chapter 16: An empirical test of the impact of smartphones on panel-based online data collection

16.1 Introduction

16.2 Method

16.3 Results

16.4 Discussion and conclusion

References

Chapter 17: Internet and mobile ratings panels

17.1 Introduction

17.2 History and development of Internet ratings panels

17.3 Recruitment and panel cooperation

17.4 Compliance and panel attrition

17.5 Measurement issues

17.6 Long tail and panel size

17.7 Accuracy and validation studies

17.8 Statistical adjustment and modeling

17.9 Representative research

17.10 The future of Internet audience measurement

References

Part VII: Operational Issues in Online Panels

Introduction to Part VII

VII.1 Management of sampling and data quality processes in online panels

VII.2 The chapters in Part VII

Chapter 18: Online panel software

18.1 Introduction

18.2 What does online panel software do?

18.3 Survey of software providers

18.4 A typology of panel research software

18.5 Support for the different panel software typologies

18.7 Panel recruitment and profile data

18.8 Panel administration

18.9 Member portal

18.10 Sample administration

18.11 Data capture, data linkage and interoperability

18.12 Diagnostics and active panel management

18.13 Conclusion and further work

References

Chapter 19: Validating respondents' identity in online samples: The impact of efforts to eliminate fraudulent respondents

19.1 Introduction

19.2 The 2011 study

19.3 The 2012 study

19.4 Results

19.5 Discussion

19.6 Conclusion

References

Appendix 19.A

WILEY SERIES IN SURVEY METHODOLOGY

Index

End User License Agreement

Pages

xv

xvi

xvii

xix

xx

xxi

xxii

xxiii

xxiv

xxv

xxvi

xxvii

xxviii

xxix

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

219

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

55

56

57

58

59

60

127

128

129

130

131

132

133

215

216

217

263

264

265

266

267

268

269

270

271

272

311

312

363

364

365

409

410

411

412

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

Guide

Cover

Table of Contents

Preface

Part I: Coverage

Chapter 1: Online panel research: History, concepts, applications and a look at the future

List of Illustrations

Figure 5.1

Figure 5.2

Figure 5.3

Figure 6.1

Figure 10.1

Figure 11.1

Figure 11.2

Figure 11.3

Figure 12.1

Figure 12.2

Figure 13.1

Figure 15.1

Figure 15.2

Figure 17.1

List of Tables

Table 2.1

Table 2.2

Table 2.3

Table 2.4

Table 2.5

Table 2.6

Table 3.1

Table 3.2

Table 3.A.1

Table 3.3

Table 3.4

Table 3.5

Table 3.6

Table 3.7

Table 3.A.2

Table 4.1

Table 4.2

Table 4.3

Table 4.4

Table 4.5

Table 4.6

Table 5.1

Table 5.2

Table 5.3

Table 5.4

Table 5.5

Table 5.6

Table 5.7

Table 6.1

Table 6.2

Table 6.3

Table 6.4

Table 6.5

Table 7.1

Table 7.2

Table 7.3

Table 7.4

Table 8.1

Table 8.2

Table 8.3

Table 8.4

Table 8.5

Table 9.1

Table 9.2

Table 9.3

Table 9.4

Table 9.5

Table 9.6

Table 9.A.1

Table 10.1

Table 10.2

Table 10.3

Table 10.4

Table 11.1

Table 11.2

Table 11.3

Table 11.4

Table 11.5

Table 11.6

Table 11.7

Table 11.8

Table 11.9

Table 11.10

Table 11

Table 12

Table 12.A.1

Table 12.A.2

Table 12.1

Table 12.A.3

Table 12.A.5

Table 12.2

Table 12.3

Table 12.A.6

Table 12.A.7

Table 13.3

Table 13.1

Table 13.2

Table 14.1

Table 14.2

Table 14.3

Table 14.4

Table 14.5

Table 15.1

Table 15.2

Table 15.3

Table 15.4

Table 15.5

Table 16.1

Table 16.2

Table 16.3

Table 16.4

Table 16.5

Table 16.6

Table 16.7

Table 16.8

Table 16.9

Table 16.10

Table 16.11

Table 16.12

Table 16.13

Table 16.14

Table 16.15

Table 18.1

Table 18.2

Table 18.3

Table 18.4

Table 18.5

Table 18.6

Table 18.7

Table 18.8

Table 18.9

Table 18.10

Table 18.11

Table 18.12

Table 18.13

Table 18.14

Table 18.15

Table 18.16

Table 18.17

Table 18.18

Table 18.19

Table 19.1

Table 19.2

Table 19.3

WILEY SERIES IN SURVEY METHODOLOGY

Established in Part by WALTER A. SHEWHART and SAMUEL S. WILKS

Editors: Mick P. Couper, Graham Kalton, J. N. K. Rao, Norbert Schwarz, Christopher Skinnerx

Editor Emeritus: Robert M. Groves

A complete list of the titles in this series appears at the end of this volume.

Online Panel Research

A Data Quality Perspective

Mario Callegaro

Survey Research Scientist, Quantitative Marketing Team, Google UK

Reg Baker

Senior Consultant, Market Strategies International, USA

Jelke Bethlehem

Senior Advisor, Statistics Netherlands, The Netherlands

Anja S. Göritz

Professor of Occupational and Consumer Psychology

University of Freiburg, Germany

Jon A. Krosnick

Professor of Political Science, Communication and Psychology

Stanford University, USA

Paul J. Lavrakas

Independent Research Psychologist/Research Methodologist, USA

This edition first published 2014

© 2014 John Wiley & Sons, Ltd

John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, United Kingdom

For details of our global editorial offices, for customer services and for information about how to apply for permission to reuse the copyright material in this book please see our website at www.wiley.com.

The right of the author to be identified as the author of this work has been asserted in accordance with the Copyright, Designs and Patents Act 1988.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by the UK Copyright, Designs and Patents Act 1988, without the prior permission of the publisher.

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books.

Designations used by companies to distinguish their products are often claimed as trademarks. All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners. The publisher is not associated with any product or vendor mentioned in this book.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. It is sold on the understanding that the publisher is not engaged in rendering professional services and neither the publisher nor the author shall be liable for damages arising herefrom. If professional advice or other expert assistance is required, the services of a competent professional should be sought.

Library of Congress Cataloging-in-Publication Data

Online panel research : a data quality perspective / edited by Mario Callegaro, Reg Baker, Jelke Bethlehem, Anja Goritz, Jon A. Krosnick, Paul J. Lavrakas.

pages cm

Includes bibliographical references and index.

ISBN 978-1-119-94177-4 (pbk.)

1. Panel analysis. 2. Internet research. 3. Marketing research–Methodology 4. Social sciences–Methodology

I. Callegaro, Mario, editor of compilation.

H61.26.O55 2014

001.4′33–dc23

2013048411

A catalogue record for this book is available from the British Library.

ISBN: 978-1-119-94177-4

Preface

Web survey data collected via online panels are an important source of information that thousands of researchers use to gain insight and which influence their decisions about the topics under study. Thousands upon thousands of completed surveys can be collected in a few days and for little money, as low as $2 per completed questionnaire (Terhanian & Bremer, 2012). Therefore, with the data user's point of view in mind, it is prudent to ask the questions: how reliable are these data and what risks do we assume when we use them?

Content of the book

This edited volume is one of the first attempts to carefully examine the quality of data from online panels. The book is organized into seven parts and contains 19 chapters. The first two chapters are critical reviews of what is known so far about the development and current status of online panel research and the quality of data obtained using these panels in terms of point estimates, relationships across variables, and reproducibility of results.

Part I discusses coverage error, in which three international case studies from probability-based panels in Germany, Finland, and the United States are reviewed.

Nonresponse is a much-debated topic that is addressed in Part II, in four chapters, examining the issue related to nonresponse in online panels from different points of view. Chapter 6 looks at nonresponse and attrition in the Dutch LISS panel, followed by Chapter 7, a review of determinants of response quantity, analyzing a large dataset of online panel surveys. In Chapter 8, data from Austria are presented to advance understanding of the reasons respondents join an online panel. The final chapter explores the effectiveness of experiments in engaging panel members by providing feedback on their participation.

Part III examines measurement error from two angles: the issue of professional respondents (Chapter 10) and the issue of speeding when completing web surveys (Chapter 11).

In Part IV, the next issue explored is adjustment of data obtained from online panels. Chapter 12 describes an example of how propensity score weighting was used to calibrate a non-probability panel and its effect when comparing the results to official statistics benchmarks. Imputation, which is another way to compensate for nonresponse, is the topic of Chapter 13. Taking advantage of a unique study in which online panel members were invited to join a specific project and thus some variables for the nonrespondents were already known, the chapter evaluates the effectiveness of imputation.

Survey researchers have demonstrated on many occasions (e.g., Groves & Peytcheva, 2008) the importance of examining the relation between nonresponse and measurement error. This relation is evaluated in Part V, in contributions by two teams, one from the United States (Chapter 14) and one from the United Kingdom (Chapter 15), both using multiple datasets. These complementary chapters demonstrate, using advanced analytics, the numerous ways of exploring the relation between nonresponse and measurement error.

Part IV on special domains looks at two recent topics: (1) the impact of smartphones on panel data collection in Chapter 16; and (2) online ratings panels in Chapter 17. In the latter type of panel, the Internet behavior of its members is tracked, providing a constant flow of traffic that can be analyzed.

The volume ends with Part VII, an examination of two important operational issues: (1) the software used to manage online panels in Chapter 18; and (2) how to validate respondents who join a panel in Chapter 19.

References

Groves, R. M., & Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse bias: A meta-analysis.

Public Opinion Quarterly

,

72

, 167–189.

Terhanian, G., & Bremer, J. (2012). A smarter way to select respondents for surveys?

International Journal of Market Research

,

54

, 751–780.

Acknowledgments

The book's initial idea was suggested by Lars Lyberg to Wiley. We are indebted to Lars for his forward thinking. We thank our Wiley editors Richard Davies, Kathryn Sharples, Heather Kay, and Jo Taylor.

Companion datasets and book format

All authors agreed to make the datasets used in the analysis publicly available. Wiley is providing a section of the book's webpage dedicated to showing readers how to obtain the anonymized datasets. The GESIS data archive kindly provided support for some datasets not archived elsewhere. We hope other similar volumes will do the same in the future. Transparency like this is a key to scientific credibility and progress. By allowing other scholars to analyze and enhance the findings of studies, these chapters will have even more impact and value over time. (Cooper, 2013; Fang, Steen, & Casadevall, 2012).

The book is released in different formats: print, e-book, and pdf version. Single chapters can be purchased or accessed online on the Wiley website where an abstract is also available. Online appendies and instructions on how to access the datasets used in the chapters are available at: www.wiley.com/go/online_panel

References

Cooper, J. (2013). On fraud, deceit and ethics.

Journal of Experimental Social Psychology

,

49

, 314.

Fang, F. C., Steen, R. G., & Casadevall, A. (2012). Misconduct accounts for the majority of retracted scientific publications.

Proceedings of the National Academy of Sciences of the United States of America

,

109

, 17028–17033.

Mario Callegaro Reg P. Baker Jelke Bethlehem Anja S. Göritz Jon A. Krosnick Paul J. Lavrakas

About the Editors

Reg Baker is the former President and Chief Operating Officer of Market Strategies International, a full-service research company specializing in healthcare, energy, financial services, telecommunications, and information technology. Prior to joining Market Strategies, he was Vice President for Research Operations at NORC where he oversaw the national field staff, the company's CATI centers, and its technology infrastructure.