Reproducibility E-Book
97,99 €
Mehr erfahren.
- Herausgeber: John Wiley & Sons
- Kategorie: Fachliteratur
- Sprache: Englisch
2017 PROSE Award Honorable Mention
The PROSE Awards draw attention to pioneering works of research and for contributions to the conception, production, and design of landmark works in their fields.
Featuring peer-reviewed contributions from noted experts in their fields of research, Reproducibility: Principles, Problems, Practices, and Prospects presents state-of-the-art approaches to reproducibility, the gold standard of sound science, from multi- and interdisciplinary perspectives. Including comprehensive coverage for implementing and reflecting the norm of reproducibility in various pertinent fields of research, the book focuses on how the reproducibility of results is applied, how it may be limited, and how such limitations can be understood or even controlled in the natural sciences, computational sciences, life sciences, social sciences, and studies of science and technology.
The book presents many chapters devoted to a variety of methods and techniques, as well as their epistemic and ontological underpinnings, which have been developed to safeguard reproducible research and curtail deficits and failures. The book also investigates the political, historical, and social practices that underlie reproducible research in contemporary science studies, including the difficulties of good scientific practice and the ethos of reproducibility in modern innovation societies.
Reproducibility: Principles, Problems, Practices, and Prospects is a guide for researchers who are interested in the general and overarching questions behind the concept of reproducibility; for active scientists who are confronted with practical reproducibility problems in their everyday work; and for economic stakeholders and political decision makers who need to better understand the challenges of reproducibility. In addition, the book is a useful in-depth primer for undergraduate and graduate-level courses in scientific methodology and basic issues in the philosophy and sociology of science from a modern perspective.
“A comprehensive, insightful treatment of the reproducibility challenges facing science today and of ways in which the scientific community can address them.” Kathleen Hall Jamieson, Elizabeth Ware Packard Professor of Communication, University of Pennsylvania
“How can we make sure that reproducible research remains a key imperative of scientific communication under increasing commercialization, media attention, and publication pressure? This handbook offers the first interdisciplinary and fundamental treatment of this important question.”Torsten Hothorn, Professor of Biostatistics, University of Zurich
Harald Atmanspacher, PhD, is Associate Fellow and staff member at Collegium Helveticum, ETH and University Zurich and is also President of the Society for Mind-Matter Research. He has pioneered advances in complex dynamical systems research and in a number of topics concerned with the relation between the mental and physical.
Sabine Maasen, PhD, is Professor for Sociology of Science and Director of the Munich Center for Technology in Society (TU Munich) and Associate Fellow at Collegium Helveticum (ETH and University Zurich). Her research focuses on the interface of science, technology, and society, notably with respect to neuroscience and its applications.
Sie lesen das E-Book in den Legimi-Apps auf:
Seitenzahl: 1108
Veröffentlichungsjahr: 2016
Ähnliche
Table of Contents
Title Page
Copyright
Contributors
Chapter 1: Introduction
References
Part I: Contextual Backgrounds
Chapter 1: Reproducibility, Objectivity, Invariance
1.1 Introduction
1.2 Reproducibility in the Empirical Sciences
1.3 Objectivity
1.4 Invariance and Symmetry
1.5 Summary
References
Chapter 2: Reproducibility between Production and Prognosis
2.1 Preliminary Remarks: Three Myths
2.2 How Does Reproducibility Connect with Production?
2.3 How Does Production Connect with Continuity?
2.4 How Does Continuity Connect with Scientific Rationality?
2.5 How Does Scientific Rationality Connect with Prognosis?
2.6 How Do Prediction and Prognosis Connect with Reproducibility?
2.7 Concluding Remarks
References
Chapter 3: Stability and Replication of Experimental Results: A Historical Perspective
3.1 Experiments and Their Reproduction in the Development of Science
3.2 Repetition of Experiments
3.3 The Power of Replicability
3.4 Cases of Failed Replication
3.5 Doing Science without Replication and Replicability
3.6 What Can We Learn from History?
Acknowledgments
References
Chapter 4: Reproducibility of Experiments: Experimenters' Regress, Statistical Uncertainty Principle, and the Replication Imperative
4.1 Introduction
4.2 The Experimenter's Regress
4.3 The Statistical Uncertainty Principle
4.4 The Replication Imperative
References
Part II: Statistical Issues
Chapter 5: Statistical Issues in Reproducibility
5.1 Introduction
5.2 A Random Sample
5.3 Structures of Variation
5.4 Regression Models
5.5 Model Development and Selection Bias
5.6 Big and High-Dimensional Data
5.7 Bayesian Statistics
5.8 Conclusions
Acknowledgments
References
Chapter 6: Model Selection, Data Distributions, and Reproducibility
6.1 Introduction
6.2 Bayesian Model Selection and Relation to Minimum Description Length
6.3 Extending BMS (and NML#): BMS*
6.4 Replication Variance and Reproducibility
6.5 Final Remark
References
Chapter 7: Reproducibility from the Perspective of Meta-Analysis
7.1 Introduction
7.2 Basics of Meta-Analysis
7.3 Meta-Analysis of Mind-Matter Experiments: A Case Study
7.4 Summary
References
Chapter 8: Why Are There So Many Clustering Algorithms, and How Valid Are Their Results?
8.1 Introduction
8.2 Supervised and Unsupervised Learning
8.3 Cluster Validity as Easiness in Classification
8.4 Applying Clustering-Quality Measures to Data
8.5 Other Clustering Models
8.6 Summary
References
Part III: Physical Sciences
Chapter 9: Facilitating Reproducibility in Scientific Computing: Principles and Practice
9.1 Introduction
9.2 A Culture of Reproducibility
9.3 Statistical Overfitting
9.4 Performance Reporting in High-Performance Computing
9.5 Numerical Reproducibility
9.6 High-Precision Arithmetic in Experimental Mathematics and Mathematical Physics
9.7 Reproducibility in Symbolic Computing
9.8 Why Should We Trust the Results of Computation?
9.9 Conclusions
References
Chapter 10: Methodological Issues in the Study of Complex Systems
10.1 Introduction
10.2 Definitions of Complexity
10.3 Complexity and Meaning
10.4 Beyond Stationarity and Ergodicity
10.5 Conclusions
Acknowledgments
References
Chapter 11: Rare and Extreme Events
11.1 Introduction
11.2 Statistics of Extremes
11.3 Predictions of Extreme Events
11.4 Evolving Systems Exposed to Extreme Events
11.5 Conclusions
Acknowledgments
References
Chapter 12: Science under Societal Scrutiny: Reproducibility in Climate Science
12.1 Reproducibility Challenges for Climate Science
12.2 Reproducibility in Observational Climate Science
12.3 Reproducibility in Climate Modeling
12.4 Reproducibility in Paleoclimatology
12.5 Conclusions and Recommendations
References
Part IV: Life Sciences
Chapter 13: From Mice to Men: Translation from Bench to Bedside
13.1 The Drug Development Process
13.2 Contributions of Animals to Medical Progress
13.3 Translation Challenges in Different Fields of Research
13.4 Increasing Translational Success: Summary and Conclusions
References
Chapter 14: A Continuum of Reproducible Research in Drug Development
14.1 Introduction
14.2 The Strategy of the Magic Bullet
14.3 Specialists and Generalists
14.4 From Single-Target to Multi-Target Drugs
14.5 Conclusions
References
Chapter 15: Randomness as a Building Block for Reproducibility in Local Cortical Networks
15.1 Introduction
15.2 Spike Trains and Reproducibility
15.3 Spike Trains
15.4 Neuronal Populations
15.5 Summary
References
Chapter 16: Neural Reuse and In-Principle Limitations on Reproducibility in Cognitive Neuroscience
16.1 Introduction
16.2 The Erosion of Modular Thinking
16.3 Intrinsic Limits on Reproducibility
16.4 Going Forward
References
Chapter 17: On the Difference between Persons and Things – Reproducibility in Social Contexts
17.1 The Problem of Other Minds and Its Evolutionary Dimension
17.2 Understanding the Inner Experience of Others
17.3 Identifying the Neural Mechanisms of Understanding Others
17.4 Abduction of the Functional Roles of Neural Networks
17.5 Psychopathology of the Inner Experience of Others
17.6 Conclusions
References
Part V: Social Sciences
Chapter 18: Order Effects in Sequential Judgments and Decisions
18.1 Introduction
18.2 Question Order Effects and QQ Equality
18.3 No Order Effect Model and Saturated Model
18.4 The Anchor Adjustment Model
18.5 The Repeat Choice Model
18.6 The Quantum Model
18.7 Concluding Comments
References
Chapter 19: Reproducibility in the Social Sciences
19.1 Introduction
19.2 Reproducibility as a Current Problem in the Social Sciences
19.3 “Reproductions Have No Meaningful Scientific Value”
19.4 Reaction from the Blogosphere
19.5 Conclusion
References
Chapter 20: Accurate But Not Reproducible? The Possible Worlds of Public Opinion Research
20.1 Introduction
20.2 Reproducibility: A Missing Criterion in Public Opinion Research?
20.3 Big Data versus Science: The Breakthrough of Modern Polling
20.4 The Birth of a Statistical Myth
20.5 Generating Trust
10
20.6 The Possible Worlds of Public Opinion Research
20.7 Looping Effects between Measurement and Measured
20.8 Swarms of Possible Worlds
References
Chapter 21: Depending on Numbers
21.1 Introduction
21.2 Statistical Error
21.3 Translation
21.4 Statistical and Substantive Significance
21.5 Irreproducible Numbers
21.6 Reproducing Calculations
References
Chapter 22: Science Between Trust and Control: Non-Reproducibility in Scholarly Publishing
22.1 Introduction
22.2 Reproducibility as the Touchstone for Distinguishing Science from Non-Science
22.3 Contested Claims: The Story behind STAP
22.4 The Structural Gap between the Production and Representation of Scientific Facts
22.5 The Increasing Awareness of Reproducibility Problems
22.6 The New Transparency: Bridging the Gap in Scholarly Publishing
22.7 Conclusions
References
Part VI: Wider Perspectives
Chapter 23: Repetition with a Difference: Reproducibility in Literature Studies
23.1 Introduction
23.3 Language and Difference
23.3 Mimesis, Imitatio, and Parody
23.4 Literary Translation: Domesticating versus Foreignizing
23.5 Reproducing Cultural Significance
23.6 Conclusions
Acknowledgments
References
Chapter 24: Repetition Impossible: Co-Affection by Mimesis and Self-Mimesis
24.1 Introduction
24.2 Repetition within the Philosophy of Time
24.3 Re-Presenting Forgetting
24.4 Repetition, Co-Affection and Trauma: Identity and Coping with the Past
24.5 The Dialectics of Remembering and Forgetting
24.6 Co-Affection and Memorizing Recall
24.7 Mimesis
References
Chapter 25: Relevance Criteria for Reproducibility: The Contextual Emergence of Granularity
25.1 Introduction
25.2 Contrast Classes, Coarse Grains, Partition Cells
25.3 Two Examples
25.4 Contextual Emergence
25.5 Ontological Relativity: Beyond Fundamentalism and Relativism
Acknowledgments
References
Chapter 26: The Quest for Reproducibility Viewed in the Context of Innovation Societies
26.1 Introduction
26.2 A Genealogical Sketch of “Innovation”
26.3 Reframing Scientific Ethos: Sound Science
26.4 Reproducibility and Innovation: A Regulative Dual
26.5 Making the Implicit Explicit I: Social Robustness of Science
26.6 Making the Implicit Explicit II: Responsible Research and Innovation
26.7 Mertonian Norms Challenged Anew: Institutional Reflexivity and Responsiveness
References
Index
End User License Agreement
Pages
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
Guide
Table of Contents
Introduction
Part I
Begin Reading
List of Tables
Chapter 5: Statistical Issues in Reproducibility
Table 5.1 The probabilities of getting the same result in the replication study, for the four possible cases of correct null hypothesis and alternative hypothesis and significance of test results in the original study. * Replication of wrong results is undesirable.
Table 5.2 Levels of validation for same and different features (middle columns) in the original and follow-up study. The different terms in the left column are related to different types of validation in the right column.
Chapter 6: Model Selection, Data Distributions, and Reproducibility
Table Matrix 6.1 Prior probabilities for data distributions (left margin), distributions predicted by model instances (right margin), and model outcomes (bottom margin). A posterior version of the same matrix can be produced where the margins are conditioned on the observed data outcome . As explained in the text, virtually all the equations and predictions in this chapter can be derived and explained with reference to this matrix. Data outcomes in bolded columns are explained in Section 6.4.2.
Table 6.1 follows the form of Matrix 6.1, with probabilities corresponding to the priors and data distributions for the toy example, for Case A described in the text. Case A assumes all distributions to be binomial, based on the corresponding value of (from 0.0 to 1.0 in steps of 0.1).
Chapter 9: Facilitating Reproducibility in Scientific Computing: Principles and Practice
Table 9.1 Run times on parallel and vector systems for different problem sizes (data for Fig. 9.5).
Chapter 13: From Mice to Men: Translation from Bench to Bedside
Table 13.1 Outcome distribution for drugs in the “main focus” category and in the “comparison” category, within category “drug type.”
Chapter 18: Order Effects in Sequential Judgments and Decisions
Table 18.1a: white-black
Table 18.1b: black-white
Table 18.1c: order effects
Table 18.2 joint probabilities for the no order effect model
Table 18.3a: anchor-adjust joint probabilities for A–B order
Table 18.3b: anchor-adjust joint probabilities for B–A order
Table 18.3c: anchor-adjust predicted order effects
Table 18.4a: repeat-choice joint probabilities for A–B order
Table 18.4b: repeat-choice joint probabilities for B–A order
Table 18.4c: repeat-choice predicted order effects
Table 18.5a: quantum model joint probabilities for A–B order
Table 18.5b: quantum model joint probabilities for B–A order
Table 18.5c: quantum model predicted order effects
Reproducibility
Principles, Problems, Practices, and Prospects
Edited by
Harald Atmanspacher
Collegium Helveticum, University and ETH Zurich, Zurich, Switzerland
Sabine Maasen
Munich Center for Technology in Society, Technical University, Munich, Germany
Copyright © 2016 by John Wiley & Sons, Inc. All rights reserved
Published by John Wiley & Sons, Inc., Hoboken, New Jersey
Published simultaneously in Canada
No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750-4470, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permissions.
Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.
For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.
Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com.
Library of Congress Cataloging-in-Publication Data:
Names: Atmanspacher, Harald. | Maasen, Sabine, 1960-
Title: Reproducibility : principles, problems, practices, and prospects /
edited by Harald Atmanspacher, Sabine Maasen.
Description: Hoboken, New Jersey : John Wiley & Sons, Inc., [2016] | Includes
index.
Identifiers: LCCN 2015036802 | ISBN 9781118864975 (cloth)
Subjects: LCSH: Observation (Scientific method) | Science–Methodology.
Classification: LCC Q175.32.O27 R47 2016 | DDC 001.4/2–dc23 LC record available at
http://lccn.loc.gov/2015036802
Contributors
Michael Anderson
Department of Psychology
Franklin and Marshal College
Lancaster PA, USA
Harald Atmanspacher
Collegium Helveticum
University and ETH Zurich
Zurich, Switzerland
Sabine Baier
Collegium Helveticum
University and ETH Zurich
Zurich, Switzerland
David H. Bailey
Lawrence Berkeley
National Laboratory
Berkeley CA, USA
Ladina Bezzola Lambert
Department of English
University of Basel
Basel, Switzerland
Jonathan Borwein
School of Mathematical
and Physical Sciences
University of Newcastle
Callaghan NSW, Australia
Jerome Busemeyer
Department of Psychological
and Brain Sciences
Indiana University
Bloomington IN, USA
Suyog Chandramouli
Department of Psychological
and Brain Sciences
Indiana University
Bloomington IN, USA
Harry Collins
School of Social Sciences
Cardiff University
Cardiff, UK
Werner Ehm
Heidelberg Institute
for Theoretical Studies
Heidelberg, Germany
Hinderk Emrich
Psychiatric Clinic
Hannover Medical School
Hannover, Germany
Vladimir Estivill-Castro
Department of Information
and Communication Technologies
University Pompeu Fabra
Barcelona, Spain
Georg Feulner
Earth System Analysis
Potsdam Institute for
Climate Impact Research
Potsdam, Germany
Gerd Folkers
Collegium Helveticum
University and ETH Zurich
Zurich, Switzerland
Martina Franzen
Wissenschaftszentrum
für Sozialforschung
Reichpietschufer 50
Berlin, Germany
Holger Kantz
Nonlinear Dynamics
and Time Series Analysis
Max-Planck-Institute for
Physics of Compex Systems
Dresden, Germany
Marianne Martic-Kehl
Collegium Helveticum
University and ETH Zurich
Zurich, Switzerland
Felix Keller
Humanities and Social Sciences
University of St. Gallen
St. Gallen, Switzerland
Johannes Lengler
Theoretical Computer Science
ETH Zurich
Zurich, Switzerland
Sabine Maasen
Center for Technology in Society
Technical University
Munich, Germany
Theodore Porter
Department of History
University of California
Los Angeles CA, USA
Martin Reinhart
Institute for Social Sciences
Humboldt University
Berlin, Germany
P. August Schubiger
Collegium Helveticum
University and ETH Zurich
Zurich, Switzerland
Richard Shiffrin
Department of Psychological
and Brain Sciences
Indiana University
Bloomington IN, USA
Werner Stahel
Seminar for Statistics
ETH Zurich
Zurich, Switzerland
Angelika Steger
Theoretical Computer Science
ETH Zurich
Zurich, Switzerland
Friedrich Steinle
Institute for Philosophy
Technical University
Berlin, Germany
Victoria Stodden
Graduate School of Library
and Information Sciences
University of Illinois
Urbana-Champaign IL, USA
Holm Tetens
Institute for Philosophy
Free University
Berlin, Germany
Kai Vogeley
Department of Psychiatry
University Hospital
Cologne, Germany
Zheng Wang
School of Communication
Ohio State University
Columbus OH, USA
Walther C. Zimmerli
Graduate School
Humboldt University
Berlin, Germany
Introduction
Harald Atmanspacher and Sabine Maasen
Reproducibility has become a hot topic both within science and at the interface of science and society. Within science, reproducibility is threatened, among other things, by new tools, technologies, and big data. At the interface of science and society, the media are particularly concerned with phenomena that question good scientific practice. As bad news sell, today problems of reproducibility seem to be ranked right next to fraud. The economy, and especially the biotechnology economy, is interested in innovation based upon novel yet robust knowledge and politics in the so-called knowledge societies seek to base their decisions on best evidence, yet is regularly confronted with competing expertise.
A key step toward increasing attention to deep problems with reproducible findings in science was the paper “Why most published research findings are false” by Ioannidis (2005). One among many recent urging proclamations following it was published in The Scientist magazine (Grant 2012):
The gold standard for science is reproducibility. Ideally, research results are only worthy of attention, publication, and citation if independent researchers can reproduce them using a particular study's methods and materials. But for much of the scientific literature, results are not reproducible at all. The reasons and remedies for this state of affairs was the topic of a recent panel discussion titled “Sense and Reproducibility”, held at the annual meeting of the American Society for Cell Biology in San Francisco, California. … The panel offered suggestions, such as raising journals' publication standards, establishing the use of electronic lab notebooks at research facilities, and helping laboratory supervisors provide improved supervision by reducing the size of labs.
Since about a decade voices abound – both in academia and in the media – that lament lacking reproducibility of scientific results and urgently call for better practice. Given that scientific achievements ultimately rest upon an effective division of labor, it is of paramount importance that we can trust in each other's findings. In principle, they should be reproducible – as a matter of course; however, we often simply rely on the evidence as published and proceed from there. What is more, current publication practices systematically discourage replication, for it is novelty that is associated with prestige. Consequently, the career image of scientists involved with cutting-edge research typically does not include a strong focus on the problems of reproducing previous results.
And, clearly, there are problems. In areas as diverse as social psychology (Nosek 2012), biomedical sciences (Huang and Gottardo 2013), computational sciences (Peng 2011), or environmental studies (Santer et al. 2011), 1 serious flaws in reproducing published results have been and keep being detected. Initiatives have been launched to counter what is regarded as dramatically undermining scientific credibility. Whether due to simple error, misrepresented data, or sheer fraud, irreproducibility corrupts both intra-academic interaction based on truth and the science–society link based on the trustworthiness of scientific evidence.
Among the initiatives introduced to improve the current state of affairs we find workshops, roundtables, and special issues addressing the topic, e.g., in Nature (Schooler 2011, Baker et al. 2012). The journal Biostatistics changed its policy with a focus on reproducible results in an editorial by Peng (2009). The journal Science devoted a special issue to the topic in December 2011, and later revised its publication guidelines concerning the issue of reproducibility (McNutt 2014). Three prominent psychology journals jointly established a “reproducibility project” recently, 2 and the journal PLOS ONE launched a “reproducibility initiative” in 2012. The European Journal of Personality published recommendations for reproducible research (Asendorpf et al. 2013) as the result of an expert meeting on “reducing non-replicable findings in personality research.”
Funding agencies have also joined forces: e.g., the National Science Foundation of the United States created the “Sustainable Digital Data Preservation and Access Network Partners” (DataNet) program to provide an infrastructure for data-driven research in 2007. And, very recently, the National Academy of Sciences of the United States hosted an internal symposium titled “Protecting the Integrity of Science” (Alberts et al. 2015). An extensive report on reproducible research as an ethical issue is due to Thompson and Burnett (2012).
In sum, these examples point to an increased attention toward reproducibility as a topic sui generis. They testify to an increasing interest in reproducibility as a scientific ethos that needs to be upheld – even more so, as new tools and technologies, massive amounts of “big data,” inter- and transdisciplinary efforts and projects, and the complexity of research questions genuinely challenge and complicate the conduction of reproducible research. Many of them call for methods, techniques (including their epistemic and ontological underpinnings), and/or best practices that are intended to improve reproducibility and safeguard against irreproducibility.
The challenges of sound reproducible research have moved into the focus of interest in an increasing number of fields. This handbook is the first comprehensive collection of articles concerning the most significant aspects of the principles and problems, the practices and prospects of achieving reproducible results in contemporary research across disciplines. The areas concerned range from natural sciences and computational sciences to life sciences and social sciences, philosophy, and science studies.
Accordingly, the handbook consists of six parts. Each of them will be introduced by separate remarks concerning the background and context of aspects and issues specific to it. These introductory remarks will also contain brief summaries of the chapters in it and highlight particularly interesting or challenging features.
Part I covers contextual background that illuminates the roots of the concept of reproducibility in the philosophy of science and of technology (Tetens, Zimmerli), and addresses pertinent historical and sociological traces of how reproducibility came to be practiced (Steinle, Collins). Part II frames the indispensable role that statistics and probability theory play in order to assess and secure reproducibility. Basic statistical concepts (Stahel), new ideas on model selection and comparison (Shiffrin and Chandramouli), the difficult methodology of meta-analysis (Ehm), and the novel area of data mining and knowledge discovery in big-data science (Estivill-Castro) are covered.
Parts III–V are devoted to three main areas of contemporary science: physical sciences, life sciences, and social sciences. Part III includes the viewpoints of computational physics (Bailey, Borwein, and Stodden), severe novel problems with reproducibility in complex systems (Atmanspacher and Demmel), the field of extreme and rare events (Kantz), and reproducibility in climate research (Feulner). Part IV moves to the life sciences, with articles on drug discovery and development (Martic-Kehl and Schubiger, Folkers and Baier), the neurobiological study of cortical networks (Lengler and Steger), cognitive neuroscience (Anderson) and social neuroscience (Vogeley).
Part V offers material from the social sciences: a critical look at the reduction of complex processes to numbers that statistics seems to render unavoidable (Porter), innovative strategies to explore question order effects in surveys and polls (Wang and Busemeyer), original views on public opinion research (Keller), issues of reproducibility as indicated in the “blogosphere” (Reinhart), and an in-depth study of notorious problems with reproducibility in scholarly communication (Franzen).
Part VI widens the perspective from reproducibility as a problem in scientific disciplines (in the narrow sense) to literature and literature studies (Bezzola Lambert) and psychopathology and psychoanalysis (Emrich). There is a clear shift in viewpoint here from the attempt to repeat experiments and reproduce their results to an analysis of why strict repetition is not only impossible but also undesirable. Another article (Atmanspacher) proposes that the way reproducibility is studied needs to be adapted to the granularity of the description of the system considered. The final contribution (Maasen) leads us back to the science–society link and the impact of extrascientific forces on research that often remains underrepresented or even disregarded.
This volume investigates the principles, problems, and practices that are connected with the concept of reproducibility, but there is a fourth “p-word” in addition: prospects. In some of the chapters the point is not only to understand principles, address problems, or scrutinize practices – there can also be a strongly constructive dimension in research questions that, on the surface, suffer from a lack of reproducibility. One pertinent example in this volume is the paper by Anderson, who builds on the limited reproducibility of neural correlates of mental states and proposes new ways of interpreting them coherently. Another example is the radical shift in theories of decision making proposed by Wang and Busemeyer, which furthers our understanding of order effects in sequential decisions for which no systematic and consistent modeling framework was available until recently.
While the handbook as a whole is dedicated to explore “reproducibilities” on cognitive and technical levels, its other goal is to scrutinize the notorious difficulties with producing reproducibility in a reflexive manner. Such reflexive perspectives are scattered throughout the individual contributions to this book, addressing, e.g., challenges enforced by information technology. However, parts I, V, and VI in particular inquire into philosophical, historical, social, and political contexts and their interaction with notions and practices of reproducibility. In this way, they elucidate the manifold conditions and consequences of the quest for reproducibility induced by those interactions.
From these perspectives, science is regarded as both a socio-epistemic endeavor and a highly specialized, yet integral part of society. Science and its particular modes of producing knowledge thus cannot be understood without considering them as historically evolved practices and as objects of societal expectations: Most importantly, science is expected to produce reliable knowledge. While peers within the same scientific discipline – under the considerations and within the limits outlined in the chapters of this volume – apply established means to verify their research, things become markedly problematic in interdisciplinary research with differing standards, and they may become impossible in extra-scientific contexts.
To put this in simple terms: Society cannot but trust in science and its internal procedures to produce knowledge that is both true and, hence, trustworthy. Therefore, the recently perceived lack of reproducibility is not only an intra-scientific affair but is also critically noticed by societal actors such as the mass media (e.g., Zimmer 2012). Moreover, it is addressed by science political actors (Maasen, this volume) or editors of scholarly journals (Franzen, this volume) and by concrete measures such as codes of conduct to improve scientific practice.
Finally, a few remarks concerning terminology are in order. Although we chose to use the notion of reproducibility to characterize the topic of this volume, there is another term that is often used interchangeably: replicability. This is visible in the surprisingly few volumes or reviews on the topic that can be found in the literature (e.g., Sidman 1960, Smith 1970, Krathwohl 1985, Schmidt 2009). And it also shows in the contributions to this volume – some authors prefer one, some the other notion, and (to the best of our knowledge) there is no authoritative delineation between them.
One feature, however, seems significant for “replication” and is very rarely addressed in terms of “reproduction”: As the literature shows (e.g., Charron-Bost 2009), replication is mostly referred to if one focuses on how data are copied (rather than reproduced by repeated observations or measurements). This is particularly evident in information technology communities (distributed systems, databases), but of course also in genetic (DNA) replication.
A third related notion is repeatability. It is primarily adopted if an observational act (measurement) or a methodological procedure (data analysis) is repeated, disregarding the replication or the reproduction of the result of that act or procedure. An experiment may be repeated within the same laboratory or across laboratories. This difference is sometimes addressed as the difference between “internal” and “external” repetitions. Whether or not its results and conclusions from them are compatible would be a matter of reproducibility.
Whatever the most appropriate notion for “reproducibility” may be, this volume shows that it would be wrong to think that it can be universally stipulated. Depending on context, one may want to reproduce system properties characterizable by single values or distributions of such values. One may be interested in patterns to be detected in data, or one may try to reproduce models inferred from data. Or reproducibility may not relate to quantitative measures at all. All these “reproducibilities” unfold an enormous interdisciplinary tension which is inspiring and challenging at the same time.
As reflexive interactions between science and society take place, novel discourses and means of control emerge which are ultimately designed to enforce the accomplishment of truly reliable knowledge. All this happens in addition to the variety of reproducibilities as well as in view of ever-more contexts regarded as relevant and ever-changing (technical) conditions. One prospect for reproducibility seems to be clear: Challenges from within science will certainly continue to meet with those from the outside and jointly leave their traces on the ways in which science and technology produce robust knowledge. Reproducibility will remain at the heart of this process.
_______________________________
The nucleus of this volume has been a long-term research project on reproducibility at Collegium Helveticum, an interdisciplinary research institution jointly operated by the University of Zurich and the Swiss Federal Institute of Technology (ETH) Zurich. Our engagement with the project as editors, grounded in the exact sciences and in the area of science studies, emerged from our long lasting affiliation with the Collegium as associate fellows. We appreciate great encouragement and support by the Collegium and its fellows, in particular by Gerd Folkers and Martin Schmid.
An editorial conference at the Munich Center for Technology in Society (Technical University Munich) in fall 2014 with all authors was instrumental for the preparation of the volume in a consistent fashion, with numerous cross-references among the papers. Our thanks go to the staff of the center for organizing this event. Various colloquia, workshops, and symposia on reproducibility at Zurich and Munich have been influential all along the way.
But most of all, we want to express our gratitude to the contributors to this volume. For none of them, thinking and writing on reproducibility is their regular day job. Nevertheless, we realized so much enthusiasm about this project that any possible grain of doubt concerning its success dissolved rapidly. As all reviewers of the proposal for the project emphasized, the volume is of utmost timeliness and significance, and this spirit pervaded all our conversations and correspondences with its authors. Without their deep commitment, and without the support of Susanne Steitz-Filler and Sari Friedman at Wiley, this book would not have become reality.
References
Alberts, B., Cicerone, R.J., Fienberg, S.E., Kamb, A., McNutt, M., Nerem, R.M., Schekman, R., Shiffrin, R., Stodden, V., Suresh, S., Zuber, M.T., Kline Pope, B., and Hall Jamieson, K. (2015): Self-correction in science at work. Science348, no. 6242, 1420–1422.
Asendorpf, J., Connor, M., de Fruyt, F., de Houwer, J., Denissen, J.J.A., Fiedler, K., Fiedler, S., Funder, D.C., Kliegl, R., Nosek, B.A., Perugini, M., Roberts, B.W., Schmitt, M., Vanaken, M.A.G., Weber, H., and Wicherts, J.M. (2013): Recommendations for increasing replicability in psychology. European Journal of Personality27, 108–119.
Baker, D., Lidster, K., Sottomayor, A., and Amor, S. (2012): Research-reporting standards fall short. Nature492, 41.
Charron-Bost, B., Pedone, F., and Schiper, A., eds. (2009): Replication – Theory and Practice, Springer, Berlin.
Grant, B. (2012): Science's reproducibility problem. The Scientist, 18 December 2012.
Huang, Y., and Gottardo, R. (2013): Comparability and reproducibility of biomedical data. Briefings in Bioinformatics14, 391–401.
Ioannidis, J. (2005): Why most published research findings are false. PLOS Medicine2(8), e124.
Krathwohl, D.R. (1985): Social and Behavioral Science Research, Jossey-Bass, San Francisco.
McNutt, M. (2014): Reproducibility. Science343, 229.
Nosek, B. (2012): An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science7, 657–660.
Open Science Collaboration (2012): An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science7, 657–660.
Open Science Collaboration (2015): Estimating the reproducibility of psychological science. Science349, No. 6251, aac4716/1–8.
Peng, R. (2009): Reproducible research and biostatistics. Biostatistics10, 405–408.
Peng, R. (2011): Reproducible research in computational science. Science334, 1226–1227.
Santer, B.D., Wigley, T.M.L., and Taylor, K.E. (2011): The reproducibility of observational estimates of surface and atmospheric temperature change. Science334, 1232–1233.
Schmidt, S. (2009): Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology13(2), 90–100.
Schooler, J. (2011): Unpublished results hide the decline effect. Nature470, 437.
Sidman, M. (1960): Scientific Research, Basic Books, New York.
Smith, N.G. jr. (1970): Replication research: A neglected aspect of psychological research. American Psychologist25, 970–975.
Thompson, P.A., and Burnett, A. (2012): Reproducible research. CORE Issues in Professional and Research Ethics1, 6.
Zimmer, C. (2012): A sharp rise in retractions prompts calls for reform. New York Times, April 17, D1.
1
These references are a tiny subset of the existing literature on problems with reproducibility. Many more examples will be addressed in the main body of this volume.
2
The project is a large-scale, open collaboration currently involving more than 150 scientists from around the world. The investigation is currently sampling from the 2008 issues of the
Journal of Personality and Social Psychology
,
Psychological Science
, and
Journal of Experimental Psychology: Learning, Memory, and Cognition
; see Open Science Collaboration (2012). The results have been published by the Open Science Collaboration (2015); see also
https://osf.io/ezcuj/
.
PART ICONTEXTUAL BACKGROUNDS
Introductory Remarks
Harald Atmanspacher
The reproducibility of results is considered one of the basic methodological pillars of “good science.” Why? As always, normative rules and other firm beliefs of this kind hardly derive from scientific research per se (although they may be informed by it) – they are typically motivated by or founded on reasons external to science. In this sense, the concept of reproducibility is a paradigm case for historical and philosophical analyses, which are the framework of the contributions to this part of the volume.
One of these extra-scientific reasons refers to a very basic ontological commitment shared by many (if not most) scientists – that there are stable fundamental structures of our universe. In contrast to sense data or introspective data, these ontic structures are assumed to be universal rather than particular. Their stability underlies the assumption that there are laws of nature (such as Maxwell's equations) and universal constants (such as the speed of light) which are valid irrespective of where, when, and by whom they are investigated. If a law of nature is assumed to hold, its empirical predictions must hold with the same sense of stability. And this means that a reproduction of an experiment under the same conditions must lead to the same result. 1
In the contribution by Holm Tetens at the Institute of Philosophy, Free University Berlin, the ontological dimension of science is expressed by its claim to unveil objective truths. But such truths themselves are not directly accessible. Instead, if empirical results are reproducible across laboratories, intersubjective agreement can be generated about those results so that they become part of the accepted body of scientific knowledge. As a supplement to this intersubjective dimension, Tetens indicates the technological aspect that every experiment is a potential prototype of an artificial device that can be controlled and manipulated for particular purposes. The history of philosophy presents this as a special truth criterion as well: Giambattista Vico's verum-factum principle says that “the true and the made are interchangeable” (see, e.g., Miner 1998).
A notion that expresses the idea of stability in mathematical terms is invariance. A property is invariant under a transformation if it does not change when the transformation is applied. The so-called conservation laws in physics are well-known examples, such as the conservation of momentum or energy. Closely related to invariance principles are symmetry principles: In the basic sciences, symmetries are major driving forces for theoretical progress (see, e.g., the classic by Weyl 1952). Of course, this is much less prominent as we move to more complex structures, or to living systems, where most if not all symmetries that govern physics are broken. And as a consequence of broken symmetries, the significance of universal laws and constants declines naturally.
Walther Ch. Zimmerli writes as a philosopher of mind and of technology at the Graduate School of Humboldt University Berlin and amplifies the theme of reproduction in its relation to technology. He advocates that technology is more than just the result of applied science, a position strongly defended by Bunge (1966). Zimmerli's approach comes closer to more recent tenets of “science and technology studies,” which emerged in the 1980s and focus on how social, political, economic, and cultural values affect scientific research and technological innovation. See Mitcham (1994) for an overview of the wide variety of perspectives in this field.
