Critical Thinking E-Book

Donald Maxwell Brunette

• Herausgeber: Quintessence Publishing Co, Inc
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 1208

Veröffentlichungsjahr: 2020

Beschreibung

The Internet has given us access to an unprecedented amount of information, but this development also facilitates the spread of misinformation if we do not think critically and evaluate what we read. The third edition of this classic textbook has been revised based on developments in biomedical research practices over the past decade to provide readers with the most up-to-date resource for navigating the ever-expanding world of dental literature. The book covers rhetoric and logic, the scientific method, components of scientific papers, research strategies, probability and statistics, diagnostic tools and testing, and experiment design. The chapter about searching the dental literature has been updated with the most current resources and search techniques available. A new chapter about clinical decision making uses a decision tree analysis with worked-out calculations to show how critical thinking skills can be used to select the most appropriate treatment decision in a clinical scenario; this is followed by a whole chapter of exercises in critical thinking. This book emphasizes how readers can practically apply critical thinking skills to evaluate scientific literature and thereby make the most informed decisions for their patients and themselves.

Leseprobe

FROM CONTRIBUTOR BEN BALEVI:

In dental school, I was taught to “do dentistry” based on my many instructors' clinical anecdotes that appeared to be true and tested over time in their respective hands. However. this led to inconsistencies in recommendations from different instructors. creating confusion on how to best manage patient care. Although hours of instruction were given to basic science, little instruction was given on clinical research design and how to critically evaluate it and apply it to patient care. The 1990s brought the era of evidence-based medicine, which promoted translating clinical research outcomes and consideration of patient preferences as well as clinical anecdotes in guided clinical decision making. The first edition of Critical Thinking: Understanding and Evaluating Dental Research (1997) was the first textbook to fill this void in dental education and practice.

Critical Thinking became an award-winning textbook and is now available in this third edition. Its success lies in its unique writing style. Textbooks are historically written in the dry tone of the passive voice. consistent with the objectivity of science. In contrast. Dr Brunette adopted a narrative writing style that engaged the reader to view the complex concept of scientific investigation through personal experience. At the time, teaching through stories was rare and almost frowned upon as being "beneath" science, which is supposed to be free of any perceived subjective influence. Yet. Dr Brunette was ahead of his time, and this style of writing has been shown to be effective for teaching complex concepts. especially to students reading about them for the first time. This style has since become more common and is now even encouraged as a mode of teaching.

This book is fundamental for teaching the basic skills of evidence-based dentistry. These skills are key to managing the dentist-patient relationship, which has changed since I graduated in 1987. At that time, the relationship was predominately paternalistic, where I was the gatekeeper of dental knowledge and my patient's predominant access was through me. I had the power to discriminate which knowledge I felt the patient should have access to and the patient had to take me at face value with no opportunity to verify what I decided to share. However. as patients gained more access to information through the Internet. dental knowledge became mainstream. This allowed patients to assert autonomy in their dental care. and my relationship with my patients changed from one where I dictated treatment to one of shared decision making between us.Today, my patients expect me to help them critically evaluate dental knowledge so we can make informed decisions about their treatment together. This is where the skills taught in Critical Thinking become invaluable to a clinician.

Critical Thinking

Understanding and Evaluating Dental Research

Third Edition

Library of Congress Cataloging-in-Publication Data

Names: Brunette, Donald Maxwell, author.

Title: Critical thinking : understanding and evaluating dental research / Donald Maxwell Brunette.

Description: Third edition. | Batavia, IL : Quintessence Publishing Co, Inc, [2019] | Includes bibliographical references and index.

Identifiers: LCCN 2019017538 (print) | LCCN 2019018451 (ebook) | ISBN 9780867158014 (ebook) | ISBN 9780867158007 (softcover)

Subjects: | MESH: Dental Research

Classification: LCC RK80 (ebook) | LCC RK80 (print) | NLM WU 20.5 | DDC 617.6/027--dc23

LC record available at https://lccn.loc.gov/2019017538

© 2020 Quintessence Publishing Co, Inc Quintessence Publishing Co Inc 411 N Raddant Road Batavia, IL 60510 www.quintpub.com

5 4 3 2 1

All rights reserved. This book or any part thereof may not be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher.

Editor: Zachary Kocanda Design: Sue Zubek Production: Sarah Minor

Printed in Korea

Dedication

Preface

1 Reasons for Studying Critical Thinking

2 Scientific Method and the Behavior of Scientists

3 The Components of a Scientific Paper

4 Rhetoric

5 Searching the Dental Literature

with Helen L. Brown

6 Logic: The Basics

7 Introduction to Abductive and Inductive Logic: Analogy, Models, and Authority

8 Causation

9 Quacks, Cranks, and Abuses of Logic

10 Elements of Probability and Statistics, Part 1: Discrete Variables

11 Elements of Probability and Statistics, Part 2: Continuous Variables

12 The Importance of Measurement in Dentistry

13 Errors of Measurement

14 Presentation of Results

15 Diagnostic Tools and Testing

with Ben Balevi

16 Research Strategies and Threats to Validity

17 Observation

18 Correlation and Association

19 Experimentation

20 Experiment Design

21 Statistics As an Inductive Argument and Other Statistical Concepts

22 Judgment

23 Introduction to Clinical Decision Making

with Ben Balevi

24 Exercises in Critical Thinking

Appendices

Index

To the late Doug Waterfield, with whom I collaborated extensively once it became clear through the in vivo studies of Babak Chehroudi that macrophages played an important role in tissue response to the topography of implants. Doug was a tireless advocate for students in the Faculty of Dentistry and spared no effort to help them succeed.

The need for a third edition of Critical Thinking was driven by developments in dental and biomedical research conditions and practice, as well as reader comments. The third edition emphasizes the practical application of critical thinking in the production and evaluation of scientific publications. A number of issues are now more prominent.

Thanks to the Internet, patients of today have much more information (both accurate and inaccurate) than the patients of yesteryear. The consequence to dentists is that they must explain and be prepared to justify their clinical decisions with evidence. This requires that dentists be capable of unearthing relevant information and evaluating alternatives by accepted criteria so that patients can be confident in their dentist’s recommendations. In line with the virtual revolution entailed in dealing with the ever-expanding scientific literature, the chapter on searching the dental literature now emphasizes scholarly publishing issues at all stages of research in addition to covering more traditional topics such as types of resources and searching techniques. The preparation of systematic reviews is discussed in greater detail, as are the strengths and weaknesses of various services such as Web of Science and Google Scholar as well as approaches to identifying relevant evidence in grey literature and its use in reviews. The author of the revised chapter is Helen Brown, who has taken over from Kathy Hornby, for whose contribution I continue to be grateful. Helen has master’s degrees in English literature and library science, along with frontline experience resolving student problems in literature searches. She has made significant changes in the emphasis of this chapter. For example, the increasing trend toward open access publications and open science initiatives is addressed, providing important information for both prospective authors and readers with limited access to expensive subscription-based resources. The strengths and weaknesses of the h-index, now widely used by researchers to demonstrate the impact of their work, is also now considered, and there is updated information on impact factors of dental journals and other bibliographic tools.

A second new contributor is Ben Balevi, who takes over the chapter on diagnostic tests and measures from my former student Carol Oakley, whose contributions continue to influence the current chapter. Ben has a master’s degree in evidence-based dentistry and has been recognized for his contributions in this area by the American Dental Association.

There have been considerable changes in other aspects of the book as well. A new chapter discusses the function of each section of the scientific paper so that researchers can check that their papers are fulfilling the expectations of readers and referees.

The coverage on logic has expanded to include more material on informal logic, for example, the use of Walton’s critical questions for evaluating causation. The informal logic used in the refereeing process in which the burden of proof shifts from authors to reviewers and back has been added to complement the argumentation maps found in the chapter on judgment that features tactical details such as rebuttals, caveats, and supporting evidence. The intent is to give investigators a practical approach to anticipating and dealing with criticisms. The chapter on judgment now includes materials of both historical (Darwin’s decision to marry and Benjamin Franklin’s moral algebra) and novel decision-making heuristics, such as fast frugal heuristics.

Clinical decision making has been given its own chapter and expanded to include detailed, worked-out calculations and clinical scenarios. In addition to decision tree analysis, two elements of central importance to patients are presented with worked-out calculations: cost-effectiveness analysis and willingness to pay analysis.

The chapter on diagnostic tools and testing has been expanded to include new worked-out calculations and examples, as well as illustrations to clarify this topic, which many find difficult and confusing.

The chapter on experimentation now covers randomized field trials, a method employed in causally dense environments such as business problems, often combined with trial-and-error approaches. The technique is evolving but has been found useful by organizations such as Google that run thousands of randomized field trials.

The statistics chapter has expanded coverage of Bayesian statistics, as this approach is expected to become more widely used. The chapter on presenting results contains new illustrations and more extensive discussion of the integrity-based principles for displaying information emphasized by Tufte. The section on the behavior of scientists has been expanded with discussion on the reproducibility crisis now shaking some scientists’ faith in the reliability of reported results, even including those published in high-impact journals. All chapters have also been improved by the addition of updates, revisions, or examples.

I acknowledge the support of my colleagues who have never hesitated to provide advice in their respective domains of expertise. I also thank my former students, friends, and collaborators, whose insights inform numerous sections. Regular University of British Columbia (UBC) contributors in this way include Babak Chehroudi, Tim Gould, Hugh Kim, Chris Overall, Mandana Nematollahi, Eli Konorti, and Nick Jaeger (Electrical and Computer Engineering). Colleagues outside UBC, Christopher McCulloch (University of Toronto), Ken Yaegaki (Nippon Dental University), and Doug Hamilton (Western University), have also provided advice and encouragement. I also thank my collaborators at the Swiss Institute of Technology (Zurich), Professors Marcus Textor and Nicolas Spencer, and my former post-doc Dr Marco Wieland, for continuing my education in surface science. Of course, I would be an ingrate if I failed to acknowledge the continuing support of my wife Liz, sons Max and Regan (and their partners Heather and Malizza), who provided various examples used in the book including the exploits of my problem-solving grandson Calixte and developing gymnast granddaughter Genèvieve.

I also thank Quintessence for their ongoing support through three editions of this work. This edition has been facilitated through the efforts of Bill Hartman and Bryn Grisham. Zach Kocanda has been exemplary in his attention to detail, often under trying circumstances, in editing the manuscript.

1

Reasons for Studying Critical Thinking

“

It has happened more than once that I found it necessary to say of one or another eminent colleague, ‘He is a very busy man and half of what he publishes is true but I don’t know which half.’”

ERWIN CHARGAFF1

Critical Thinking

Critical thinking has been defined many ways, from the simple—“Critical thinking is deciding rationally what to or what not to believe”2—to the more detailed “Critical thinking is concerned with reason, intellectual honesty, and open-mindedness, as opposed to emotionalism, intellectual laziness, and closed-mindedness”3—to the nearly comprehensive:

Critical thinking involves following evidence where it leads; considering all possibilities; relying on reason rather than emotion; being precise; considering a variety of possible viewpoints and explanations; weighing the effects of motives and biases; being concerned more with finding the truth than with being right; not rejecting unpopular views out of hand; being aware of one’s own prejudices and biases; and not allowing them to sway one’s judgment.3

Self-described practitioners of critical thinking range from doctrinaire postmodernists who view the logic of science with its “grand narratives” as inherently subordinating4 to market-driven dentists contemplating the purchase of a digital impression scanner. In this book, critical thinking, and in particular the evaluation of scientific information, is conceived as “organized common sense” following Bronowski’s view of science in general.5 Of course, common sense can be quite uncommon. A secondary use of the term critical thinking implies that common sense involves a set of unexamined and erroneous assumptions. For example, prior to Galileo, everyone “knew” that heavy objects fell faster than lighter ones. Critical thinking as organized common sense takes the systematic approach of examining assumptions. The professional use of critical thinking is particularly complex for dental professionals because they live in two different worlds. On the one hand, they are health professionals treating patients who suffer from oral diseases. On the other hand, dentists typically also inhabit the business world, where decisions may be based on the principle of maximizing income from their investment. Dental practice is based only very loosely on responding to disease6; less than one-third of patient visits result in identifying a need for restorative care.7 Twenty percent of work is elective, such as most of orthodontics, tooth whitening, and veneers, and typically that work comprises the most lucrative aspects of practice. Thus, the information that must be evaluated in performing these disparate roles covers the spectrum from advertisements to financial reports to systematic meta-analysis of health research.

Dentists are health professionals, people with specialized training in the delivery of scientifically sound health services. The undergraduate dental curriculum is designed to give dental students the basic knowledge to practice dentistry scientifically, at least to the extent allowed by the current state of knowledge. But if any guarantee can be made to dental students, it is that dentistry will change, because the knowledge base of biomedical and biomaterial sciences grows continually. Most dentists today have had to learn techniques and principles that were not yet known when they were in dental school. In the future, as the pace of technologic innovation continues to increase and the pattern of dental diseases shifts, the need to keep up-to-date will be even more pressing. Means of staying current include interacting with colleagues, reading the dental literature, and attending continuing education courses—activities that require dentists to evaluate information. Yet, there is abundant historical evidence that dentists have not properly evaluated information. Perhaps the best documented example in dentistry of a widely accepted yet erroneous hypothesis is the focal infection theory. Proposed in 1904 and accepted by some clinicians until the Second World War, this untested theory resulted in the extraction of millions of sound teeth.8 But errors are not restricted to the past; controversial topics exist in dentistry today because new products or techniques are continually introduced and their usefulness debated. Ideally, dentists should become sophisticated consumers of research who can distinguish between good and bad research and know when to suspend judgment. This goal is different from proposing that dentists become research workers. One objective of this book is to provide the skills enabling a systematic method for the evaluation of scientific papers and presentations.

A marked addition to the challenges of dental practice in recent years is that patients have increased access through the Internet to information as well as misinformation. Dentists thus are more likely to be questioned by patients on proposed treatment plans and options. In responding to such questions, it is clearly advantageous for dentists to be able to present a rational basis for their choices. Chapter 23 covers an evidence-based approach to clinical decision making and appendix 9 provides a template for dental offices to use in documenting their decisions based on recent evidence.

A systematic approach to analyzing scientific papers has to be studied, because this activity requires more rigor than the reasoning used in everyday life. Faced with an overabundance of information and limited time, most of us adopt what is called a make-sense epistemology. The truth test of this epistemology or theory of knowledge is whether propositions make superficial sense.9 This approach minimizes the cognitive load and often works well for day-to-day short-term decision making. In 1949, Zipf of Harvard University published Human Behaviour and the Principle of Least Effort, in which he stated:

The Principle of Least Effort means, for example, that in solving his immediate problems he will view these against a background of his probable future problems, as estimated by himself. Moreover, he will strive to solve his problems in such a way as to minimize the total work that he must expend in solving both his immediate problems and his probable future problems.10

Zipf used data from diverse sources ranging from word frequencies to sensory sampling to support his thesis. Although the methods and style of psychologic research have changed, some more recent discoveries, such as the concept of cognitive miser in studies of persuasion,11 coincide with Zipf’s principle. Kahneman in Thinking, Fast and Slow has elevated the principle to a law noting that we “conduct our mental lives by the law of least effort.”12

In science, the objective is not to make easy short-term decisions but rather to explain the phenomena of the physical world. The goal is accuracy, not necessarily speed, and different, more sophisticated, more rigorous approaches are required. Perkins et al9 have characterized the ideal skilled reasoner as a critical epistemologist who can challenge and elaborate hypothetical models. Where the makes-sense epistemologist or naive reasoner asks only that a given explanation or model makes intuitive sense, the critical epistemologist moves beyond that stage and asks why a model may be inadequate. That is, when evaluating and explaining, the critical epistemologist asks both why and why not a postulated model may work. The critical epistemologist arrives at models of reality, using practical tactics and skills and drawing upon a large repertoire of logical and heuristic methods.9

Table 1-1| Level of evidence guideline recommendations of the United States Agency for Healthcare Research and Quality

Level

Type of study

Grade of recommendation

1

Supportive evidence from well-conducted RCTs that include 100 patients or more

A

2

Supportive evidence from well-conducted RCTs that include fewer than 100 patients

A

3

Supportive evidence from well-conducted cohort studies

A

4

Supportive evidence from well-conducted case-control studies

B

5

Supportive evidence from poorly controlled or uncontrolled studies

B

6

Conflicting evidence with the weight of evidence supporting the recommendation

B

7

Expert opinion

C

RCTs, randomized controlled trials.

Psychologic studies have indicated that everyday cognition comprises two sets of mental processes, System 1 and System 2, which work in concert, but there is some debate whether they operate in a parallel or sequential manner. System 1 operates quickly and effortlessly, whereas System 2 is deliberate and requires attention and effort.12 System 2 is a rule-based system, and engaging System 2 is the surest route to fallacy-free reasoning.13 System 2 becomes engaged when it catches an error made by the intuitive System 1. The good news is that extensive work by Nisbett and colleagues (briefly reviewed by Risen and Gilovich13) showed that people can be trained to be better reasoners and that people with statistical backgrounds were less likely to commit logical fallacies. Nisbitt and colleagues further demonstrated that even very brief training was effective in substantially reducing logical and statistical errors. Thus this book has chapters on logic and statistics.

A second objective of the book is to inculcate the habits of thought of the critical epistemologist in readers concerned with dental science and clinical dentistry.

The scope of the problem

In brief, the problems facing anyone wishing to keep up with developments in dentistry or other health professions are that (1) there is a huge amount of literature, (2) it is growing fast, (3) much of it is useless in terms of influencing future research (less than 25% of all papers will be cited 10 times in all eternity,14 and a large number are never cited at all), and (4) a good deal of the research on a clinical problem may be irrelevant to a particular patient’s complaint

The actual rate of growth of the scientific literature has been estimated to be 7% per year of the extant literature, which in 1976 comprised close to 7.5 million items.11 This rate of growth means that the biomedical literature doubles every 10 years. In dentistry, there are about 500 journals available today.15 Many dental articles are found in low-impact journals, but, ignoring these, there were still 2,401 articles published in 1980 in the 30 core journals.16 More recently, it has been estimated that about 43,000 dental-related articles are published per year.

However, the problem is not intractable. Relman,17 a former editor of the New England Journal of Medicine, believes that most of the important business of scientific communication in medicine is conducted in a very small sector of top-quality journals. The average practitioner needs to read only a few well-chosen periodicals.17 The key to dealing with the problem of the information explosion is in choosing what to read and learning to evaluate the information.

Dentists are exposed to diverse information sources, and the important issues vary depending on the source. For example, a dentist may wish to determine whether potassium nitrate toothpastes reduce dentin hypersensitivity. One approach would be to look up a systematic review on this topic in the Cochrane Library,18 which many regard as the highest level in the hierarchy of evidence (Table 1-1). The skills required to understand the review would include a basic knowledge of statistics and research design. The same dentist, facing the competitive pressures of his or her local market, might also want to determine whether a particular laser-bleaching process should be adopted for the practice. In that instance, there might not be a relevant Cochrane review, and there may not even be a relevant paper in a refereed journal to support a decision. Available evidence might consist of advertising brochures and anecdotes of colleagues. The dentist may have to employ a different set of skills, ranging from evaluating the lie factor in graphics (see chapter 14) to disentangling rhetoric from fact. Advertisements and salesmanship are persuasive exercises; the chapter on rhetoric (chapter 4) deals with means of persuasion.

Typically, dentists acquire information on innovative procedures through participation in networks in which their colleagues supply informal data on the effectiveness of the innovations. Nevertheless, dentists cite reading peer-reviewed dental literature and experimental studies as the gold standard for determining the quality of innovations.19 New technology is often introduced into their practices through trial and error; dentists take the pragmatic approach of directly determining what works in their hands in their practice.19 Doubtless, some of the personal and financial expenses typical of the trial-and-error approach could be reduced with more effective evaluation of information prior to selecting a material or technique for testing.

This book focuses on evaluating refereed scientific papers, but many of the issues of informational quality and questions that should be asked apply equally to other less formal channels of communication.

What is a scientific paper?

The Council of Biology Editors defines a scientific paper as follows:

An acceptable primary scientific publication must be the first disclosure containing sufficient information to enable peers (1) to assess observations; (2) to repeat experiments; and (3) to evaluate intellectual processes; moreover, it must be sensible to sensory perception, essentially permanent, available to the scientific community without restriction, and available for regular screening by one or more of the major recognized secondary services.20

Similar ideas were stated more succinctly by DeBakey,21 who noted that the contents of an article should be new, true, important, and comprehensible. A good deal of the literature circulated to dentists does not meet these requirements. But even excluding the throwaway or controlled-circulation magazines that are little more than vehicles for advertisements, the amount of information published annually appears formidable.

One approach to dealing with a large number of papers is to disregard original papers and receive information secondhand. Dental and medical journals present reviews of current research in specific clinical or scientific fields; some journals, such as Dental Clinics of North America and the Journal of Evidence Based Dentistry, are exclusively devoted to this approach. Although this tactic reduces the volume of literature to be covered, it does not solve the problem of evaluating the information contained in the reviews. To perform this task effectively, a researcher must be able to assess the soundness of the reviewer’s conclusions. In deciding to accept information secondhand, the researcher is also deciding whether the author of the review is a reliable, objective authority. Thus, the problem of evaluation has been changed, but not eliminated.

This book focuses on the primary literature, where it is hoped that new, true, important, and comprehensible information is published. The systematic review, a relatively new review form, attempts to deal with some of the more glaring problems of traditional reviews and is covered briefly in chapter 5. Although useful for some purposes, the systematic review has its own shortcomings, and the researcher must judge how these affect the conclusions. Journals vary in quality; chapter 5 also discusses bibliometric approaches of ranking journals. In the following section, I present a brief review of how articles get published that may help explain some of this variation.

The Road to Publication

The author

The author’s goal is to make a significant contribution to the scientific literature: a published paper. To accomplish that goal, the author will have to produce a submission for publication whose contents are new, true, important, and comprehensible. Moreover, the author wants to publish the paper in a journal whose readers will likely find the paper of interest and hopefully be influenced by it. As journals vary in the rigor they demand and the length of papers they accept, the author needs to identify the best journal for his or her purposes.

Refereed versus nonrefereed journals

The first hurdle faced by an article submitted for publication is an editor’s decision on the article’s suitability for the journal. Different journals have different audiences, and the editors are the arbiters of topic selection for their journal. Editors can reject papers immediately if they think the material is unsuited to their particular journal.

In some journals, acceptance or rejection hinges solely on the opinion of the editor. However, this method is problematic because informed decisions on some papers can only be made by experts in a particular field. Therefore, as a general rule, the most highly regarded journals ask the opinion of such specialists, called referees or editorial consultants. Referees attempt to ensure that a submitted paper does not demonstrably deviate from scientific method and the standards of the journal. Whether a journal is refereed can be determined by consulting Ulrich’s Periodicals Directory (ulrichsweb.com). Editors usually provide referees with an outline of the type of information that they desire from the referee. The criteria for acceptance will necessarily include both objective (eg, obvious errors of fact or logic) and subjective (eg, priority ratings) components. Unfortunately, the task of refereeing is difficult and typically unpaid. Refereeing is often squeezed in among other academic activities, so it should not be surprising that it sometimes is not done well and that referees often disagree.

Studies of the reliability of peer-review ratings are disappointing for readers wanting to keep faith in the peer-review system. Reliability quotients, which can range from 0 (no reliability) to 1 (perfect reliability), for various attributes of papers submitted to a psychology journal22 follow:

• Probable interest in the problem: 0.07

• Importance of present contribution: 0.28

• Attention to relevant literature: 0.37

• Design and analysis: 0.19

• Style and organization: 0.25

• Succinctness: 0.31

• Recommendation to accept or reject: 0.26

Despite such issues, there is evidence that the review process frequently raises important issues that, when resolved, improve the manuscript substantially.23

After consulting with referees, the editor decides whether the paper should be (1) published as is—a comparatively rare event; (2) published after suitable revision; or (3) rejected. Journals reject papers in proportions varying from 0% to greater than 90%. The literature available to dental health professionals ranges the spectrum of refereed to nonrefereed, from low (or no) rejection rates to high rejection rates. The Journal of Dental Research, for example, used to have a 50% rejection rate (Dawes, personal communication, 1990), but that has risen so that 25 years later some 90% of submissions are rejected.24 Even among high-impact journals, however, there is no guarantee that the referees did a good job. In fact, these considerations only serve to reinforce the view “caveat lector”—let the reader beware.

Editors and referees

The editor of the journal and the referees are the “gatekeepers” who decide whether a manuscript is accepted. In science the basic rule appears to be something akin to “if it doesn’t get published, it doesn’t exist.” Thus the rewards in science go to those who publish first, not the first scientist to observe a phenomenon. Obviously, pleasing these gatekeepers is essential to a scientific career.

The editor and the referees are the representatives of the readers of the journal. They protect the readers from wasting their time on obviously erroneous or uninteresting or unsuitable or unoriginal or opaque or trivial submissions. The papers in the journal must be relevant to the readership.

An important part of the editor’s job is to protect authors from unjust criticism that can arise from such things as personal animosity between an author and a referee or an attempt by a referee to block publication of a competitor’s work. Unfortunately, the scientists who are best able to evaluate a submission may be individuals who can suffer most from its publication, as for example occurs when the referee’s own work is “scooped” (ie, published earlier by a competitor).

To justify readers’ expenditure of time the paper should address a significant problem or concern and provide a significant amount of information. The length of journal articles varies; some journals publish “letters” rather than full-length papers for interesting but only briefly developed findings. Editors are interested in publishing papers that are likely to be cited in the future or, expressed another way, are building blocks for future research or clinical application. Tacker25 notes that journals differ in the sophistication of their readership. A general medical journal (eg, JAMA) is written at the comprehension level of a third-year medical student, whereas a specialty journal is written for a first- or second-year resident. A scientific journal should be understandable to third- or fourth-year PhD candidates or above in the general field.

The editor

The editor decides ultimately whether to accept or reject a submission. As a general rule the editor is an unpaid (or lowly paid) volunteer of distinguished standing in the field covered by the journal. The editor defines the scope of the journal (ie, what subjects are published in it), and if a manuscript falls outside the journal’s mandate, it will probably be returned promptly to the author. Similarly, an editor may reject a paper on the grounds that a submission does not advance the field sufficiently or has a low potential for future impact. Such judgments are subjective but nevertheless may need to be made. I call this the “de gustibus” standard after the Latin adage, De gustibus non est disputandum: “In matters of taste, there can be no disputes.” As the adage indicates, if a decision is made on this basis it will be difficult to persuade the editor to reverse it.

Editors are often responsible for diverse other tasks such as recruiting referees and persuading them to submit their reviews in a timely manner. Some journals have associate editors who oversee submissions in their area of expertise, and the editor must coordinate their activities as well as consult with editorial boards and deal with the various business matters. Despite the importance of their job, editors are not always appreciated by their colleagues, who may resent some decisions. Chernin playfully suggests, “Editors are also the people who separate the wheat from the chaff and frequently publish the chaff.”26

After the manuscript is accepted by the editor, it may be passed on to a managing editor to take the manuscript though the production and publication process. Day27 states that editors and managing editors have jobs that are made impossible by the attitudes of authors who submit to their journals. For example, authors might ignore the rules and conventions specified by the journal (eg, the format for the references). Or authors and referees may have irreconcilable views, and the editor may be caught in the middle. Given that the editor’s decision could affect the author’s career, it is clearly wise not to irritate editors or referees, but rather to make their job in dealing with the submission as easy as possible. That is, authors want the editor to like them, and as has been extensively studied in the psychology literature,28 liking can be a key factor in persuasion, in this case persuading the editor that the submission should be published.

An indicator of what editors want is provided by the instructions given to referees of journals, often in the form of a checklist or a score sheet that incorporates specific questions for reviews completed online. As an example, I compiled an indicator of some of editors’ concerns by simply looking at the instructions sent to me by ten journals. The following characteristics were emphasized:

• 90% (ie, 9/10) concise

• 70% clear

• 70% evaluate by section (eg, introduction, methods)

• 70% adequacy of references

• 60% originality

• 60% adequacy of illustrations

• 50% relationship of conclusions to results

Overall the instructions emphasize economy of expression, ignoring the folk wisdom that “Sermons on brevity and chastity are about equally effective.”26 Nevertheless it is useful for prospective authors to obtain a specific checklist for the journal to which they are submitting so that they can attempt to meet the journal’s expectations.

The referees

The referees are unpaid volunteers; nevertheless they do receive some nonmonetary rewards. They get first access to new information in a field that interests them, and their decisions can influence the direction of that field. On occasion that information may be useful—for example, a submission could contain a reference of which the referee was unaware or a new technique that might be beneficial to the referee’s own research, or reading the article might prompt an idea for the referee’s future research. Finally, in doing a favor to the editor in refereeing a manuscript, the referee might acquire a store of goodwill that might help when his or her own manuscript is submitted to the journal. (Another well-accepted persuasive factor—reciprocation).28 Nevertheless, refereeing papers is a low-yield task—the referees’ efforts help the editor and those whose papers are published, but the referee typically gets no tangible benefit save the good feeling that comes from doing the right thing. Spending their own time on work for which others will benefit is bound to lead to resentment if those potentially benefitted make the task more difficult than it need be. The applicable golden rule then is to do unto the referees as you would have them do unto you. Make it easy for the referees in the hope they will make it easy for you. In this spirit then authors should attempt to meet the expectations of referees, in particular not wasting their time. In general, referees expect a scientific writing style characterized by the following qualities:

•Objectivity. Data obtained from scientific observation should be capable of being repeated by any competent observer, and the interpretations should be similarly identical among investigators. Expressed another way, in the “Storybook” version of scientific method, there is no room in science for subjective personal data collection and interpretation. Sometimes writers attempt to emphasize their objectivity, and this desire to appear objective can lead to overuse of the passive voice. Of course investigators do have an axe to grind, as they want to be published so that they can reap the rewards of publication—recognition and employment being the chief among these. So a tradition has arisen whereby authors attempt to appear to be objective while being strong advocates for their position. Thus, authors make “Verbal choices . . . that capitalize on a convenient myth . . . reason has subjugated the passions.”29 In any case, readers have come to expect that scientific writers will present at least a veneer of objectivity (practitioners of qualitative methods might disagree), but readers have other expectations of authors as well.

•Logic. Logic not only in organization but in sentence structure and meeting reader expectations.30

•Modesty. Related to the scientific norm of humility (extravagant claims will attract close and probably critical attention).

•Clarity. Scientific writers should follow the common advice to all writers such as avoiding misplaced modifiers, dangling participles, nonparallel constructions, stacked modifiers, etc. (There are numerous books on writing style, such as Strunk and White’s The Elements of Style31 or Zinnser’s On Writing Well.32)

•Precision. Use of precise terminology to avoid confusion and the fallacy of equivocation.

•Brevity. To conserve readers’ time.

•Justified reasoning. Making the reason for statements clear by referring to data in the paper (eg, “see Figure 1”) or references to the literature.

• Use of signposts (eg, “our first objective…,” “our second objective…”), linkage, etc.

Referees typically submit their reports by filling out forms online often accompanied by explanatory remarks in an uploaded text file.

Typically the form starts with what might be called high-level assessments—questions like accept or reject, priority, overall length of the paper. More detailed points are given in the comments to authors or the editor or editor’s assistant. The confidential comments to the editor give the referee the possibility to offer frank criticism that might be construed by some authors as being insulting. For example, a referee might comment that the paper is poorly written and needs revision by a native English speaker, and such a comment might be insulting to an author who was in fact a native English speaker.

Referees versus authors

Typically referees make critical comments on the papers they are reviewing ranging from the easily correctable, such as typographic errors or formatting, through more difficult problems to correct, such as lack of clarity in organization, deficiencies such as inappropriate methodology, or erroneous logic, that lead to unsupported conclusions. Typically the referees will number their comments, and the editor will require the author to address each of them. So in effect the authors and each of the referees enter into a debate presided over by the editor, who might also provide some comments, that might be classed according to the conventions of informal logic or pragmatics as a “persuasion dialogue.”33 The participants are obligated to give helpful and honest replies to their opponents’ questions. In theory each participant in the dialogue is supposed to use arguments exclusively composed of premises that are commitments of the other participant. But in argumentation, as in life, commitments are notoriously difficult to extract from an opponent, and pretty much the best one can hope for is plausible commitment to an opinion based on reasoned evidence. In conducting the argument the participants are also obligated with a burden of proof, which shifts from one to the other during the dialog. For example, in submitting the paper the author, as proponent, assumes the burden of proof for the conclusions of the paper, and the components of the paper (ie, methods, data, figures, tables, and logic), constitute the means of bearing that burden. Similarly the referee, in making a criticism, assumes the burden of proof of justifying the criticism. This may be done by various means such as citing deficiencies in the evidence in the paper, external scientific evidence (such as previously published papers), or expected standards in the field of study. The editor forwards the referees’ criticisms along with a preliminary decision to the authors who, if they want the submission to proceed to publication, are expected to bear the burden of proof in responding to the criticisms. This dialogue can be carried over several cycles. Often in my experience, it seems that referees seldom accept or commit to the author’s arguments; rather they merely concede by terminating discussion. In science, as in life, it is difficult to say “Sorry, I was wrong.” In some instances agreement between the referees and the authors is never achieved, but the issues are clarified to an extent that the editor can make a decision.

The question arises of the logic used by editors in making their decisions. First it should be noted that different types of reasoning employ different standards of proof, and this is not unusual in human affairs. In law for example the standard of proof in criminal cases is “beyond reasonable doubt” whereas civil cases are decided on the “balance of probabilities.” Scientific arguments can be complex and may entail various forms of logic ranging from the certainty of deductive logic employed in mathematics to inductive logic, which can deal with calculated probabilities, to informal logic that balances many factors but does not necessarily proceed by strict numeric calculation so that the conclusions are classed qualitatively in terms of their relative plausibility.

Perhaps the reasoning process most employed by editors, who have to make a practical decision, would be the pragmatic model devised by the philosopher Stephen Toulmin34 (see also chapter 22 for more on argumentation maps), which specifies a system for scientific explanation that includes Claims (such as conclusions in the paper) justified by Evidence and Warrants. A Warrant is the means that connects the Claim to the Evidence; it may be, for example, a scientific principle or a connection established by previous work. An important aspect of Toulmin’s approach is that it is field dependent so that appropriate standards are employed for differing types of scientific endeavor. One can see this aspect in action in the scientific literature by observing the content of papers where the rigor of the methods, the quantity of data, or the articulation of the findings differ among different fields of science or among the journals within one field of science. It is the editor who determines the standards of his/her journal, and differences between editors in what they consider important findings or flaws can result in a paper rejected by an editor of one journal being accepted by another one. There are other elements in the Toulmin model, including the Rebuttal arguments that restrict or counter the claim and the Qualifier, which indicates the degree of certainty that the proponent assigns to the Claim (eg unlikely, possibly, highly probable, or beyond any reasonable doubt) and this feature can hold the key to resolution of conflicts. Authors can back off or limit their claims to account for the views of the referees, and the editor can in good conscience publish the article.

The readers

The end users of the published paper, the readers, have been defined as anyone who reads the text “with an intentional search for meaning.”35 Editors and referees are knowledgeable about their fields and like the authors suffer from the problem of familiarity with the assumptions, conventions, and expectations of investigators in their field so that they tend to “fill in” what an author might leave out. General readers, however, differ from the editors and referees in that on average they are less familiar with the research field and may lack information required to understand the submission. Expressed another way, they can’t fill in what the author leaves out. As the readers vary widely in their expertise, it falls to the author to determine what they are likely to know (ie, what is common knowledge to everyone in the field), and conversely what the author needs to point out to them. Anything that is novel or unusual needs to be described in detail; for example, investigators may vary from the standard methods in their measurements or calculations, and such changes need to be highlighted and explained.

Editorial independence

Ideally, the contents of the journal should be independent of economic issues, but this is not necessarily the case. Publication of color illustrations can be prohibitively expensive, and many respected journals are publications of learned societies that operate on lean budgets. The Journal of Dental Research, for example, is published and subsidized by the International Association for Dental Research. Such a journal would be expected not to be subject to advertisers’ influence. Other journals have a need to generate income, and, in some instances, entire issues appear to be sponsored by a commercial interest. It is not unreasonable to wonder whether the advertiser influenced the editorial content, for “he who pays the piper calls the tune.” In recent years, Internet-based journals have arisen that are financed by authors through charges per page. As hard copy of the articles are not produced or distributed, costs are minimal, and the potential for profit is great. There is thus an incentive for such journals to have a very low (or no) rejection rate, and questionable quality may result.

Three general questions

A scientific paper is not necessarily an unbiased account of observations; it is more likely an attempt to convince the reader of the truth of a position. As noted by Ziman,36 it is important to realize that much of the research literature of science is intended, rhetorically, to persuade other scientists of the validity of received opinions. Thus, a reader can expect an author to present his or her data in the most favorable light. Tables, figures, and even calculations may be done so that differences between groups are accentuated and the appearance of error is minimized. A reader’s defense as a consumer of this information is an attitude of healthy skepticism. Three general questions a skeptical reader should ask are: Is it new? Is it true? Is it important?37

Is it new?

A minimum requirement for publication in most instances is that the information is new. However, new can be defined in various ways. If a paper using standard histologic techniques reporting the development of teeth in lynx were to be published tomorrow, it might well be new, because, as far as I am aware, the development of lynx teeth has not been described previously. However, it probably would not be new in adding anything to our knowledge of tooth development in general. Such a paper would merely fill in the gaps, however small, in our knowledge. I think that journal editors are fairly lenient in their judgments on what constitutes new information. Kuhn38 states that one of the reasons why normal puzzle-solving science seems to progress so rapidly is that its practitioners concentrate on problems that do not tax their own lack of ingenuity.

The quality that often distinguishes good scientific papers from the mediocre is originality. Funding agencies are probably better gatekeepers of science in this regard, because an essential criterion for funding is originality. Originality can appear in any component of the research process, including the questions being asked, the methods employed, the research design, or even the interpretation. Because science is a progressive business, approaches that were once original and sufficient can with time become derivative and deficient. Returning to the example, because scientists have been studying tooth development for decades using standard histologic techniques, there is not much hope that reworking the same approach would provide anything exciting; new methods would be required to bring new insights.

As a consequence of scientific progress, methods become outdated and standards change. Changing standards can be seen in biochemistry by examining the standards for publication of data using polyacrylamide gels. Early publications using the technique showed photographs of gels that did not have good resolution or uniformity and showed poor staining. The photographs of gels were often so uninformative that Archives of Oral Biology instructed authors to submit densitometric tracings of the gels. Currently, gel separations are done in two dimensions with superb resolution, and the proteins are stained with much greater sensitivity. A photograph of a gel that would have been acceptable 30 years ago would not be acceptable for publication today. In judging papers, therefore, a key question is whether the techniques and approach are up-to-date as well as whether the question is original.

This principle is so well accepted that investigators sometimes rush to apply new techniques to appear up-to-date. Fisher,39 the pioneer statistician and author of a classic work on experimental design, warned, “any brilliant achievement . . . may give prestige to the method employed, or to some part of it, even in applications to which it has no special appropriateness.”

An exception to the requirement of “newness” for a publication is the need to report confirmations of previous work. One journal for which I refereed placed the category “valuable confirmation of previous work” in third place in their ranking system below exciting original research and interesting new findings, but above categories related to rejection. This type of research is taking on increasing importance in the light of the “reproducibility crisis” to be discussed later.

Is it true?

Sound conclusions are the result of reliable observations combined with valid logic. Knowledge of measurement, types of observational errors, experimental design, and controls give some basis to assessments of the reliability of observations. Thus, sections of this book deal with these topics and the logic used to interpret the observations. But the ultimate test of any scientific observation is reproducibility; indeed, a practical definition of truth for the purposes of pragmatic working scientists is that a scientific statement is true if it allows us to make useful, reliable predictions that are confirmed when tested by a competent scientist under the specified conditions. There are theoretical or practical limitations to any approach. Newton’s laws of motion would be perfectly valid when applied to billiard balls colliding on a pool table but not useful at very small scales of the world of subatomic particles where quantum physics would be preferred. Note that confirmation does not imply the exact same numeric result, but rather one that is within the specified interval of reported uncertainty.

A clue to the reproducibility of an observation is the consistency of the results within the body of the paper. Another means for evaluating the reliability of observations in a paper is to read what other scientists have written about the work, and citation analysis is an efficient means of uncovering that information. For various reasons, to be discussed later, there is a current “reproducibility crisis” perceived in which confidence in the reproducibility of findings, even those published in high-impact journals, is waning.

A student might wonder whether it is necessary to learn such diverse concepts and examine the literature to such a detailed extent, particularly when it seems likely that the vast majority of publications are produced in good faith and come from institutions of higher learning. Ioannidis,40 however, has argued that most published research findings are false. Ioannidis’ estimate is sensitive to the pretest probability of the hypothesis being true, and a low estimate of this value will lead to a higher proportion of papers’ conclusions being classed as false. Nevertheless, as will be discussed later, current research into reproducibility of findings has provided more direct evidence indicating a significant proportion of findings are false in that they cannot be reproduced. In Ioannidis’ common sense view, a research finding is less likely to be true when effect sizes are small, when there are a large number of tested hypotheses that have not been preselected, and when there are great flexibilities in designs, definitions, outcomes, and data analyses. Other problems impacting the truth of the conclusion are financial issues and other interests and prejudices as well as the number of teams in a field chasing statistical significance. I believe it is unlikely that most research findings are false, because if they were there would be more papers reporting failure to confirm results (though admittedly publishing such negative results can be difficult) and many fewer papers confirming—albeit often indirectly—replication. Nevertheless, the considerations listed by Ioannidis serve to warn readers of the dental and medical literature that there is no shortage of well-documented threats to truth.

Is it important?

The importance of a paper cannot be tested in a completely objective manner. Maxwell41 has argued—in my opinion, persuasively—that real progress in science is assessed in terms of the amount of valuable factual truth that is being discovered and that the accumulation of vast amounts of trivia (even if factually correct) does not amount to progress. The problem is that value judgments are highly subjective. One approach to measuring impact of a paper is the number of citations to the paper, an aspect that will be discussed in chapters 5 and 22. Many scientists have accepted this criterion and include the citation record of their papers in their curriculum vitae or include indices derived from their citation record such as the h-index (discussed in chapter 5).

One can speculate about what qualities an ideal evaluator should have. Beveridge42 has suggested the concept of scientific taste, which he described as a sense of beauty or esthetic sensibility. Beveridge explained scientific taste by stating that:

The person who possesses the flair for choosing profitable lines of investigation is able to see further where the work is leading than are other people because he has the habit of using his imagination to look far ahead, instead of restricting his thinking to established knowledge and the immediate problem.

A person with scientific taste would be a good judge of the importance of a scientific paper. Traditionally, the skill of judgment is learned in the apprentice- master relationship formed between graduate student and supervisor. Techniques may come and go, but judging what is important and how it can be innovatively studied are the core business of scientists, and these skills are learned similarly to a child learning his prayers at his mother’s knee: Graduate students hone their critical skills in the supervisor’s office or lab meetings. Thus, much importance is attached to the pedigree of a scientist, and some scientists take pride in tracing their scientific pedigrees to leading figures in a field of study.

Given the large variation in laboratory and supervisor quality, there will always be significant differences in judgment. This diversity is evident in an extensive study of proposals submitted to the National Science Foundation. The study found that getting a research grant significantly depends on chance, because there is substantial disagreement among eligible reviewers, and the success of a proposal rests on which reviewers happen to be accepted.43 Moreover, there is evidence that complete disagreement between pairs of referees assessing the same paper is common.43 In biomedical science, the frequency of agreement between referees was not much better than that which would be expected by chance.44 Hence, it appears that objective and absolute criteria for the evaluation of a paper prior to publication are not available. Chapter 22 attempts to cultivate the skill of judgment by providing information on recognized sources of errors in judgments as well as citation analysis, a technique that can be used to access broadly based scientific assessments of published works.

References

1. Chargaff E. Triviality in science: A brief meditation on fashions. Perspect Biol Med 1976;19:324.

2. Norris SP. Synthesis of research on critical thinking. Educ Leadersh 1985;42:40–45.

3. Kurland DJ. I Know What It Says—What Does It Mean? Critical Skills for Critical Reading. Belmont, CA: Wadsworth, 1995:164.

4. Butler CB. New ways of seeing the world. In: Butler CB (ed). Postmodernism: A Very Short Introduction. New York: Oxford University, 2002:37–42.

5. Bronowski J. The common sense of science. In: Bronowski J (ed). The Common Sense of Science. New York: Vintage, 1967:97–118.

6. Chambers DW. Lessons from badly behaved technology transfer to a professional context. Int J Technol Transfer Commercialisation 2005;4:63.

7. Chambers DW. Changing dental disease patterns. Contact Point 1985;63:1–17.

8. Fish W. Framing and testing hypotheses. In: Cohen B, Kramer IRH (eds). Scientific Foundations of Dentistry. London: William Heinemann, 1976:669.

9. Perkins DN, Allen R, Hafner J. Difficulties in everyday reasoning. In: Maxwell W (ed). Thinking: The Expanding Frontier. Philadelphia: Franklin Institute, 1983:177.

10. Zipf GK. Human Behaviour and the Principle of Least Effort. Cambridge: Addison-Wesley, 1949.

11. Pratkaris AR, Aronson E. Age of Propaganda: The Everyday Use and Abuse of Persuasion. New York: WH Freeman, 2001:38.

12. Kahneman D. Attention and effort. In: Kahneman D (ed). Thinking, Fast and Slow. Toronto: Anchor Canada, 2013:31–38.

13. Risen J, Gilovich T. Informal logical fallacies. In: Sternberg RJ, Roediger HL, Halpern DF (eds). Critical Thinking in Psychology. Cambridge: Cambridge University, 2007:110–130.

14. Garfield E. Current comments: Is the ratio between number of citations and publications cited a true constant? Curr Contents 1976;6:5–7.

15. Glenny A, Hooper L. Why are systematic reviews useful? In: Clarkson J, Harrison JE, Ismail A (eds). Evidence Based Dentistry for Effective Practice. London: Martin Dunitz, 2003:59.

16. Garfield E. The literature of dental science vs the literature used by dental researchers. In: Garfield E (eds). Essays of an Information Scientist. Philadelphia: ISI, 1982:373.

17. Relman AS. Journals. In: Warren KS (ed). Coping with the Biomedical Literature. New York: Praeger, 1981:67.

18. Worthington H, Clarkson J. Systematic reviews in dentistry: The role of the Cochrane oral health group. In: Clarkson J, Harrison JE, Ismail A (eds). Evidence Based Dentistry for Effective Practice. London: Martin Dunitz, 2003:97.

19. Chambers DW. Habits of the reflective practitioner. Contact Point 1999;79:8–10.

20. Day RA. How to Write and Publish a Scientific Paper. Philadelphia: ISI, 1979:2.

21. DeBakey L. The Scientific Journal. Editorial Policies and Practices: Guidelines for Editors, Reviewers, and Authors. St Louis: Mosby, 1976:1–3.

22. Simonton DK. Creativity in Science: Chance, Logic, Genius, and Zeitgeist. Cambridge: Cambridge University, 2004:85–86.

23. Goodman SN, Berlin J, Fetcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Ann Intern Med 1994;121(1):11.

24. Giannobile WV. Editor’s Report for the Journal of Dental Research–2015. http://dentalresearchblog.org/jdr/?p=338. Accessed 18 July 2018.

25. Tacker MM. Parts of the research report: The title. Int J Prosthodont 1990;3:396–397.

26. Chernin E. Producing biomedical information: First, do no harm. In: Warren KS (ed). Coping with the Biomedical Literature. New York: Praeger, 1981:40–65.

27. Day RA. How to Write and Publish a Scientifc Paper. Philadelphia: ISI, 1979:72.

28. Cialdini RB. The science of persuasion. Sci Am 2001;284:76 –81.

29. Gross AG. The Rhetoric of Science. Cambridge: Harvard University, 1990.

30. Gopen G, Swan J. The science of scientific writing. Am Sci 1990;78:550–558.

31. Strunk W, White EB The Elements of Style, ed 4. Boston: Allyn and Bacon, 1999.

32. Zinnser W. On Writing Well, ed 2. New York: Harper and Row, 1980.

33. Walton D. Argument as reasoned dialogue. In: Walton D (ed). Informal Logic: A Pragmatic Approach, ed 2. New York: Cambridge University, 2008:1–34.

34. Toulmin SE,The Uses of Argument, updated ed. Cambridge: Cambridge University, 2003.

35. Lang TA. How to Write Publish and Present in the Health Sciences: A Guide for Clinicians and Laboratory Researchers. Philadelphia: American College of Physicians, 2010:41.

36. Ziman J. Reliable Knowledge: An Exploration of the Grounds for Belief in Science. Cambridge: Cambridge University, 1978:7.

37. DeBakey L. The Scientific Journal: Editorial Policies and Practices: Guidelines for Editors, Reviewers, and Authors. St Louis: Mosby, 1976:1.

38. Kuhn TS. The Structure of Scientific Revolutions, ed 2. Chicago: Chicago University, 1970:184.

39. Fisher RA. The Design of Experiments, ed 8. Edinburgh: Oliver & Boyd, 1953:184.

40. Ioannidis JP. Why most published research findings are false. PLoS Med 2005;2:e124.

41. Maxwell N. Articulating the aims of science. Nature 1977;265:2.

42. Beveridge WI. The Art of Scientific Investigation. New York: Vintage, 1950:106.

43. Cole S, Cole JR, Simon GA. Chance and consensus in peer review. Science 1981;214:881.

44. Gordon M. Evaluating the evaluators. New Sci 1977;73:342.