WHY SCIENCE IS BROKEN AND
HOW WE CAN HELP TO FIX IT
Written by Jens Klinzing
I love science. I really love science (and particularly neuroscience), humanity’s great cultural
endeavor to understand the world and ourselves. Unfortunately, science is broken. It is broken
in many ways, not only in the way it distributes funding, its bizarre publishing scheme, or the
way scientists are employed. It is also fundamentally broken in the way we obtain and evaluate
our findings.
Most published scientifc literature is
false
News traveled quickly when the results of a large replication initiative
were published last year in the journal
Science1. The Open Science Collaboration, a team of 270 authors, had tried
to replicate 100 studies from four
important psychology journals. The
replication studies were conducted as
similar to the original studies as possible, often in close cooperation with
the original authors. The results were
alarming. In only 39 of the 100 replications could the original finding be
successfully reproduced*. Considering
the average statistical power of the
replications, one would have expected
around 89 successful reproductions if
all the original effects were real. And it
16 | NEUROMAG | July 2016
gets worse. The replication outcomes
did not even correlate well with the
statistical significance of the original
studies. The most significant effects,
having very low p-values, did not have
much higher chances to be reproduced
than moderately significant effects.
In my opinion, all this teaches us two
things:
P-values are a bad indicator for how
likely an effect is to be reproduced.
Next time you read a publication and
there are three impressive asterisks
(***) over a bar graph2, remember,
this does not mean the effect is much
more reliable than if there was only
one asterisk (*). Why this is the case
can be seen in this video (dance of the
p-values) in quite an entertaining way.
We have way too many false positives in science. Low reproducibility
is certainly not a problem exclusive to
psychology. There are similar findings
for neuroscience3, genetics4, epidemiology5, etc. It is a general problem
of science, particularly when samples
(and thus statistical power) are small,
effects are small, and variance is high.
We have good reason to believe that
more than half of all peer-reviewed
scientific literature is false6. More than
half of all the papers you read. More
than half of all posters you visit, talks
you listen to, and more than half of all
studies you are going to publish yourself.
But where do all the false positives
come from?