firmatory. That is a big problem and
one of the major precedents causing
the replication crisis and the growing
public mistrust of science. How can we
ensure that we clearly distinguish ex-
ploratory and confirmatory analyses,
both for ourselves and others?
Report exploratory research honestly
If you know you are doing exploratory
research, then report it as such! There
should be big red flags in your discus-
sion saying things like “these findings
are exploratory; a confirmatory study
is needed before firm conclusions can
be drawn”. There is nothing wrong with
exploratory research – it must con-
tinue to be published – but we need
more confirmatory experiments in the
literature (particularly for surprising or
weak effects). And in my opinion, the
media should be barred from report-
ing on anything in the experimental /
biological sciences until it has been in-
dependently replicated.
But as I discussed above, it can be
all too easy to fool yourself (“oh, of
course I was going to normalise that
way all along”). How can you ensure
you are doing what you said you would
do before you saw the data?
Pre-registration
Pre-registering an experiment means
writing down your experimental hy-
potheses, data collection plan, experi-
mental design, outcome measures,
data preprocessing, exclusion criteria,
analyses, and the final test of your hy-
pothesis all before having begun data
collection. You consider everything you
can think of (greatly helped by run-
ning a pilot experiment) as explicitly as
you can. Having analysis scripts pre-
pared ahead of time would be the gold
standard. Once you have collected the
data you perform the analysis and re-
port exactly that result.
As someone who has been doing ex-
periments for years without pre-reg-
istration, I can tell you that it is hard.
I am used to having some theoretical
hypothesis and concrete experimen-
tal idea, then rushing in and testing
it, then sorting out the details later (in
the same dataset). Pre-registration
means trying to specify lots of things
that I would have sorted out on the
way. Nevertheless, it is a rewarding
way to guard yourself against the gar-
den of forking paths.
What happens when (not if) you realise
you really should do that normalisa-
tion or change the analysis, after you
have collected the data? That is fine:
just report it as such (along with the
results of the original analysis). Pre-
registration is a mechanism to help
you be honest about what is confirm-
atory and what is exploratory. Readers
of your paper can decide how much
the deviations from what you had said
you would do matter.
So how do you actually pre-register
an experiment? This can be as simple
as a text document on your comput-
er. However, an even better idea is to
pre-register the experiment online in a
time stamped repository (e.g. Github or
the Open Science Framework). Then,
when you write up the study, you can
point readers to a link with your pre-
registration as evidence that you are
really trying to delineate exploratory
and confirmatory analyses. For this
purpose I really like aspredicted.org.
This site has a 9-question template
for pre-registration, which encourages
a short and structured document. At
a later time, you can choose to make
your preregistration pdf publicly avail-
able with a static URL link, which could
be included in the methods of your pa-
per.
Registered reports
At the next level in scientific transpar-
ency are registered reports. A scien-
tific publication in a registered report
format is one in which (a) the authors
submit a manuscript consisting of an
introduction, methods, and an analysis
plan; (b) reviewers critique the meth-
ods and planned analyses, suggesting
changes; (c) if the authors and review-
ers agree on a protocol, then the paper
is in-principle accepted at the journal;
(d) the authors go collect and analyse
the data; (e) the journal publishes the
results. There are now over 20 jour-
nals that offer this format, including
Cortex, Attention Perception and Psycho-
physics, and Nature Human Behavior [7].
In my opinion, this format is excel-
lent for guarding against researcher
degrees of freedom and is particu-
larly suited for either direct replica-
tion studies or for studies whose out-
come you expect will be contentious.
Registered reports could allow you to
sidestep fighting with reviewers who
ultimately simply do not like your con-
clusions. After all, if they agreed on
your experimental protocol, then data
could presumably have gone their
way. The data could have confirmed
their theory.
However, the time-scale required for
registered reports make them unsuit-
able for use in something like a mas-
ter’s project or lab rotation. By the
time you have received comments
from reviewers and adjusted your
protocols, your time in the lab is likely
up. Self-pre-registration on the other
hand will also improve your ability to
discriminate exploratory and confirm-
atory research and will typically take
you only a few hours.
Open data and materials
Scientists should want to make it as
easy as possible for others to inde-
pendently check our results and at-
tempt replic ations. To facilitate these
outcomes and to promote transpar-
ency in the research process more
generally, it is becoming increasingly
standard to make raw data and study
materials (such as stimuli or analysis
code) publicly available at the time of
publication. This not only helps others
to verify your work, but can also help
you. Proper archiving of study mate-
rial is a requirement of most research
funding and this way your materials
and data are archived as part of the
publication process.
There are a number of ways to share
data and materials online. My favour-
ite resource for this is zenodo.org. It is
a freely available EC-funded research-
sharing framework capable of hosting
tens of gigabytes of data (terabytes
available upon request), and it is host-
ed on the same infrastructure as the
LHC data from CERN. Every upload
gets a DOI (digital object identifier),
meaning you can cite your materials in
the text of your paper with a reference
that is never lost (no more self- or
publisher-hosted content going miss-
ing after a few years).
What if you expect to get several pa-
pers from the same dataset? Justifi-
May 2017 | NEUROMAG |
17