Empowering the next generation of open scientists

AUG 8, 2017

 

Karthik Ram

How to succeed in reproducible research without really trying

AUG 8, 2017

 

Reproducible research is hard but you should do it anyway

AUG 8, 2017

 

The

reproducibility crisis

is widespread

Baker, 2015. Baker & Dolgin 2017, Aschwanden, C. 2016, Casadevall & Fang 2010

P values are just the tip of the iceberg

Peng & Leek, 28 Apr 15

Medicine

▪ study power and bias
▪ ratio of true to no relationship among all relationships probed

 

Medicine

▪ study power and bias
▪ ratio of true to no relationship among all relationships probed

▪ p-hacking

 

Psychology

Medicine

▪ study power and bias
▪ ratio of true to no relationship among all relationships probed

▪ p-hacking

Lack of access to full datasets and protocols

Psychology

Biomed

This is quite widespred even in applied computational research

Collberg et al 2014​

Computational science has culture problems

Computational science has culture problems

no verification,

no transparency 

no efficiency

Lack of verification

Where's the bug?

In the method, the implementation, somewhere in the dependencies?

 ¯\_(ツ)_/¯

Lack of transparency

Impossible to debug without
public code or data.

Do your results only show "ideal" cases?

Efficiency

It's really difficult and wasteful to reimplement from scratch.

Without record keeping, it's very hard to collaborate, revise, or rebuild.

4 kinds of reproducibility

Computational reproducibility and transparency

Scientific reproducibility & transparency

Computational correctness

Statistical reproducibility

Millman et al , 2016

The extent to which code would actually build with reasonable effort is quite low

Collberg et al 2014​

< 20%

Verification

Transparency

Efficiency

Easier to find and fix bugs

> citation counts, broader impact

Fewer people have to rebuild your work

Why reproducibility matters

" The actual scholarship is in the full software environment, code, & data that produce the result"

David Donoho

We have the technology to fix these problems

Two primary issues: tooling and cultural change

practice
reproducible
research.org

1

Leveling up skills

Biggest bottleneck to adoption of reproducible research practices was related to diversity of skills or lack thereof

More homogeneity in tool familiarity = better reproducibility

2

Dependencies, build systems, and packaging

Scientific software often built on numerous dependencies

We need improved build systems for software, data, and

workflows

3

Testing

Code that went beyond simple script reported testing systematically

However, many scientists were discouraged by the perceived effort of unit testing

4

Publishing

There is still a need for publication formats that allow for effortless collaboration.

5

Data sharing & versioning

Versioning data is hard,
as is finding a reliable
place to archive them

 

 

6

Time and incentives

"time and efforts spent on creating reproducible research are not very well rewarded"

Ram & Marwick, 2016

Reality is that software stacks are complex and record keeping is really, really tedious

How to overcome these challenges

Training in data skills is one of the largest unmet needs

Barone et al, 2017

Data Carpentry trains researchers in the core data skills for efficient, shareable, and reproducible research practices.

import
tidy
Grolemund and Wickham 2017
transform
visualize
model
communicate
import
tidy
Grolemund and Wickham 2017
transform
visualize
model
communicate

Software

100+ software packages to support data science. e.g. spatial data, biodiversity informatics & climate change, glue for workflows.

Community

Developing a community of research software engineers, and the next generation of data science mentors.

unconf17.ropensci.org

Culture change

Bringing the best of academic peer-review to research software

 

Incentivize scientists who engage in reproducible research (50k fellowships)

Glue software

Data retrieval (APIs, data storage services, journals)

Data visualization (e.g. plot.ly)

Data sharing (figshare, Zenodo, dat)

Reproducibility

plot.ly

Computing resources of R + interactivity of JavaScript

tidy text

Making text analysis easier and reproducible

CI + deployment

Preinstalled steps
step_hello_world: 
step_run_covr: 
step_install_ssh_key: make available a private SSH key (which has been added before to your project by travis::use_travis_deploy())
step_test_ssh
step_build_pkgdown

Ocean health index

import
tidy
Grolemund and Wickham 2017
transform
visualize
model
communicate

Software

peer-review

“A sinking feeling

  in my gut”

   — Diary of a retraction

Recently, Dr. Tony Wilson from CUNY Brooklyn tried to recreate my analysis, so that he could figure out how it worked and apply it to his own data ... he couldn’t quite recreate some of my core results.

I dug up my original code, sent it to him, and after a couple of back-and-forth emails we found my error.

I immediately sent a retraction email to the journal (Evolutionary Ecology Research), which will be appearing soon in the print version. So let me say this clearly, I was wrong.

I suspect that I am not the only biologist out there to make a small mistake in my code that has a big impact.

So: how many results, negative or positive, that enter the published literature are tainted by a coding mistake as mine was. We just don’t know. Which raises an important question: why don’t we review code (or other custom software) as part of the peer-review process?

of researcher-contributed packages have unit tests

Keyes et al 2015

19%

onboarding

OSI compatible license

Detailed README, complete docs

High test coverage

Readable code

Usability

Pre-submission inquiry

Fit based on our criteria

🤔

Peer-review 😒

evaluate the package for usability, quality, and style based on our guidelines

Acceptance 💯

Packages are badged and added to our system

Open & non-adverserial

No rejections

Makes the process constructive for everyone involved

“I don’t really see myself writing another serious package without having it go through code review.”

The ReScience journal

Replication is the scholarship

The ReScience journal

Original

Article

Successful replication

Unsuccessful replication

Journal of Open Source Software

joss.theoj.org

A mechanism for research software developers to get credit within the current merit system of science

Submission only require a Github repository URL, an ORCID, and a succinct high-level description of the software

joss.theoj.org
10.7554/eLife.16800.001

Partial reproducibility is
better than nothing

C. Titus Brown

"What we need right now is scientists actually using stuff that already exists, not engineers building new stuff that no one will ever use."

"Start small provide raw data, post any scripts, and versions of programs you used."

Karl Broman

Version your code 

Practices you can tell researchers to adopt now

Open your data

Automate everywhere

Document your processes 

Test everything

Avoid excessive dependencies
DOIs everywhere
Build workflow & provenance frameworks around familiar tools

Practices you can tell researchers to adopt now

inundata.org/talks/stsci