#Believe it or not but Science is in a reproducibility crisis
#Believe it or not, but Science is in a reproducibility crisis
%% Cell type:markdown id: tags:
### Irreproducibility in biological sciences is largely criticized all over the web
### Irreproducibility in biomedical research is largely criticized all over the web
%% Cell type:markdown id: tags:
### And in academic reviews
### ... and in academic reviews
*[Ioannidis, JPA (2005). *Why Most Published Research Findings Are False*. PLoS Med](http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124)
*[Florian Prinz et al (2011), *How much can we rely on published data on potential drug targets?* Nat. Rev. Drug Dis.](http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.html)
*[Landis et al. (2012). *A call for transparent reporting to optimize the predictive value of preclinical research.* Nature](http://www.nature.com/nature/journal/v490/n7419/full/nature11556.html)
...
%% Cell type:markdown id: tags:
###And Nature group created the open acces collection
### ... and Nature group created the open acces collection
### "[Challenges in irreproducible research](http://www.nature.com.gate2.inist.fr/nature/focus/reproducibility/index.html)"
#### "[Challenges in irreproducible research](http://www.nature.com.gate2.inist.fr/nature/focus/reproducibility/index.html)"
%% Cell type:markdown id: tags:
#What are the consequences on industrial productivity?
%% Cell type:markdown id: tags:
### Drug companies fail to reproduce a large proportion of academic discoveries
**Amgen : biotech company in cancer research**
* Try to reproduce the findings of 53 “landmark” articles in cancer research
* Only 6 of the 53 studies were reproduced (about 10%)
**Bayer, the pharmaceutical company**
**Bayer : Big-Pharma company**
* Examined 67 target-validation projects in oncology, women’s health, and cardiovascular medicine
* Published results were reproduced in only 14 out of 67 projects (about 21%).
%% Cell type:markdown id: tags:
George A. Robertson (Ex Senior Scientist at Merck)
*"It drives people in industry crazy. Why are we seeing a collapse of the pharma and biotech industries? One possibility is that academia is not providing accurate findings"*
%% Cell type:markdown id: tags:
### High failure rate of clinical trial
[Arrowsmith, J (2011). *Trial watch: Phase II failures: 2008–2010*. Nature Reviews Drug Discovery 10: 328–329.](http://www.nature.com/nrd/journal/v10/n5/full/nrd3439.html)
108 failures of clinical trials divided according to cause and therapeutic area.
###Scientific fraud is real issue but it concerns only an small proportion of irreproductible studies
%% Cell type:markdown id: tags:
### Unrelated data panels / missing references / incorrect controls
* Comparing mdx mice with C57/B6J
*Example: Comparing a murine model vs an "healthy" control with a different genetic background
%% Cell type:markdown id: tags:
### Pseudo replicates
* Failure to understand the difference linked and independent samples
* Failure to understand the difference between linked and independent samples
%% Cell type:markdown id: tags:
### Data handling
* Interim data analysis and ad hoc exclusion of data
%% Cell type:markdown id: tags:
### Misusage and Over-interpretation of statistical tests
Nuzzo, R (2014). *Scientific method: Statistical errors* Nature
Halsey, LG, et al (2015). *The fickle P value generates irreproducible results* Nat. Meth.
%% Cell type:markdown id: tags:
### Underpowered studies = Small effect sizes
**When unlikely hypotheses are tested, most positive results of underpowered studies can be wrong**
Button, KS, et al (2013). *Power failure: why small sample size undermines the reliability of neuroscience*. Nat. Rev. Neuro.
Krzywinski, M & Altman, N (2013). *Points of significance: Power and sample size*. Nat. Meth.
%% Cell type:markdown id: tags:
### Inaccurate and incomplete reporting of methods
* Negative results are virtually never reported
* Impossibility to reproduce results due to missing details or errors
* Chain of references back to 1952...
%% Cell type:markdown id: tags:
#How do we fix it?
%% Cell type:markdown id: tags:
### Formal training in statistics and study design
**Core training for PhD students postdoc and principal investigators**
### Formal training in statistics and study design for PhD students, postdocs and principal investigators
* Basic statistics and graphic representation for life sciences
* Experimental methods and experimental design
* Data handling and analysis
* Basic statistics applied to life sciences
* Experimental methods design and controls
* Whole study design
* Data handling, analysis and representation
* Understand the issues of emerging technologies
* Compulsory refresher courses
%% Cell type:markdown id: tags:
*"In essence, this is the right thing to do, if only it was not for me to do it"*
%% Cell type:markdown id: tags:
**Or hire a biostatistician/methodologist to design YOUR studies and analyse YOUR data**
%% Cell type:markdown id: tags:
### Basic study design, in the twenty-first century
* Study have to be designed **before starting**new projects
* Define **exclusion criteria**for experimments and to outlyer
*Requierement that **subjective end points**are assessed by blinded investigators
* Inclusion of **appropriate controls**
* Studies have to be designed **before starting** projects (QA documentation?)
* Define **criteria**to validate experiments and remove outlyers thanks to previous works or pilot studies
***Subjective end points**must be assessed blindly
* Inclusion of **appropriate control(s)**
* Use **highly relevant** animal models in biomedical research
%% Cell type:markdown id: tags:
### How to deal with small sample ( 5 to 30 animals per group)
# EXAMPLES !!
* Randomization of animals
%% Cell type:markdown id: tags:
* Prefer biological replicates over technical
* Mix together what is mixeable
### How to deal with small samples (5 to 30 animals per group)
* Real **randomization** of animals or experimental conditions
* For advanced preclinical studies
* Reduce the number of experiental conditions
***Biological replicates** instead of technical replicates
* Mix together what is mixeable
* Be more stringent in statistical tests
* Use non parametric analyses unless you have a very good reason to use parametric tests
%% Cell type:markdown id: tags:
***Reduce the number of experiental conditions** for advanced preclinical studies
# EXAMPLES !!
* Use **stringent statistical tests** and **adapt your graphical representation**
* Non-parametric tests, unless you have a very good reason to use parametric tests
%% Cell type:markdown id: tags:
#Large scale efforts and future directions
%% Cell type:markdown id: tags:
### Standardization
**Participate and follow international community guidelines**
**Actively looks for international community guidelines**
* Quantitative PCR experiments => MIQE
* Digital Quantitative PCR => dMIQE
* Flow cytometry => MIFlowCyt
* T-cell assays => MIATA
*genome sequencing => MIGS
*Genome sequencing => MIGS
**Use international reference material to normalyze data when available**
**Use international reference material when available to normalyze data**
* Ayuso, E et al. (2014). *Manufacturing and Characterization of a rAAV Type 8 Reference Standard Material*. Hum. Gene Ther
%% Cell type:markdown id: tags:
### Replication
Replicates findings by other methods and external teams (for high impact journals)
**Replicate** dogma-killer findings by other methods and external teams (for high impact journals)
%% Cell type:markdown id: tags:
### Lab Robotics & Automation
Reduce experimenter variability and contaminations during preparation of samples (in conjunction with standardization)
Check Hayden, E (2014). *The automated lab.* Nature 516: 131–132.
%% Cell type:markdown id: tags:
### Open and extensive reporting of materials & methods
### Open and extensive reporting
* Report extensively **materials & methods**
* Supplementary data
* Raw data used for graphical representation
* Large datasets uploaded to public repositories
the world's first peer reviewed scientific video journal : the Journal of Visualized Experiments http://www.jove.com/
* Report **more and differently**
* Very short papers format
* Video Journal (http://www.jove.com/)
* Negative resuts specific journals
%% Cell type:markdown id: tags:
the world's first peer reviewed scientific video journal : the Journal of Visualized Experiments
posting datasets to online repositories like Dryad
Publish Negative result : Journal of Negative Results in Biomedicine, the All Results Journals (with subject-specific versions for chemistry, biology, nanotechnology and physics), and the Journal of Negative Results
%% Cell type:markdown id: tags:
**Journals unite for reproducibility**
### Journals unite for reproducibility
%% Cell type:markdown id: tags:
**Principles and Guidelines for Reporting Preclinical Research**
#Believe it or not but Science is in a reproducibility crisis
#Believe it or not, but Science is in a reproducibility crisis
%% Cell type:markdown id: tags:
### Irreproducibility in biological sciences is largely criticized all over the web
### Irreproducibility in biomedical research is largely criticized all over the web
%% Cell type:markdown id: tags:
### And in academic reviews
### ... and in academic reviews
*[Ioannidis, JPA (2005). *Why Most Published Research Findings Are False*. PLoS Med](http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124)
*[Florian Prinz et al (2011), *How much can we rely on published data on potential drug targets?* Nat. Rev. Drug Dis.](http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.html)
*[Landis et al. (2012). *A call for transparent reporting to optimize the predictive value of preclinical research.* Nature](http://www.nature.com/nature/journal/v490/n7419/full/nature11556.html)
...
%% Cell type:markdown id: tags:
###And Nature group created the open acces collection
### ... and Nature group created the open acces collection
### "[Challenges in irreproducible research](http://www.nature.com.gate2.inist.fr/nature/focus/reproducibility/index.html)"
#### "[Challenges in irreproducible research](http://www.nature.com.gate2.inist.fr/nature/focus/reproducibility/index.html)"
%% Cell type:markdown id: tags:
#What are the consequences on industrial productivity?
%% Cell type:markdown id: tags:
### Drug companies fail to reproduce a large proportion of academic discoveries
**Amgen : biotech company in cancer research**
* Try to reproduce the findings of 53 “landmark” articles in cancer research
* Only 6 of the 53 studies were reproduced (about 10%)
**Bayer, the pharmaceutical company**
**Bayer : Big-Pharma company**
* Examined 67 target-validation projects in oncology, women’s health, and cardiovascular medicine
* Published results were reproduced in only 14 out of 67 projects (about 21%).
%% Cell type:markdown id: tags:
George A. Robertson (Ex Senior Scientist at Merck)
*"It drives people in industry crazy. Why are we seeing a collapse of the pharma and biotech industries? One possibility is that academia is not providing accurate findings"*
%% Cell type:markdown id: tags:
### High failure rate of clinical trial
[Arrowsmith, J (2011). *Trial watch: Phase II failures: 2008–2010*. Nature Reviews Drug Discovery 10: 328–329.](http://www.nature.com/nrd/journal/v10/n5/full/nrd3439.html)
108 failures of clinical trials divided according to cause and therapeutic area.
