"#Believe it or not but Science is in a reproducibility crisis"
"#Believe it or not, but Science is in a reproducibility crisis"
]
},
{
...
...
@@ -113,7 +115,7 @@
}
},
"source": [
"### Irreproducibility in biological sciences is largely criticized all over the web\n",
"### Irreproducibility in biomedical research is largely criticized all over the web\n",
"\n",
""
]
...
...
@@ -126,7 +128,7 @@
}
},
"source": [
"### And in academic reviews \n",
"### ... and in academic reviews \n",
"\n",
"* [Ioannidis, JPA (2005). *Why Most Published Research Findings Are False*. PLoS Med](http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124)\n",
"\n",
...
...
@@ -147,9 +149,10 @@
}
},
"source": [
"###And Nature group created the open acces collection\n",
"### ... and Nature group created the open acces collection\n",
"\n",
"### \"[Challenges in irreproducible research](http://www.nature.com.gate2.inist.fr/nature/focus/reproducibility/index.html)\"\n",
"\n",
"#### \"[Challenges in irreproducible research](http://www.nature.com.gate2.inist.fr/nature/focus/reproducibility/index.html)\"\n",
"\n",
"\n"
]
...
...
@@ -178,11 +181,13 @@
"**Amgen : biotech company in cancer research**\n",
"\n",
"* Try to reproduce the findings of 53 “landmark” articles in cancer research\n",
"\n",
"* Only 6 of the 53 studies were reproduced (about 10%)\n",
"\n",
"**Bayer, the pharmaceutical company**\n",
"**Bayer : Big-Pharma company**\n",
"\n",
"* Examined 67 target-validation projects in oncology, women’s health, and cardiovascular medicine\n",
"\n",
"* Published results were reproduced in only 14 out of 67 projects (about 21%)."
]
},
...
...
@@ -213,9 +218,7 @@
"\n",
"[Arrowsmith, J (2011). *Trial watch: Phase II failures: 2008–2010*. Nature Reviews Drug Discovery 10: 328–329.](http://www.nature.com/nrd/journal/v10/n5/full/nrd3439.html)\n",
"\n",
"108 failures of clinical trials divided according to cause and therapeutic area.\n",
"### Unrelated data panels / missing references / incorrect controls\n",
"\n",
"* Comparing mdx mice with C57/B6J"
"* Example: Comparing a murine model vs an \"healthy\" control with a different genetic background "
]
},
{
...
...
@@ -279,7 +282,7 @@
},
"source": [
"### Pseudo replicates\n",
"* Failure to understand the difference linked and independent samples"
"* Failure to understand the difference between linked and independent samples"
]
},
{
...
...
@@ -371,13 +374,12 @@
}
},
"source": [
"### Formal training in statistics and study design\n",
"### Formal training in statistics and study design for PhD students, postdocs and principal investigators\n",
" \n",
"**Core training for PhD students postdoc and principal investigators**\n",
"\n",
"* Basic statistics and graphic representation for life sciences\n",
"* Experimental methods and experimental design\n",
"* Data handling and analysis\n",
"* Basic statistics applied to life sciences\n",
"* Experimental methods design and controls\n",
"* Whole study design\n",
"* Data handling, analysis and representation\n",
"* Understand the issues of emerging technologies\n",
"* Compulsory refresher courses"
]
...
...
@@ -416,10 +418,10 @@
"source": [
"### Basic study design, in the twenty-first century \n",
"\n",
"* Study have to be designed **before starting** new projects\n",
"* Define **exclusion criteria** for experimments and to outlyer\n",
"* Requierement that **subjective end points** are assessed by blinded investigators\n",
"* Inclusion of **appropriate controls** \n",
"* Studies have to be designed **before starting** projects (QA documentation?)\n",
"* Define **criteria** to validate experiments and remove outlyers thanks to previous works or pilot studies\n",
"* **Subjective end points** must be assessed blindly\n",
"* Inclusion of **appropriate control(s)** \n",
"* Use **highly relevant** animal models in biomedical research\n",
"\n",
""
...
...
@@ -429,35 +431,35 @@
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
"slide_type": "slide"
}
},
"source": [
"### How to deal with small sample ( 5 to 30 animals per group)\n",
"\n",
"* Randomization of animals\n",
"\n",
"* Prefer biological replicates over technical\n",
" * Mix together what is mixeable\n",
"\n",
"\n",
"* For advanced preclinical studies\n",
" * Reduce the number of experiental conditions\n",
" \n",
" \n",
"* Be more stringent in statistical tests\n",
" * Use non parametric analyses unless you have a very good reason to use parametric tests"
"# EXAMPLES !!"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
"slide_type": "subslide"
}
},
"source": [
"# EXAMPLES !!"
"### How to deal with small samples (5 to 30 animals per group)\n",
"\n",
"* Real **randomization** of animals or experimental conditions\n",
"\n",
"\n",
"* **Biological replicates** instead of technical replicates\n",
" * Mix together what is mixeable\n",
"\n",
"\n",
"* **Reduce the number of experiental conditions** for advanced preclinical studies\n",
"\n",
"\n",
"* Use **stringent statistical tests** and **adapt your graphical representation**\n",
" * Non-parametric tests, unless you have a very good reason to use parametric tests"
]
},
{
...
...
@@ -481,15 +483,15 @@
"source": [
"### Standardization\n",
"\n",
"**Participate and follow international community guidelines**\n",
"**Actively looks for international community guidelines**\n",
"\n",
"* Quantitative PCR experiments => MIQE\n",
"* Digital Quantitative PCR => dMIQE\n",
"* Flow cytometry => MIFlowCyt\n",
"* T-cell assays => MIATA\n",
"* genome sequencing => MIGS\n",
"* Genome sequencing => MIGS\n",
"\n",
"**Use international reference material to normalyze data when available**\n",
"**Use international reference material when available to normalyze data**\n",
"\n",
"* Ayuso, E et al. (2014). *Manufacturing and Characterization of a rAAV Type 8 Reference Standard Material*. Hum. Gene Ther"
]
...
...
@@ -504,7 +506,7 @@
"source": [
"### Replication\n",
"\n",
"Replicates findings by other methods and external teams (for high impact journals)\n",
"**Replicate** dogma-killer findings by other methods and external teams (for high impact journals)\n",
"\n",
""
]
...
...
@@ -535,19 +537,46 @@
}
},
"source": [
"### Open and extensive reporting of materials & methods\n",
"### Open and extensive reporting\n",
"\n",
"* Report extensively **materials & methods**\n",
" * Supplementary data\n",
" * Raw data used for graphical representation\n",
" * Large datasets uploaded to public repositories\n",
" \n",
"\n",
"the world's first peer reviewed scientific video journal : the Journal of Visualized Experiments http://www.jove.com/\n",
"* Report **more and differently**\n",
" * Very short papers format \n",
" * Video Journal (http://www.jove.com/)\n",
" * Negative resuts specific journals"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "notes"
}
},
"source": [
"the world's first peer reviewed scientific video journal : the Journal of Visualized Experiments \n",
"\n",
"posting datasets to online repositories like Dryad\n",
"\n",
"Publish Negative result : Journal of Negative Results in Biomedicine, the All Results Journals (with subject-specific versions for chemistry, biology, nanotechnology and physics), and the Journal of Negative Results\n",
"\n",
"\n",
"**Journals unite for reproducibility**\n",
"Publish Negative result : Journal of Negative Results in Biomedicine, the All Results Journals (with subject-specific versions for chemistry, biology, nanotechnology and physics), and the Journal of Negative Results"
#Believe it or not but Science is in a reproducibility crisis
#Believe it or not, but Science is in a reproducibility crisis
%% Cell type:markdown id: tags:
### Irreproducibility in biological sciences is largely criticized all over the web
### Irreproducibility in biomedical research is largely criticized all over the web
%% Cell type:markdown id: tags:
### And in academic reviews
### ... and in academic reviews
*[Ioannidis, JPA (2005). *Why Most Published Research Findings Are False*. PLoS Med](http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124)
*[Florian Prinz et al (2011), *How much can we rely on published data on potential drug targets?* Nat. Rev. Drug Dis.](http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.html)
*[Landis et al. (2012). *A call for transparent reporting to optimize the predictive value of preclinical research.* Nature](http://www.nature.com/nature/journal/v490/n7419/full/nature11556.html)
...
%% Cell type:markdown id: tags:
###And Nature group created the open acces collection
### ... and Nature group created the open acces collection
### "[Challenges in irreproducible research](http://www.nature.com.gate2.inist.fr/nature/focus/reproducibility/index.html)"
#### "[Challenges in irreproducible research](http://www.nature.com.gate2.inist.fr/nature/focus/reproducibility/index.html)"
%% Cell type:markdown id: tags:
#What are the consequences on industrial productivity?
%% Cell type:markdown id: tags:
### Drug companies fail to reproduce a large proportion of academic discoveries
**Amgen : biotech company in cancer research**
* Try to reproduce the findings of 53 “landmark” articles in cancer research
* Only 6 of the 53 studies were reproduced (about 10%)
**Bayer, the pharmaceutical company**
**Bayer : Big-Pharma company**
* Examined 67 target-validation projects in oncology, women’s health, and cardiovascular medicine
* Published results were reproduced in only 14 out of 67 projects (about 21%).
%% Cell type:markdown id: tags:
George A. Robertson (Ex Senior Scientist at Merck)
*"It drives people in industry crazy. Why are we seeing a collapse of the pharma and biotech industries? One possibility is that academia is not providing accurate findings"*
%% Cell type:markdown id: tags:
### High failure rate of clinical trial
[Arrowsmith, J (2011). *Trial watch: Phase II failures: 2008–2010*. Nature Reviews Drug Discovery 10: 328–329.](http://www.nature.com/nrd/journal/v10/n5/full/nrd3439.html)
108 failures of clinical trials divided according to cause and therapeutic area.
###Scientific fraud is real issue but it concerns only an small proportion of irreproductible studies
%% Cell type:markdown id: tags:
### Unrelated data panels / missing references / incorrect controls
* Comparing mdx mice with C57/B6J
*Example: Comparing a murine model vs an "healthy" control with a different genetic background
%% Cell type:markdown id: tags:
### Pseudo replicates
* Failure to understand the difference linked and independent samples
* Failure to understand the difference between linked and independent samples
%% Cell type:markdown id: tags:
### Data handling
* Interim data analysis and ad hoc exclusion of data
%% Cell type:markdown id: tags:
### Misusage and Over-interpretation of statistical tests
Nuzzo, R (2014). *Scientific method: Statistical errors* Nature
Halsey, LG, et al (2015). *The fickle P value generates irreproducible results* Nat. Meth.
%% Cell type:markdown id: tags:
### Underpowered studies = Small effect sizes
**When unlikely hypotheses are tested, most positive results of underpowered studies can be wrong**
Button, KS, et al (2013). *Power failure: why small sample size undermines the reliability of neuroscience*. Nat. Rev. Neuro.
Krzywinski, M & Altman, N (2013). *Points of significance: Power and sample size*. Nat. Meth.
%% Cell type:markdown id: tags:
### Inaccurate and incomplete reporting of methods
* Negative results are virtually never reported
* Impossibility to reproduce results due to missing details or errors
* Chain of references back to 1952...
%% Cell type:markdown id: tags:
#How do we fix it?
%% Cell type:markdown id: tags:
### Formal training in statistics and study design
**Core training for PhD students postdoc and principal investigators**
### Formal training in statistics and study design for PhD students, postdocs and principal investigators
* Basic statistics and graphic representation for life sciences
* Experimental methods and experimental design
* Data handling and analysis
* Basic statistics applied to life sciences
* Experimental methods design and controls
* Whole study design
* Data handling, analysis and representation
* Understand the issues of emerging technologies
* Compulsory refresher courses
%% Cell type:markdown id: tags:
*"In essence, this is the right thing to do, if only it was not for me to do it"*
%% Cell type:markdown id: tags:
**Or hire a biostatistician/methodologist to design YOUR studies and analyse YOUR data**
%% Cell type:markdown id: tags:
### Basic study design, in the twenty-first century
* Study have to be designed **before starting**new projects
* Define **exclusion criteria**for experimments and to outlyer
*Requierement that **subjective end points**are assessed by blinded investigators
* Inclusion of **appropriate controls**
* Studies have to be designed **before starting** projects (QA documentation?)
* Define **criteria**to validate experiments and remove outlyers thanks to previous works or pilot studies
***Subjective end points**must be assessed blindly
* Inclusion of **appropriate control(s)**
* Use **highly relevant** animal models in biomedical research
%% Cell type:markdown id: tags:
### How to deal with small sample ( 5 to 30 animals per group)
# EXAMPLES !!
* Randomization of animals
%% Cell type:markdown id: tags:
* Prefer biological replicates over technical
* Mix together what is mixeable
### How to deal with small samples (5 to 30 animals per group)
* Real **randomization** of animals or experimental conditions
* For advanced preclinical studies
* Reduce the number of experiental conditions
***Biological replicates** instead of technical replicates
* Mix together what is mixeable
* Be more stringent in statistical tests
* Use non parametric analyses unless you have a very good reason to use parametric tests
%% Cell type:markdown id: tags:
***Reduce the number of experiental conditions** for advanced preclinical studies
# EXAMPLES !!
* Use **stringent statistical tests** and **adapt your graphical representation**
* Non-parametric tests, unless you have a very good reason to use parametric tests
%% Cell type:markdown id: tags:
#Large scale efforts and future directions
%% Cell type:markdown id: tags:
### Standardization
**Participate and follow international community guidelines**
**Actively looks for international community guidelines**
* Quantitative PCR experiments => MIQE
* Digital Quantitative PCR => dMIQE
* Flow cytometry => MIFlowCyt
* T-cell assays => MIATA
*genome sequencing => MIGS
*Genome sequencing => MIGS
**Use international reference material to normalyze data when available**
**Use international reference material when available to normalyze data**
* Ayuso, E et al. (2014). *Manufacturing and Characterization of a rAAV Type 8 Reference Standard Material*. Hum. Gene Ther
%% Cell type:markdown id: tags:
### Replication
Replicates findings by other methods and external teams (for high impact journals)
**Replicate** dogma-killer findings by other methods and external teams (for high impact journals)
%% Cell type:markdown id: tags:
### Lab Robotics & Automation
Reduce experimenter variability and contaminations during preparation of samples (in conjunction with standardization)
Check Hayden, E (2014). *The automated lab.* Nature 516: 131–132.
%% Cell type:markdown id: tags:
### Open and extensive reporting of materials & methods
### Open and extensive reporting
* Report extensively **materials & methods**
* Supplementary data
* Raw data used for graphical representation
* Large datasets uploaded to public repositories
the world's first peer reviewed scientific video journal : the Journal of Visualized Experiments http://www.jove.com/
* Report **more and differently**
* Very short papers format
* Video Journal (http://www.jove.com/)
* Negative resuts specific journals
%% Cell type:markdown id: tags:
the world's first peer reviewed scientific video journal : the Journal of Visualized Experiments
posting datasets to online repositories like Dryad
Publish Negative result : Journal of Negative Results in Biomedicine, the All Results Journals (with subject-specific versions for chemistry, biology, nanotechnology and physics), and the Journal of Negative Results
%% Cell type:markdown id: tags:
**Journals unite for reproducibility**
### Journals unite for reproducibility
%% Cell type:markdown id: tags:
**Principles and Guidelines for Reporting Preclinical Research**
"#Believe it or not but Science is in a reproducibility crisis"
"#Believe it or not, but Science is in a reproducibility crisis"
]
},
{
...
...
@@ -113,7 +115,7 @@
}
},
"source": [
"### Irreproducibility in biological sciences is largely criticized all over the web\n",
"### Irreproducibility in biomedical research is largely criticized all over the web\n",
"\n",
""
]
...
...
@@ -126,7 +128,7 @@
}
},
"source": [
"### And in academic reviews \n",
"### ... and in academic reviews \n",
"\n",
"* [Ioannidis, JPA (2005). *Why Most Published Research Findings Are False*. PLoS Med](http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124)\n",
"\n",
...
...
@@ -147,9 +149,10 @@
}
},
"source": [
"###And Nature group created the open acces collection\n",
"### ... and Nature group created the open acces collection\n",
"\n",
"### \"[Challenges in irreproducible research](http://www.nature.com.gate2.inist.fr/nature/focus/reproducibility/index.html)\"\n",
"\n",
"#### \"[Challenges in irreproducible research](http://www.nature.com.gate2.inist.fr/nature/focus/reproducibility/index.html)\"\n",
"\n",
"\n"
]
...
...
@@ -178,11 +181,13 @@
"**Amgen : biotech company in cancer research**\n",
"\n",
"* Try to reproduce the findings of 53 “landmark” articles in cancer research\n",
"\n",
"* Only 6 of the 53 studies were reproduced (about 10%)\n",
"\n",
"**Bayer, the pharmaceutical company**\n",
"**Bayer : Big-Pharma company**\n",
"\n",
"* Examined 67 target-validation projects in oncology, women’s health, and cardiovascular medicine\n",
"\n",
"* Published results were reproduced in only 14 out of 67 projects (about 21%)."
]
},
...
...
@@ -213,9 +218,7 @@
"\n",
"[Arrowsmith, J (2011). *Trial watch: Phase II failures: 2008–2010*. Nature Reviews Drug Discovery 10: 328–329.](http://www.nature.com/nrd/journal/v10/n5/full/nrd3439.html)\n",
"\n",
"108 failures of clinical trials divided according to cause and therapeutic area.\n",