Commit 5f6f3824 authored by Adrien Leger's avatar Adrien Leger
Browse files

Update presentation

parent 4a1ad757
%% Cell type:markdown id: tags:
# PREPARE RISE PARAMETERS
%% Cell type:code id: tags:
``` python
from IPython.html.services.config import ConfigManager
from IPython.utils.path import locate_profile
cm = ConfigManager(profile_dir=locate_profile(get_ipython().profile))
cm.update('livereveal', {
'theme': 'blood',
'transition': 'linear',
'start_slideshow_at': 'selected'})
# transition = default/cube/page/concave/zoom/linear/none
# theme = beige/blood/default/moon/night/serif/simple/sky/solarized
# start_slideshow_at = selected/beginning
```
%%%% Output: execute_result
{'start_slideshow_at': 'selected', 'theme': 'blood', 'transition': 'linear'}
%% Cell type:markdown id: tags:
# PRESENTATION STARTS BELOW
%% Cell type:markdown id: tags:
#Can we improve our methodology ?
%% Cell type:markdown id: tags:
**Sure we can!**
%% Cell type:markdown id: tags:
#Why we have to improve our methodologies, analyses and reporting
**Emilie Lecomte / Jean-Baptiste Dupont / Célia Couzinié / Adrien Leger**
**Retraite AGT / 29 Avril 2015**
%% Cell type:markdown id: tags:
#Believe it or not but Science is in a reproducibility crisis
%% Cell type:markdown id: tags:
### Irreproducibility in biological sciences is largely criticized all over the web
![](files/fig/presse.png)
%% Cell type:markdown id: tags:
### And in academic reviews
* [Ioannidis, JPA (2005). *Why Most Published Research Findings Are False*. PLoS Med](http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124)
* [Florian Prinz et al (2011), *How much can we rely on published data on potential drug targets?* Nat. Rev. Drug Dis.](http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.html)
* [Editorial board (2012). *Must try harder*. Nature](http://www.nature.com/nature/journal/v483/n7391/full/483509a.html)
* [Landis et al. (2012). *A call for transparent reporting to optimize the predictive value of preclinical research.* Nature](http://www.nature.com/nature/journal/v490/n7419/full/nature11556.html)
...
%% Cell type:markdown id: tags:
###And Nature group created the open acces collection
### "[Challenges in irreproducible research](http://www.nature.com.gate2.inist.fr/nature/focus/reproducibility/index.html)"
![Design](files/fig/nature.png)
%% Cell type:markdown id: tags:
#What are the consequences on industrial productivity?
%% Cell type:markdown id: tags:
### Drug companies fail to reproduce a large proportion of academic discoveries
**Amgen : biotech company in cancer research**
* Try to reproduce the findings of 53 “landmark” articles in cancer research
* Only 6 of the 53 studies were reproduced (about 10%)
**Bayer, the pharmaceutical company**
* Examined 67 target-validation projects in oncology, women’s health, and cardiovascular medicine
* Published results were reproduced in only 14 out of 67 projects (about 21%).
%% Cell type:markdown id: tags:
![](files/fig/reproducibility_piechart.png)
George A. Robertson (Ex Senior Scientist at Merck)
*"It drives people in industry crazy. Why are we seeing a collapse of the pharma and biotech industries? One possibility is that academia is not providing accurate findings"*
%% Cell type:markdown id: tags:
### High failure rate of clinical trial
[Arrowsmith, J (2011). *Trial watch: Phase II failures: 2008–2010*. Nature Reviews Drug Discovery 10: 328–329.](http://www.nature.com/nrd/journal/v10/n5/full/nrd3439.html)
108 failures of clinical trials divided according to cause and therapeutic area.
![](http://www.nature.com/nrd/journal/v10/n5/images/nrd3439-f1.jpg)
%% Cell type:markdown id: tags:
#What caused the reproducibility crisis?
%% Cell type:markdown id: tags:
### Scientific fraud
"*How to succeed in science (without doing any)*"
![](files/fig/fraud-saint.jpg)
%% Cell type:markdown id: tags:
###Scientific fraud is real issue but it concerns only an small proportion of irreproductible studies
![](files/fig/Research conduct.jpg)
%% Cell type:markdown id: tags:
### Unrelated data panels / missing references / incorrect controls
* Comparing mdx mice with C57/B6J
%% Cell type:markdown id: tags:
### Pseudo replicates
* Failure to understand the difference linked and independent samples
%% Cell type:markdown id: tags:
### Data handling
* Interim data analysis and ad hoc exclusion of data
%% Cell type:markdown id: tags:
### Misusage and Over-interpretation of statistical tests
Nuzzo, R (2014). *Scientific method: Statistical errors* Nature
Halsey, LG, et al (2015). *The fickle P value generates irreproducible results* Nat. Meth.
![](files/fig/pvalue_art.jpg)
%% Cell type:markdown id: tags:
### Underpowered studies = Small effect sizes
**When unlikely hypotheses are tested, most positive results of underpowered studies can be wrong**
Button, KS, et al (2013). *Power failure: why small sample size undermines the reliability of neuroscience*. Nat. Rev. Neuro.
Krzywinski, M & Altman, N (2013). *Points of significance: Power and sample size*. Nat. Meth.
![](files/fig/underpower.png)
%% Cell type:markdown id: tags:
### Inaccurate and incomplete reporting of methods
* Negative results are rarely reported
* Negative results are virtually never reported
* Impossibility to reproduce results due to missing details or errors
* Chain of references back to 1952...
![](files/fig/type_machine.jpeg)
%% Cell type:markdown id: tags:
#How do we fix it?
%% Cell type:markdown id: tags:
### Formal training in statistics and study design
**Core training for PhD students postdoc and principal investigators**
* Basic statistics and graphic representation for life sciences
* Experimental methods and experimental design
* Data handling and analysis
* Understand the issues of emerging technologies
* Compulsory refresher courses
%% Cell type:markdown id: tags:
![](files/fig/animated_principle.gif)
*"In essence, this is the right thing to do, if only it was not for me to do it"*
%% Cell type:markdown id: tags:
**Or hire a biostatistician/methodologist to design YOUR studies and analyse YOUR data**
%% Cell type:markdown id: tags:
### Basic study design, in the twenty-first century
* Study have to be designed **before starting** new projects
* Define **exclusion criteria** for experimments and to outlyer
* Requierement that **subjective end points** are assessed by blinded investigators
* Inclusion of **appropriate controls**
* Use **highly relevant** animal models in biomedical research
![](files/fig/blindfolded.jpg)
%% Cell type:markdown id: tags:
### How to deal with small sample ( 5 to 30 animals per group)
* Randomization of animals
* Prefer biological replicates over technical
* Mix together what is mixeable
* For advanced preclinical studies
* Reduce the number of experiental conditions
* Be more stringent in statistical tests
* Use non parametric analyses unless you have a very good reason to use parametric tests
%% Cell type:markdown id: tags:
### Large scale efforts
# EXAMPLES !!
**Standardization**
%% Cell type:markdown id: tags:
#Large scale efforts and future directions
%% Cell type:markdown id: tags:
### Standardization
**Participate and follow international community guidelines**
* Quantitative PCR experiments => MIQE
* Digital Quantitative PCR => dMIQE
* Flow cytometry => MIFlowCyt
* T-cell assays => MIATA
* genome sequencing => MIGS
**Use international reference material to normalyze data when available**
Participate and follow international community guidelines
* Ayuso, E et al. (2014). *Manufacturing and Characterization of a rAAV Type 8 Reference Standard Material*. Hum. Gene Ther
**Replication**
%% Cell type:markdown id: tags:
### Replication
Replicates findings by other methods and external teams (for high impact journals)
![](files/fig/reproducibility_opener.jpg)
%% Cell type:markdown id: tags:
### Future directions
### Lab Robotics & Automation
**Lab Robotics & Automation**
Reduce experimenter variability and contamination during preparation of samples.
Reduce experimenter variability and contaminations during preparation of samples (in conjunction with standardization)
Check Hayden, E (2014). *The automated lab.* Nature 516: 131–132.
![](files/fig/Tool.jpg)
%% Cell type:markdown id: tags:
### Open and extensive reporting of materials & methods
the world's first peer reviewed scientific video journal : the Journal of Visualized Experiments http://www.jove.com/
posting datasets to online repositories like Dryad
Publish Negative result : Journal of Negative Results in Biomedicine, the All Results Journals (with subject-specific versions for chemistry, biology, nanotechnology and physics), and the Journal of Negative Results
**Journals unite for reproducibility**
![](files/fig/journals.png)
%% Cell type:markdown id: tags:
**Principles and Guidelines for Reporting Preclinical Research**
![](files/fig/Core_rules.png)
......
%% Cell type:markdown id: tags:
# PREPARE RISE PARAMETERS
%% Cell type:code id: tags:
``` python
from IPython.html.services.config import ConfigManager
from IPython.utils.path import locate_profile
cm = ConfigManager(profile_dir=locate_profile(get_ipython().profile))
cm.update('livereveal', {
'theme': 'blood',
'transition': 'linear',
'start_slideshow_at': 'selected'})
# transition = default/cube/page/concave/zoom/linear/none
# theme = beige/blood/default/moon/night/serif/simple/sky/solarized
# start_slideshow_at = selected/beginning
```
%%%% Output: execute_result
{'start_slideshow_at': 'selected', 'theme': 'blood', 'transition': 'linear'}
%% Cell type:markdown id: tags:
# PRESENTATION STARTS BELOW
%% Cell type:markdown id: tags:
#Can we improve our methodology ?
%% Cell type:markdown id: tags:
**Sure we can!**
%% Cell type:markdown id: tags:
#Why we have to improve our methodologies, analyses and reporting
**Emilie Lecomte / Jean-Baptiste Dupont / Célia Couzinié / Adrien Leger**
**Retraite AGT / 29 Avril 2015**
%% Cell type:markdown id: tags:
#Believe it or not but Science is in a reproducibility crisis
%% Cell type:markdown id: tags:
### Irreproducibility in biological sciences is largely criticized all over the web
![](files/fig/presse.png)
%% Cell type:markdown id: tags:
### And in academic reviews
* [Ioannidis, JPA (2005). *Why Most Published Research Findings Are False*. PLoS Med](http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124)
* [Florian Prinz et al (2011), *How much can we rely on published data on potential drug targets?* Nat. Rev. Drug Dis.](http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.html)
* [Editorial board (2012). *Must try harder*. Nature](http://www.nature.com/nature/journal/v483/n7391/full/483509a.html)
* [Landis et al. (2012). *A call for transparent reporting to optimize the predictive value of preclinical research.* Nature](http://www.nature.com/nature/journal/v490/n7419/full/nature11556.html)
...
%% Cell type:markdown id: tags:
###And Nature group created the open acces collection
### "[Challenges in irreproducible research](http://www.nature.com.gate2.inist.fr/nature/focus/reproducibility/index.html)"
![Design](files/fig/nature.png)
%% Cell type:markdown id: tags:
#What are the consequences on industrial productivity?
%% Cell type:markdown id: tags:
### Drug companies fail to reproduce a large proportion of academic discoveries
**Amgen : biotech company in cancer research**
* Try to reproduce the findings of 53 “landmark” articles in cancer research
* Only 6 of the 53 studies were reproduced (about 10%)
**Bayer, the pharmaceutical company**
* Examined 67 target-validation projects in oncology, women’s health, and cardiovascular medicine
* Published results were reproduced in only 14 out of 67 projects (about 21%).
%% Cell type:markdown id: tags:
![](files/fig/reproducibility_piechart.png)
George A. Robertson (Ex Senior Scientist at Merck)
*"It drives people in industry crazy. Why are we seeing a collapse of the pharma and biotech industries? One possibility is that academia is not providing accurate findings"*
%% Cell type:markdown id: tags:
### High failure rate of clinical trial
[Arrowsmith, J (2011). *Trial watch: Phase II failures: 2008–2010*. Nature Reviews Drug Discovery 10: 328–329.](http://www.nature.com/nrd/journal/v10/n5/full/nrd3439.html)
108 failures of clinical trials divided according to cause and therapeutic area.
![](http://www.nature.com/nrd/journal/v10/n5/images/nrd3439-f1.jpg)
%% Cell type:markdown id: tags:
#What caused the reproducibility crisis?
%% Cell type:markdown id: tags:
### Scientific fraud
"*How to succeed in science (without doing any)*"
![](files/fig/fraud-saint.jpg)
%% Cell type:markdown id: tags:
###Scientific fraud is real issue but it concerns only an small proportion of irreproductible studies
![](files/fig/Research conduct.jpg)
%% Cell type:markdown id: tags:
### Unrelated data panels / missing references / incorrect controls
* Comparing mdx mice with C57/B6J
%% Cell type:markdown id: tags:
### Pseudo replicates
* Failure to understand the difference linked and independent samples
%% Cell type:markdown id: tags:
### Data handling
* Interim data analysis and ad hoc exclusion of data
%% Cell type:markdown id: tags:
### Misusage and Over-interpretation of statistical tests
Nuzzo, R (2014). *Scientific method: Statistical errors* Nature
Halsey, LG, et al (2015). *The fickle P value generates irreproducible results* Nat. Meth.
![](files/fig/pvalue_art.jpg)
%% Cell type:markdown id: tags:
### Underpowered studies = Small effect sizes
**When unlikely hypotheses are tested, most positive results of underpowered studies can be wrong**
Button, KS, et al (2013). *Power failure: why small sample size undermines the reliability of neuroscience*. Nat. Rev. Neuro.
Krzywinski, M & Altman, N (2013). *Points of significance: Power and sample size*. Nat. Meth.
![](files/fig/underpower.png)
%% Cell type:markdown id: tags:
### Inaccurate and incomplete reporting of methods
* Negative results are rarely reported
* Negative results are virtually never reported
* Impossibility to reproduce results due to missing details or errors
* Chain of references back to 1952...
![](files/fig/type_machine.jpeg)
%% Cell type:markdown id: tags:
#How do we fix it?
%% Cell type:markdown id: tags:
### Formal training in statistics and study design
**Core training for PhD students postdoc and principal investigators**
* Basic statistics and graphic representation for life sciences
* Experimental methods and experimental design
* Data handling and analysis
* Understand the issues of emerging technologies
* Compulsory refresher courses
%% Cell type:markdown id: tags:
![](files/fig/animated_principle.gif)
*"In essence, this is the right thing to do, if only it was not for me to do it"*
%% Cell type:markdown id: tags:
**Or hire a biostatistician/methodologist to design YOUR studies and analyse YOUR data**
%% Cell type:markdown id: tags:
### Basic study design, in the twenty-first century
* Study have to be designed **before starting** new projects
* Define **exclusion criteria** for experimments and to outlyer
* Requierement that **subjective end points** are assessed by blinded investigators
* Inclusion of **appropriate controls**
* Use **highly relevant** animal models in biomedical research
![](files/fig/blindfolded.jpg)
%% Cell type:markdown id: tags:
### How to deal with small sample ( 5 to 30 animals per group)
* Randomization of animals
* Prefer biological replicates over technical
* Mix together what is mixeable
* For advanced preclinical studies
* Reduce the number of experiental conditions
* Be more stringent in statistical tests
* Use non parametric analyses unless you have a very good reason to use parametric tests
%% Cell type:markdown id: tags:
### Large scale efforts
# EXAMPLES !!
**Standardization**
%% Cell type:markdown id: tags:
#Large scale efforts and future directions
%% Cell type:markdown id: tags:
### Standardization
**Participate and follow international community guidelines**
* Quantitative PCR experiments => MIQE
* Digital Quantitative PCR => dMIQE
* Flow cytometry => MIFlowCyt
* T-cell assays => MIATA
* genome sequencing => MIGS
**Use international reference material to normalyze data when available**
Participate and follow international community guidelines
* Ayuso, E et al. (2014). *Manufacturing and Characterization of a rAAV Type 8 Reference Standard Material*. Hum. Gene Ther
**Replication**
%% Cell type:markdown id: tags:
### Replication
Replicates findings by other methods and external teams (for high impact journals)
![](files/fig/reproducibility_opener.jpg)
%% Cell type:markdown id: tags:
### Future directions