Skip to content

GitLab

Commit 4a1ad757 authored by Adrien Leger's avatar Adrien Leger
Browse files

Finishing presentation

parent 7774cc00
Hide whitespace changes
Inline Side-by-side
2015_04_29_Improve_methodology_retraite_AGTI/.ipynb_checkpoints/2015_04_29_Improve_methodology_retraite_AGTI-checkpoint.ipynb
View file @ 4a1ad757
%% Cell type:code id: tags:
``` python
```
%% Cell type:markdown id: tags:
# PREPARE RISE PRESENTATION
# PREPARE RISE PARAMETERS
%% Cell type:code id: tags:
``` python
from IPython.html.services.config import ConfigManager
from IPython.utils.path import locate_profile
cm = ConfigManager(profile_dir=locate_profile(get_ipython().profile))
cm.update('livereveal', {
'theme': 'blood',
'transition': 'linear',
'start_slideshow_at': 'selected'})
# transition = default/cube/page/concave/zoom/linear/none
# theme = beige/blood/default/moon/night/serif/simple/sky/solarized
# start_slideshow_at = selected/beginning
```
%%%% Output: execute_result
{'start_slideshow_at': 'selected', 'theme': 'blood', 'transition': 'linear'}
%% Cell type:markdown id: tags:
# PRESENTATION STARTS BELOW
%% Cell type:markdown id: tags:
#Can we improve our methodology ?
%% Cell type:markdown id: tags:
**Sure we can!**
%% Cell type:markdown id: tags:
#Why we have to improve our methodologies, analyses and reporting
**Emilie Lecomte / Jean-Baptiste Dupont / Célia Couzinié / Adrien Leger**
**Retraite AGT / 29 Avril 2015**
%% Cell type:markdown id: tags:
#Believe it or not but Science is in a reproducibility crisis
* Florian Prinz et al (2011), *How much can we rely on published data on potential drug targets?* Nature Reviews Drug Discovery 10, 712
* Editorial board (2012). *Must try harder*. Nature 483: 509–509.
%% Cell type:markdown id: tags:
* Landis et al. (2012). *A call for transparent reporting to optimize the predictive value of preclinical research.* Nature 490: 187–191.
### Irreproducibility in biological sciences is largely criticized all over the web
* Ioannidis, JPA (2005). *Why Most Published Research Findings Are False*. PLoS Med 2: e124.
![](files/fig/presse.png)
%% Cell type:markdown id: tags:
### I have numbers...
### And in academic reviews
* [Ioannidis, JPA (2005). *Why Most Published Research Findings Are False*. PLoS Med](http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124)
%% Cell type:markdown id: tags:
* [Florian Prinz et al (2011), *How much can we rely on published data on potential drug targets?* Nat. Rev. Drug Dis.](http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.html)
###More numbers ...
* [Editorial board (2012). *Must try harder*. Nature](http://www.nature.com/nature/journal/v483/n7391/full/483509a.html)
%% Cell type:markdown id: tags:
* [Landis et al. (2012). *A call for transparent reporting to optimize the predictive value of preclinical research.* Nature](http://www.nature.com/nature/journal/v490/n7419/full/nature11556.html)
bla bla
...
%% Cell type:markdown id: tags:
###And Nature group created the open acces collection
### "[Challenges in irreproducible research](http://www.nature.com.gate2.inist.fr/nature/focus/reproducibility/index.html)"
![Design](files/fig/nature.png)
%% Cell type:markdown id: tags:
#What are the consequences ?
#What are the consequences on industrial productivity?
%% Cell type:markdown id: tags:
### Drug companies fail to reproduce a large proportion of academic discoveries
Public sourcing Drug development pipeline from Bayer HealthCare (and many other companies)
**Amgen : biotech company in cancer research**
* Find exciting published data in scientific literature
* In house replication program
* Large scale clinical trial
* Try to reproduce the findings of 53 “landmark” articles in cancer research
* Only 6 of the 53 studies were reproduced (about 10%)
![](files/fig/reproducibility_piechart.png)
**Bayer, the pharmaceutical company**
%% Cell type:markdown id: tags:
* Examined 67 target-validation projects in oncology, women’s health, and cardiovascular medicine
* Published results were reproduced in only 14 out of 67 projects (about 21%).
### High failure rate of clinical trial
%% Cell type:markdown id: tags:
Arrowsmith, J (2011). *Trial watch: Phase II failures: 2008–2010*. Nature Reviews Drug Discovery 10: 328–329.
![](files/fig/reproducibility_piechart.png)
108 failures of clinical trials divided according to cause and therapeutic area.
![](http://www.nature.com/nrd/journal/v10/n5/images/nrd3439-f1.jpg)
George A. Robertson (Ex Senior Scientist at Merck)
*"It drives people in industry crazy. Why are we seeing a collapse of the pharma and biotech industries? One possibility is that academia is not providing accurate findings"*
%% Cell type:markdown id: tags:
###Biotech and Big Pharma lose confidence in academics
### High failure rate of clinical trial
[Arrowsmith, J (2011). *Trial watch: Phase II failures: 2008–2010*. Nature Reviews Drug Discovery 10: 328–329.](http://www.nature.com/nrd/journal/v10/n5/full/nrd3439.html)
Drug companies face this probem all the time. They read about cutting-edge discoveries being made in academic labs, but when they try to reproduce the experiments, they can’t. Scientists at a German pharmaceutical company who tried to reproduce the results in 67 published studies told the readers of Nature that they only succeeded one quarter of the time. Likewise, the American company Amgen found they could only replicate the results for 6 out of 53 published cancer studies.
108 failures of clinical trials divided according to cause and therapeutic area.
![](files/fig/irrepro_target_validation.jpg)
![](http://www.nature.com/nrd/journal/v10/n5/images/nrd3439-f1.jpg)
%% Cell type:markdown id: tags:
#What caused the reproducibility crisis?
%% Cell type:markdown id: tags:
### Scientific fraud
"*How to succeed in science (without doing any)*"
![](files/fig/fraud-saint.jpg)
%% Cell type:markdown id: tags:
** Real issue but concerns only an small proportion of irreproductible studies**
###Scientific fraud is real issue but it concerns only an small proportion of irreproductible studies
![](files/fig/Research conduct.jpg)
%% Cell type:markdown id: tags:
### Unrelated data panels / missing references / incorrect controls
=> big classic
* Comparing mdx mice with C57/B6J
%% Cell type:markdown id: tags:
### Pseudo replicates
Failure to understand the difference between technical and biological replicates =
* Failure to understand the difference linked and independent samples
%% Cell type:markdown id: tags:
### Data handling
* Interim data analysis
* Ad hoc exclusion of data
* Interim data analysis and ad hoc exclusion of data
%% Cell type:markdown id: tags:
### Misusage and Over-interpretation of statistical tests
Nuzzo, R (2014). *Scientific method: Statistical errors* Nature 506: 150–152.
Nuzzo, R (2014). *Scientific method: Statistical errors* Nature
Halsey, LG, et al (2015). *The fickle P value generates irreproducible results* Nature Methods 12: 179–185.
Halsey, LG, et al (2015). *The fickle P value generates irreproducible results* Nat. Meth.
![](files/fig/pvalue_art.jpg)
%% Cell type:markdown id: tags:
### Underpowered studies = Small effect sizes
When unlikely hypotheses are tested, most positive results of underpowered studies can be wrong
**When unlikely hypotheses are tested, most positive results of underpowered studies can be wrong**
Button, KS, et al (2013). *Power failure: why small sample size undermines the reliability of neuroscience*. Nat. Rev. Neuro.
Krzywinski, M & Altman, N (2013). *Points of significance: Power and sample size*. Nature Methods 10: 1139–1140.
Krzywinski, M & Altman, N (2013). *Points of significance: Power and sample size*. Nat. Meth.
![](files/fig/underpower.png)
%% Cell type:markdown id: tags:
### Inaccurate and incomplete reporting of methods
Impossibility to reproduce results example
* Negative results are rarely reported
* Impossibility to reproduce results due to missing details or errors
* Chain of references back to 1952...
![](files/fig/type_machine.jpeg)
%% Cell type:markdown id: tags:
#How do we fix it?
%% Cell type:markdown id: tags:
### Formal training in statistics and study design
**Core training for PhD students postdoc and principal investigators**
* Basic statistics and graphic representation for life sciences
* Experimental methods and experimental design
* Data handling and analysis
* Understand the issues of emerging technologies
* Compulsory refresher courses
%% Cell type:markdown id: tags:
![](files/fig/animated_principle.gif)
*"In essence, this is the right thing to do, if only it was not for me to do it"*
%% Cell type:markdown id: tags:
**Or hire a biostatistician/methodologist**
**Or hire a biostatistician/methodologist to design YOUR studies and analyse YOUR data**
%% Cell type:markdown id: tags:
### Basic study design, in the twenty-first century
* Study have to be designed **before starting** new projects
* Define **exclusion criteria** for experimments and to outlyer
* Requierement that **subjective end points** are assessed by blinded investigators
* Inclusion of **appropriate controls**
* Use **highly relevant** animal models in biomedical research
![](files/fig/blindfolded.jpg)
%% Cell type:markdown id: tags:
### How to deal with small sample ( 5 to 30 animals per group)
* Randomization of animals
* Prefer biological replicates over technical => Mix together what is mixeable = independant
* Prefer biological replicates over technical
* Mix together what is mixeable
* For advanced preclinical studies
* Reduce the number of experiental conditions
* Be more stringent in statistical tests
* Use non parametric analyses unless you have a very good reason to use parametric tests
%% Cell type:markdown id: tags:
### Large scale efforts
**Standardization**
Participate and follow international community guidelines
**Replication**
Replicates findings by other methods and external teams (for high impact journals)
![](files/fig/reproducibility_opener.jpg)
%% Cell type:markdown id: tags:
### Future directions
**Lab Robotics & Automation**
Reduce experimenter variability and contamination during preparation of samples.
Check Hayden, E (2014). *The automated lab.* Nature 516: 131–132.
![](files/fig/Tool.jpg)
%% Cell type:markdown id: tags:
### Open and extensive reporting of methods
### Open and extensive reporting of materials & methods
**Journals unite for reproducibility**
![](files/fig/journals.png)
%% Cell type:markdown id: tags:
**=> Report the methodology extensively, Provide raw tables in sup data in addition to graphics**
**Principles and Guidelines for Reporting Preclinical Research**
* Non publication of negative results
![](files/fig/Core_rules.png)
......
2015_04_29_Improve_methodology_retraite_AGTI/2015_04_29_Improve_methodology_retraite_AGTI.ipynb
View file @ 4a1ad757
%% Cell type:code id: tags:
``` python
```
%% Cell type:markdown id: tags:
# PREPARE RISE PRESENTATION
# PREPARE RISE PARAMETERS
%% Cell type:code id: tags:
``` python
from IPython.html.services.config import ConfigManager
from IPython.utils.path import locate_profile
cm = ConfigManager(profile_dir=locate_profile(get_ipython().profile))
cm.update('livereveal', {
'theme': 'blood',
'transition': 'linear',
'start_slideshow_at': 'selected'})
# transition = default/cube/page/concave/zoom/linear/none
# theme = beige/blood/default/moon/night/serif/simple/sky/solarized
# start_slideshow_at = selected/beginning
```
%%%% Output: execute_result
{'start_slideshow_at': 'selected', 'theme': 'blood', 'transition': 'linear'}
%% Cell type:markdown id: tags:
# PRESENTATION STARTS BELOW
%% Cell type:markdown id: tags:
#Can we improve our methodology ?
%% Cell type:markdown id: tags:
**Sure we can!**
%% Cell type:markdown id: tags:
#Why we have to improve our methodologies, analyses and reporting
**Emilie Lecomte / Jean-Baptiste Dupont / Célia Couzinié / Adrien Leger**
**Retraite AGT / 29 Avril 2015**
%% Cell type:markdown id: tags:
#Believe it or not but Science is in a reproducibility crisis
* Florian Prinz et al (2011), *How much can we rely on published data on potential drug targets?* Nature Reviews Drug Discovery 10, 712
* Editorial board (2012). *Must try harder*. Nature 483: 509–509.
%% Cell type:markdown id: tags:
* Landis et al. (2012). *A call for transparent reporting to optimize the predictive value of preclinical research.* Nature 490: 187–191.
### Irreproducibility in biological sciences is largely criticized all over the web
* Ioannidis, JPA (2005). *Why Most Published Research Findings Are False*. PLoS Med 2: e124.
![](files/fig/presse.png)
%% Cell type:markdown id: tags:
### I have numbers...
### And in academic reviews
* [Ioannidis, JPA (2005). *Why Most Published Research Findings Are False*. PLoS Med](http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124)
%% Cell type:markdown id: tags:
* [Florian Prinz et al (2011), *How much can we rely on published data on potential drug targets?* Nat. Rev. Drug Dis.](http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.html)
###More numbers ...
* [Editorial board (2012). *Must try harder*. Nature](http://www.nature.com/nature/journal/v483/n7391/full/483509a.html)
%% Cell type:markdown id: tags:
* [Landis et al. (2012). *A call for transparent reporting to optimize the predictive value of preclinical research.* Nature](http://www.nature.com/nature/journal/v490/n7419/full/nature11556.html)
bla bla
...
%% Cell type:markdown id: tags:
###And Nature group created the open acces collection
### "[Challenges in irreproducible research](http://www.nature.com.gate2.inist.fr/nature/focus/reproducibility/index.html)"
![Design](files/fig/nature.png)
%% Cell type:markdown id: tags:
#What are the consequences ?
#What are the consequences on industrial productivity?
%% Cell type:markdown id: tags:
### Drug companies fail to reproduce a large proportion of academic discoveries
Public sourcing Drug development pipeline from Bayer HealthCare (and many other companies)
**Amgen : biotech company in cancer research**
* Find exciting published data in scientific literature
* In house replication program
* Large scale clinical trial
* Try to reproduce the findings of 53 “landmark” articles in cancer research
* Only 6 of the 53 studies were reproduced (about 10%)
![](files/fig/reproducibility_piechart.png)
**Bayer, the pharmaceutical company**
%% Cell type:markdown id: tags:
* Examined 67 target-validation projects in oncology, women’s health, and cardiovascular medicine
* Published results were reproduced in only 14 out of 67 projects (about 21%).
### High failure rate of clinical trial
%% Cell type:markdown id: tags:
Arrowsmith, J (2011). *Trial watch: Phase II failures: 2008–2010*. Nature Reviews Drug Discovery 10: 328–329.
![](files/fig/reproducibility_piechart.png)
108 failures of clinical trials divided according to cause and therapeutic area.
![](http://www.nature.com/nrd/journal/v10/n5/images/nrd3439-f1.jpg)
George A. Robertson (Ex Senior Scientist at Merck)
*"It drives people in industry crazy. Why are we seeing a collapse of the pharma and biotech industries? One possibility is that academia is not providing accurate findings"*
%% Cell type:markdown id: tags:
###Biotech and Big Pharma lose confidence in academics
### High failure rate of clinical trial
[Arrowsmith, J (2011). *Trial watch: Phase II failures: 2008–2010*. Nature Reviews Drug Discovery 10: 328–329.](http://www.nature.com/nrd/journal/v10/n5/full/nrd3439.html)
Drug companies face this probem all the time. They read about cutting-edge discoveries being made in academic labs, but when they try to reproduce the experiments, they can’t. Scientists at a German pharmaceutical company who tried to reproduce the results in 67 published studies told the readers of Nature that they only succeeded one quarter of the time. Likewise, the American company Amgen found they could only replicate the results for 6 out of 53 published cancer studies.
108 failures of clinical trials divided according to cause and therapeutic area.
![](files/fig/irrepro_target_validation.jpg)
![](http://www.nature.com/nrd/journal/v10/n5/images/nrd3439-f1.jpg)
%% Cell type:markdown id: tags:
#What caused the reproducibility crisis?
%% Cell type:markdown id: tags:
### Scientific fraud
"*How to succeed in science (without doing any)*"
![](files/fig/fraud-saint.jpg)
%% Cell type:markdown id: tags:
** Real issue but concerns only an small proportion of irreproductible studies**
###Scientific fraud is real issue but it concerns only an small proportion of irreproductible studies
![](files/fig/Research conduct.jpg)
%% Cell type:markdown id: tags:
### Unrelated data panels / missing references / incorrect controls
=> big classic
* Comparing mdx mice with C57/B6J
%% Cell type:markdown id: tags:
### Pseudo replicates
Failure to understand the difference between technical and biological replicates =
* Failure to understand the difference linked and independent samples
%% Cell type:markdown id: tags:
### Data handling
* Interim data analysis
* Ad hoc exclusion of data
* Interim data analysis and ad hoc exclusion of data
%% Cell type:markdown id: tags:
### Misusage and Over-interpretation of statistical tests
Nuzzo, R (2014). *Scientific method: Statistical errors* Nature 506: 150–152.
Nuzzo, R (2014). *Scientific method: Statistical errors* Nature
Halsey, LG, et al (2015). *The fickle P value generates irreproducible results* Nature Methods 12: 179–185.
Halsey, LG, et al (2015). *The fickle P value generates irreproducible results* Nat. Meth.
![](files/fig/pvalue_art.jpg)
%% Cell type:markdown id: tags:
### Underpowered studies = Small effect sizes
When unlikely hypotheses are tested, most positive results of underpowered studies can be wrong
**When unlikely hypotheses are tested, most positive results of underpowered studies can be wrong**
Button, KS, et al (2013). *Power failure: why small sample size undermines the reliability of neuroscience*. Nat. Rev. Neuro.
Krzywinski, M & Altman, N (2013). *Points of significance: Power and sample size*. Nature Methods 10: 1139–1140.
Krzywinski, M & Altman, N (2013). *Points of significance: Power and sample size*. Nat. Meth.
![](files/fig/underpower.png)
%% Cell type:markdown id: tags:
### Inaccurate and incomplete reporting of methods
Impossibility to reproduce results example
* Negative results are rarely reported
* Impossibility to reproduce results due to missing details or errors
* Chain of references back to 1952...
![](files/fig/type_machine.jpeg)
%% Cell type:markdown id: tags:
#How do we fix it?
%% Cell type:markdown id: tags:
### Formal training in statistics and study design
**Core training for PhD students postdoc and principal investigators**
* Basic statistics and graphic representation for life sciences
* Experimental methods and experimental design
* Data handling and analysis
* Understand the issues of emerging technologies
* Compulsory refresher courses
%% Cell type:markdown id: tags:
![](files/fig/animated_principle.gif)
*"In essence, this is the right thing to do, if only it was not for me to do it"*
%% Cell type:markdown id: tags:
**Or hire a biostatistician/methodologist**
**Or hire a biostatistician/methodologist to design YOUR studies and analyse YOUR data**
%% Cell type:markdown id: tags:
### Basic study design, in the twenty-first century
* Study have to be designed **before starting** new projects
* Define **exclusion criteria** for experimments and to outlyer
* Requierement that **subjective end points** are assessed by blinded investigators