update presentation retraite AGTI

7774cc00 · Adrien Leger · 94ff3cab · 94ff3cab · 7774cc00 · 7774cc00
Commit 7774cc00 authored Apr 21, 2015 by Adrien Leger
--- a/2015_04_29_Improve_methodology_retraite_AGTI.html
+++ b/2015_04_29_Improve_methodology_retraite_AGTI.html
--- a/2015_04_29_Improve_methodology_retraite_AGTI/.ipynb_checkpoints/2015_04_29_Improve_methodology_retraite_AGTI-checkpoint.ipynb
+++ b/2015_04_29_Improve_methodology_retraite_AGTI/.ipynb_checkpoints/2015_04_29_Improve_methodology_retraite_AGTI-checkpoint.ipynb
+%% Cell type:code id: tags:
+
+``` python
+```
+
 %% Cell type:markdown id: tags:

 # PREPARE RISE PRESENTATION

 %% Cell type:code id: tags:

 ``` python
 from IPython.html.services.config import ConfigManager
 from IPython.utils.path import locate_profile
 cm = ConfigManager(profile_dir=locate_profile(get_ipython().profile))
 cm.update('livereveal', {
-              'theme': 'solarized',
+              'theme': 'blood',
              'transition': 'linear',
              'start_slideshow_at': 'selected'})

 # transition = default/cube/page/concave/zoom/linear/none
 # theme = beige/blood/default/moon/night/serif/simple/sky/solarized
 # start_slideshow_at = selected/beginning
 ```

 %%%% Output: execute_result

-    {'start_slideshow_at': 'selected',
-     'theme': 'solarized',
-     'transition': 'linear'}
-
-%% Cell type:code id: tags:
-
-``` python
-from IPython.display import HTML, Image, SVG, YouTubeVideo
-```
+    {'start_slideshow_at': 'selected', 'theme': 'blood', 'transition': 'linear'}

 %% Cell type:markdown id: tags:

 # PRESENTATION STARTS BELOW

 %% Cell type:markdown id: tags:

 #Can we improve our methodology ?

 %% Cell type:markdown id: tags:

 **Sure we can!**

 %% Cell type:markdown id: tags:

 #Why we have to improve our methodologies, analyses and reporting

 **Emilie Lecomte / Jean-Baptiste Dupont / Célia Couzinié / Adrien Leger**

 **Retraite AGT / 29 Avril 2015**

 %% Cell type:markdown id: tags:

 #Believe it or not but Science is in a reproducibility crisis

-Florian Prinz et al (2011), *How much can we rely on published data on potential drug targets?* Nature Reviews Drug Discovery 10, 712
+* Florian Prinz et al (2011), *How much can we rely on published data on potential drug targets?* Nature Reviews Drug Discovery 10, 712

-Editorial board (2012). *Must try harder*. Nature 483: 509–509.
+* Editorial board (2012). *Must try harder*. Nature 483: 509–509.

-Landis et al. (2012). *A call for transparent reporting to optimize the predictive value of preclinical research.* Nature 490: 187–191.
+* Landis et al. (2012). *A call for transparent reporting to optimize the predictive value of preclinical research.* Nature 490: 187–191.

+* Ioannidis, JPA (2005). *Why Most Published Research Findings Are False*. PLoS Med 2: e124.

 %% Cell type:markdown id: tags:

 ### I have numbers...

-%% Cell type:markdown id: tags:
-
-![](http://www.nature.com.gate2.inist.fr/nrd/journal/v10/n9/images/nrd3439-c1-f1.jpg)

 %% Cell type:markdown id: tags:

 ###More numbers ...

 %% Cell type:markdown id: tags:

 bla bla

 %% Cell type:markdown id: tags:

 ###And Nature group created the open acces collection

 ### "[Challenges in irreproducible research](http://www.nature.com.gate2.inist.fr/nature/focus/reproducibility/index.html)"

 ![Design](files/fig/nature.png)

 %% Cell type:markdown id: tags:

 #What are the consequences ?

 %% Cell type:markdown id: tags:

-* Biotech failure
-* ...
+### Drug companies fail to reproduce a large proportion of academic discoveries
+
+Public sourcing Drug development pipeline from Bayer HealthCare (and many other companies)
+
+* Find exciting published data in scientific literature
+* In house replication program
+* Large scale clinical trial
+
+![](files/fig/reproducibility_piechart.png)
+
+%% Cell type:markdown id: tags:
+
+### High failure rate of clinical trial
+
+Arrowsmith, J (2011). *Trial watch: Phase II failures: 2008–2010*. Nature Reviews Drug Discovery 10: 328–329.
+
+108 failures of clinical trials divided according to cause and therapeutic area.
+
+![](http://www.nature.com/nrd/journal/v10/n5/images/nrd3439-f1.jpg)

 %% Cell type:markdown id: tags:

-###Biotech and Big Pharma loose confidence in academics
+###Biotech and Big Pharma lose confidence in academics
+
+Drug companies face this probem all the time. They read about cutting-edge discoveries being made in academic labs, but when they try to reproduce the experiments, they can’t. Scientists at a German pharmaceutical company who tried to reproduce the results in 67 published studies told the readers of Nature that they only succeeded one quarter of the time. Likewise, the American company Amgen found they could only replicate the results for 6 out of 53 published cancer studies.

-![](http://www.addconsortium.org/EVENTS/EV1000165.jpg)
+![](files/fig/irrepro_target_validation.jpg)

 %% Cell type:markdown id: tags:

 #What caused the reproducibility crisis?

 %% Cell type:markdown id: tags:

 ### Scientific fraud

+"*How to succeed in science (without doing any)*"
+
+![](files/fig/fraud-saint.jpg)
+
+%% Cell type:markdown id: tags:
+
 ** Real issue but concerns only an small proportion of irreproductible studies**

-![](http://languagelog.ldc.upenn.edu/myl/NoahsPublisher.gif)
+![](files/fig/Research conduct.jpg)

 %% Cell type:markdown id: tags:

 ### Unrelated data panels / missing references / incorrect controls

 => big classic

 %% Cell type:markdown id: tags:

 ###  Pseudo replicates
 Failure to understand the difference between technical and biological replicates =

 %% Cell type:markdown id: tags:

-### Underpowered studies = Small effect sizes
-
-There may be an significant difference between 2 experimental groups but the study is underpowered and the conclusion indicate an absence of difference ...
-
-Example Garcia 1 dog...
-
-%% Cell type:markdown id: tags:
-
 ### Data handling

 * Interim data analysis
 * Ad hoc exclusion of data

 %% Cell type:markdown id: tags:

 ### Misusage and Over-interpretation of statistical tests

-Nuzzo, R (2014). *Scientific method: Statistical errors*. Nature 506: 150–152.
+Nuzzo, R (2014). *Scientific method: Statistical errors* Nature 506: 150–152.
+
+Halsey, LG, et al (2015). *The fickle P value generates irreproducible results* Nature Methods 12: 179–185.
+

-![](http://www.nature.com.gate2.inist.fr/polopoly_fs/7.15481.1392118822!/image/pvalue_art.jpg_gen/derivatives/landscape_630/pvalue_art.jpg)
+![](files/fig/pvalue_art.jpg)

 %% Cell type:markdown id: tags:

-### Inaccurate and incomplete reporting of methods
+### Underpowered studies = Small effect sizes

-Impossibility to reproduce results example = experiment acinar Ed
+When unlikely hypotheses are tested, most positive results of underpowered studies can be wrong
+
+Krzywinski, M & Altman, N (2013). *Points of significance: Power and sample size*. Nature Methods 10: 1139–1140.
+
+![](files/fig/underpower.png)

-**=> Report the methodology extensively, Provide raw tables in sup data in addition to graphics**
+
+%% Cell type:markdown id: tags:
+
+### Inaccurate and incomplete reporting of methods
+
+Impossibility to reproduce results example

 %% Cell type:markdown id: tags:

 #How do we fix it?

 %% Cell type:markdown id: tags:

- ### Formal training in statistics and study design of project leader
- ### Or
- ###work with consultants in methodology design and statistics
+### Formal training in statistics and study design
+
+**Core training for PhD students postdoc and principal investigators**
+
+* Basic statistics and graphic representation for life sciences
+* Experimental methods and experimental design
+* Data handling and analysis
+* Understand the issues of emerging technologies
+* Compulsory refresher courses
+
+%% Cell type:markdown id: tags:
+
+![](files/fig/animated_principle.gif)
+
+*"In essence, this is the right thing to do, if only it was not for me to do it"*
+
+%% Cell type:markdown id: tags:
+
+**Or hire a biostatistician/methodologist**

 %% Cell type:markdown id: tags:

-**Pas le temps! Pas l'argent! ...**
+### Basic study design, in the twenty-first century
+
+* Study have to be designed **before starting** new projects
+* Define **exclusion criteria** for experimments and to outlyer
+* Requierement that **subjective end points** are assessed by blinded investigators
+* Inclusion of **appropriate controls**
+* Use **highly relevant** animal models in biomedical research

- ![](http://38.media.tumblr.com/df91e5cf7e1bfe9f8568b8e1a5a471f3/tumblr_nld3boY3Il1s8pd5uo1_500.gif)
+![](files/fig/blindfolded.jpg)

 %% Cell type:markdown id: tags:

-**Study have to be designed before starting new projects**
-**Exclusion criteria for all experimental protocols**
-**
+### How to deal with small sample ( 5 to 30 animals per group)
+
+* Randomization of animals
+
+* Prefer biological replicates over technical => Mix together what is mixeable = independant
+
+* For advanced preclinical studies
+    * Reduce the number of experiental conditions
+
+* Be more stringent in statistical tests
+    * Use non parametric analyses unless you have a very good reason to use parametric tests

 %% Cell type:markdown id: tags:

-### Small sample effects
+### Large scale efforts
+
+**Standardization**
+
+Participate and follow international community guidelines

-* Randomisation of animals
+**Replication**

-* For advanced preclinical studies = Reduce the number of experiental conditions
+Replicates findings by other methods and external teams (for high impact journals)

-* Be more stringent in statistical tests = don't choose a statistical test because it looks better with your data
+![](files/fig/reproducibility_opener.jpg)

 %% Cell type:markdown id: tags:

-Standardization
-Blinding
+### Future directions
+
+**Lab Robotics & Automation**
+
+Reduce experimenter variability and contamination during preparation of samples.
+
+Check Hayden, E (2014). *The automated lab.* Nature 516: 131–132.
+
+
+![](files/fig/Tool.jpg)

 %% Cell type:markdown id: tags:

-![](http://38.media.tumblr.com/d165f749635c27b7500f8adf2f37628a/tumblr_nl8zsrgLWw1s8pd5uo1_400.gif)
+### Open and extensive reporting of methods
+
+**=> Report the methodology extensively, Provide raw tables in sup data in addition to graphics**
+
+* Non publication of negative results

--- a/2015_04_29_Improve_methodology_retraite_AGTI/2015_04_29_Improve_methodology_retraite_AGTI.ipynb
+++ b/2015_04_29_Improve_methodology_retraite_AGTI/2015_04_29_Improve_methodology_retraite_AGTI.ipynb
+%% Cell type:code id: tags:
+
+``` python
+```
+
 %% Cell type:markdown id: tags:

 # PREPARE RISE PRESENTATION

 %% Cell type:code id: tags:

 ``` python
 from IPython.html.services.config import ConfigManager
 from IPython.utils.path import locate_profile
 cm = ConfigManager(profile_dir=locate_profile(get_ipython().profile))
 cm.update('livereveal', {
-              'theme': 'solarized',
+              'theme': 'blood',
              'transition': 'linear',
              'start_slideshow_at': 'selected'})

 # transition = default/cube/page/concave/zoom/linear/none
 # theme = beige/blood/default/moon/night/serif/simple/sky/solarized
 # start_slideshow_at = selected/beginning
 ```

 %%%% Output: execute_result

-    {'start_slideshow_at': 'selected',
-     'theme': 'solarized',
-     'transition': 'linear'}
-
-%% Cell type:code id: tags:
-
-``` python
-from IPython.display import HTML, Image, SVG, YouTubeVideo
-```
+    {'start_slideshow_at': 'selected', 'theme': 'blood', 'transition': 'linear'}

 %% Cell type:markdown id: tags:

 # PRESENTATION STARTS BELOW

 %% Cell type:markdown id: tags:

 #Can we improve our methodology ?

 %% Cell type:markdown id: tags:

 **Sure we can!**

 %% Cell type:markdown id: tags:

 #Why we have to improve our methodologies, analyses and reporting

 **Emilie Lecomte / Jean-Baptiste Dupont / Célia Couzinié / Adrien Leger**

 **Retraite AGT / 29 Avril 2015**

 %% Cell type:markdown id: tags:

 #Believe it or not but Science is in a reproducibility crisis

-Florian Prinz et al (2011), *How much can we rely on published data on potential drug targets?* Nature Reviews Drug Discovery 10, 712
+* Florian Prinz et al (2011), *How much can we rely on published data on potential drug targets?* Nature Reviews Drug Discovery 10, 712

-Editorial board (2012). *Must try harder*. Nature 483: 509–509.
+* Editorial board (2012). *Must try harder*. Nature 483: 509–509.

-Landis et al. (2012). *A call for transparent reporting to optimize the predictive value of preclinical research.* Nature 490: 187–191.
+* Landis et al. (2012). *A call for transparent reporting to optimize the predictive value of preclinical research.* Nature 490: 187–191.

+* Ioannidis, JPA (2005). *Why Most Published Research Findings Are False*. PLoS Med 2: e124.

 %% Cell type:markdown id: tags:

 ### I have numbers...

-%% Cell type:markdown id: tags:
-
-![](http://www.nature.com.gate2.inist.fr/nrd/journal/v10/n9/images/nrd3439-c1-f1.jpg)

 %% Cell type:markdown id: tags:

 ###More numbers ...

 %% Cell type:markdown id: tags:

 bla bla

 %% Cell type:markdown id: tags:

 ###And Nature group created the open acces collection

 ### "[Challenges in irreproducible research](http://www.nature.com.gate2.inist.fr/nature/focus/reproducibility/index.html)"

 ![Design](files/fig/nature.png)

 %% Cell type:markdown id: tags:

 #What are the consequences ?

 %% Cell type:markdown id: tags:

-* Biotech failure
-* ...
+### Drug companies fail to reproduce a large proportion of academic discoveries
+
+Public sourcing Drug development pipeline from Bayer HealthCare (and many other companies)
+
+* Find exciting published data in scientific literature
+* In house replication program
+* Large scale clinical trial
+
+![](files/fig/reproducibility_piechart.png)
+
+%% Cell type:markdown id: tags:
+
+### High failure rate of clinical trial
+
+Arrowsmith, J (2011). *Trial watch: Phase II failures: 2008–2010*. Nature Reviews Drug Discovery 10: 328–329.
+
+108 failures of clinical trials divided according to cause and therapeutic area.
+
+![](http://www.nature.com/nrd/journal/v10/n5/images/nrd3439-f1.jpg)

 %% Cell type:markdown id: tags:

-###Biotech and Big Pharma loose confidence in academics
+###Biotech and Big Pharma lose confidence in academics
+
+Drug companies face this probem all the time. They read about cutting-edge discoveries being made in academic labs, but when they try to reproduce the experiments, they can’t. Scientists at a German pharmaceutical company who tried to reproduce the results in 67 published studies told the readers of Nature that they only succeeded one quarter of the time. Likewise, the American company Amgen found they could only replicate the results for 6 out of 53 published cancer studies.

-![](http://www.addconsortium.org/EVENTS/EV1000165.jpg)
+![](files/fig/irrepro_target_validation.jpg)

 %% Cell type:markdown id: tags:

 #What caused the reproducibility crisis?

 %% Cell type:markdown id: tags:

 ### Scientific fraud

+"*How to succeed in science (without doing any)*"
+
+![](files/fig/fraud-saint.jpg)
+
+%% Cell type:markdown id: tags:
+
 ** Real issue but concerns only an small proportion of irreproductible studies**

-![](http://languagelog.ldc.upenn.edu/myl/NoahsPublisher.gif)
+![](files/fig/Research conduct.jpg)

 %% Cell type:markdown id: tags:

 ### Unrelated data panels / missing references / incorrect controls

 => big classic

 %% Cell type:markdown id: tags:

 ###  Pseudo replicates
 Failure to understand the difference between technical and biological replicates =

 %% Cell type:markdown id: tags:

-### Underpowered studies = Small effect sizes
-
-There may be an significant difference between 2 experimental groups but the study is underpowered and the conclusion indicate an absence of difference ...
-
-Example Garcia 1 dog...
-
-%% Cell type:markdown id: tags:
-
 ### Data handling

 * Interim data analysis
 * Ad hoc exclusion of data

 %% Cell type:markdown id: tags:

 ### Misusage and Over-interpretation of statistical tests

-Nuzzo, R (2014). *Scientific method: Statistical errors*. Nature 506: 150–152.
+Nuzzo, R (2014). *Scientific method: Statistical errors* Nature 506: 150–152.
+
+Halsey, LG, et al (2015). *The fickle P value generates irreproducible results* Nature Methods 12: 179–185.
+

-![](http://www.nature.com.gate2.inist.fr/polopoly_fs/7.15481.1392118822!/image/pvalue_art.jpg_gen/derivatives/landscape_630/pvalue_art.jpg)
+![](files/fig/pvalue_art.jpg)

 %% Cell type:markdown id: tags:

-### Inaccurate and incomplete reporting of methods
+### Underpowered studies = Small effect sizes

-Impossibility to reproduce results example = experiment acinar Ed
+When unlikely hypotheses are tested, most positive results of underpowered studies can be wrong
+
+Krzywinski, M & Altman, N (2013). *Points of significance: Power and sample size*. Nature Methods 10: 1139–1140.
+
+![](files/fig/underpower.png)

-**=> Report the methodology extensively, Provide raw tables in sup data in addition to graphics**
+
+%% Cell type:markdown id: tags:
+
+### Inaccurate and incomplete reporting of methods
+
+Impossibility to reproduce results example

 %% Cell type:markdown id: tags:

 #How do we fix it?

 %% Cell type:markdown id: tags:

- ### Formal training in statistics and study design of project leader
- ### Or
- ###work with consultants in methodology design and statistics
+### Formal training in statistics and study design
+
+**Core training for PhD students postdoc and principal investigators**
+
+* Basic statistics and graphic representation for life sciences
+* Experimental methods and experimental design
+* Data handling and analysis
+* Understand the issues of emerging technologies
+* Compulsory refresher courses
+
+%% Cell type:markdown id: tags:
+
+![](files/fig/animated_principle.gif)
+
+*"In essence, this is the right thing to do, if only it was not for me to do it"*
+
+%% Cell type:markdown id: tags:
+
+**Or hire a biostatistician/methodologist**

 %% Cell type:markdown id: tags:

-**Pas le temps! Pas l'argent! ...**
+### Basic study design, in the twenty-first century
+
+* Study have to be designed **before starting** new projects
+* Define **exclusion criteria** for experimments and to outlyer
+* Requierement that **subjective end points** are assessed by blinded investigators
+* Inclusion of **appropriate controls**
+* Use **highly relevant** animal models in biomedical research

- ![](http://38.media.tumblr.com/df91e5cf7e1bfe9f8568b8e1a5a471f3/tumblr_nld3boY3Il1s8pd5uo1_500.gif)
+![](files/fig/blindfolded.jpg)

 %% Cell type:markdown id: tags:

-**Study have to be designed before starting new projects**
-**Exclusion criteria for all experimental protocols**
-**
+### How to deal with small sample ( 5 to 30 animals per group)
+
+* Randomization of animals