Commit a42d545a authored by Erwan Bousse's avatar Erwan Bousse

Updating the website with everything concerning the evaluation tool suite.

parent f938de90
output
tmp
crash.log
---
title: Benchmarking
---
Running the evaluation
=======
Coming soon.
---
title: Running the evaluation
---
Running the evaluation
=======
## Bash scripts
We wrote <i>bash</i> scripts to simplify the run of the evaluation. They are available in the folder <code>evaluation-scripts</code> of the [tool suite](../tools/evaluation).
* <code>generate-all</code> generates 10 OSGi bundles each with the java code for 10 generated metamodels
* <code>run-all</code> takes generated OSGi bundle as arguments, and run the evaluation for both 1 clone and 1000 clones
* <code>plot-all</code> takes generated CSV files as arguments, and created many plots to visualize the results
They can be easily run in a linux environement, but also in a Windows environment using [win-bash](http://win-bash.sourceforge.net/).
## Running
cd /path/to/evaluation-scripts
./generate-all
A folder <code>benchmarktool-workingdir</code> is created, in which will be created 10 OSGi bundles named bundle1.jar, bundle2.jar, etc.
./run-all benchmarktool-workingdir/*.jar
The evaluation starts, and eventually creates CSV files in <code>benchmarktool-workingdir</code>.
./plot-all benchmarktool-workingdir/*.csv
[R](http://www.r-project.org/) is called to plot the data, and PDF files are created in <code>benchmarktool-workingdir</code>.
...@@ -41,7 +41,7 @@ Using the cloning strategies ...@@ -41,7 +41,7 @@ Using the cloning strategies
----- -----
In this second part we use the provided API to clone models using the different strategies. Here is a sample of code that creates a model using the same metamodel as previsously, and then clones it using the four cloning strategies: In this second part we use the provided API to clone models using the different strategies. Here is a sample of code that creates a model using the same metamodel as previsously, and then clones it using the four cloning strategies:
<hr>
#!java #!java
/* Creating a model */ /* Creating a model */
...@@ -83,4 +83,3 @@ In this second part we use the provided API to clone models using the different ...@@ -83,4 +83,3 @@ In this second part we use the provided API to clone models using the different
Cloner shareAllCloner = new ClonerImpl(MyMMShareAllCloningMaterial.getInstance()); Cloner shareAllCloner = new ClonerImpl(MyMMShareAllCloningMaterial.getInstance());
ResourceSet clone4 = shareAllCloner.clone(mymodel, "shareAlldeepClones"); ResourceSet clone4 = shareAllCloner.clone(mymodel, "shareAlldeepClones");
<hr>
/*code { background: #f8f8f8; }*/ span { font-family: monospace; }
pre { display: block; background: #f8f8f8; }
code { background: #f8f8f8; font-family: monospace; }
code .hll { background-color: #ffffcc } code .hll { background-color: #ffffcc }
code .c { color: #8f5902; font-style: italic } /* Comment */ code .c { color: #8f5902; font-style: italic } /* Comment */
code .err { color: #a40000; border: 1px solid #ef2929 } /* Error */ code .err { color: #a40000; border: 1px solid #ef2929 } /* Error */
......
* { * {
margin: 0;
padding: 0; padding: 0;
font-family: Georgia, Palatino, serif; font-family: Georgia, Palatino, serif;
...@@ -28,7 +27,8 @@ a:hover { ...@@ -28,7 +27,8 @@ a:hover {
top: 40px; top: 40px;
left: 280px; left: 280px;
width: 500px; width: 1000px;
padding-bottom: 90px;
} }
#main h1 { #main h1 {
......
...@@ -6,6 +6,10 @@ title: Evaluation tool suite ...@@ -6,6 +6,10 @@ title: Evaluation tool suite
Evaluation tool suite Evaluation tool suite
======= =======
To measure the obtained memory gains when using Moclodash for cloning models, we made a complete evaluation tool suite.
## Downloading the tool
You can download the latest build of the evaluation tool suite at [this page](https://ci.inria.fr/moclodash/job/moclodash-evaluation/lastSuccessfulBuild/). You can download the latest build of the evaluation tool suite at [this page](https://ci.inria.fr/moclodash/job/moclodash-evaluation/lastSuccessfulBuild/).
Five versions are available: Five versions are available:
...@@ -15,3 +19,68 @@ Five versions are available: ...@@ -15,3 +19,68 @@ Five versions are available:
* Macos x86_64 * Macos x86_64
* Windows x86 * Windows x86
* Windows x86_64 * Windows x86_64
Note that this tool suite contains all the required dependencies, including the moclodash plugins themselves. Thus, you don't have to download them manually to run the evaluation.
Java 1.7 required.
## Using the tool
This CLI tool has three modes: <code>mmgen</code>, <code>matgen</code> and <code>run</code>.
### Metamodel generation mode
The mode <code>mmgen</code> is used to randomly generate metamodels (ie ecore models).
$ ./cloning-benchmark mmgen
Listening for transport dt_socket at address: 35337
Invalid options: Missing required option(s) ['nbMetamodels', 'maxMutProp', 'outputFolder', 'maxProp', 'maxClasses']
Option (* = required) Description
--------------------- -----------
* --maxClasses <Integer: natural> The maximum number of classes of a
metamodel.
* --maxMutProp <Integer: natural> The maximum number of mutable
properties of a metamodel.
* --maxProp <Integer: natural> The maximum number of properties of a
* --nbMetamodels <Integer: natural> The number of metamodels to generate.
* --outputFolder <File: path> The folder in which metamodels are
generated.
### Cloning material generation mode
The mode <code>matgen</code> is used to call Moclodash cloning material generator on a set of metamodels in order to generate an OSGi bundle ready for evaluation.
$ ./cloning-benchmark matgen
Listening for transport dt_socket at address: 60140
[INFO.] Starting cloning benchmark material generator.
[ERROR] Invalid options: Missing required option(s) ['metamodelsFolder', 'outputFile']
Option (* = required) Description
--------------------- -----------
* --metamodelsFolder <File: path> A folder containing ecore files to
process.
* --outputFile <File: path> The OSGI jar bundle to create.
--outputProjectFolder <File: path> The folder in which all the code is
generated, as a complete eclipse
project. NO SPACE CHARACTERS /!\.
## Run mode
The mode <code>run</code> is used to run the actual evaluation process on an OSGi bundle previously generated (ie a set of metamodels).
$ ./cloning-benchmark run
Listening for transport dt_socket at address: 37191
[ERROR] Invalid options: Missing required option(s) ['nbtries', 'outputFile', 'benchmarkMetamodels', 'nbClones']
Option (* = required) Description
--------------------- -----------
* --benchmarkMetamodels <File: path1, OSGI bundles with all metamodels,
path2,...> created with the generator.
--cloningOperators Cloning operators to use.
--dumpsFolder <File: path> A folder in which memory dumps will be
written.
--keepDumps To keep memory dumps.
--metamodels Specific metamodels to use.
* --nbClones <Integer: 1,10,100...> Series of numbers of clones to create.
* --nbtries <Integer> The number of tries, for better
time/performance results
* --outputFile <File: path> The CSV file to write.
...@@ -32,8 +32,8 @@ ...@@ -32,8 +32,8 @@
<h2>Documentation</h2> <h2>Documentation</h2>
<ul> <ul>
<li><a href="/doc/getting-started">Getting started</a></li> <li><a href="/doc/getting-started">Getting started</a></li>
<li><a href="/doc/benchmarking">Running the evaluation</a></li> <li><a href="/doc/evaluation">Running the evaluation</a></li>
<li><a href="/doc/research">Research</a></li> <!--<li><a href="/doc/research">Research</a></li>-->
</ul> </ul>
</div> </div>
</body> </body>
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment