[jira] [Created] (IGNITE-3756) Yardstick benchmark roadmap

["Sergey Kozlov (JIRA)" <ji...@apache.org>]

Sergey Kozlov created IGNITE-3756:
-------------------------------------

             Summary: Yardstick benchmark roadmap
                 Key: IGNITE-3756
                 URL: https://issues.apache.org/jira/browse/IGNITE-3756
             Project: Ignite
          Issue Type: Task
          Components: yardstick
            Reporter: Sergey Kozlov


Now I'm trying to collect most important things that can significantly improve it and make Yardstick development/use more clear and simpler.

h3. 1. Usage

h4. 1.1 Deploy script/mode over network must be provided. 
Localhost benchmark has to require no ssh connection. 
It's really weird that key SSH authentication and SSH server is running required to run benchmarks on local laptop.

h4. 1.2 One script for deploy, run servers, run drivers, collect logs from servers, generate HTML report. 
It should have minimal set of options: path to configuration and the operations. For instance:
make everything: {{./becnhmark.sh --config=ignite-local-benchmarks.xml}}
deploy only: {{./becnhmark.sh --config=ignite-local-benchmarks.xml --operation=deploy}}

h4. 1.3 Introduction one XML configuration.
Instead of {{benchmark.properties}}, {{ignite.xml}} and command line options for a benchmarks just put ones into a spring XML file. 
For instance it may be org.apache.ignite.yardstick.configuration for common parameters and
org.apache.ignite.yardstick.configuration.<benchmark name> for specific parameters

h4. 1.4 Driver logging. 
Driver(client) should provide more details why it can't connect to servers, deploy code and etc, what it is doing right now. Key points:
a) No key SSH authentication configured
b) Can't connect to servers
c) Print out "alive" message (e.g. "NNNNN operations performed for <benchmark name>") every X seconds (configuratin option) to prevent late detecting of deadlocks, hangs
d) Print out average numbers at end of benchmark run (which usually calculated by jfreechart). 

h3. 2. Deploy binaries

Add the maven profile that will include yardstick in fabric binary package (at least for Apache Ignite). It will provide the yardstick 
out-of-box and will make easier for first run.

h3. 3. Sources

Is there any reason why at the moment we don't include the benchmark code for competitors like HZ/Coherence into Apache Ignite Git reporsitory?
Time to time we can't even compile the yardstick-<competitor> due uncompatible changes and etc. 
Put such benchmark code into Apache Ignite yardsctick module will make maintaining the code up-to-date, compilable and runnable.

h3. 4. New benchmark introduction

We should to create rules to introduction of a new benchmark. Now I see that some benchmarks have hardcoded options, others add new command-line arguments, config directory like a bin. It should be splitted somehow.
Ideally any commit should be reviewed as we do it for product code.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)