[perf tests] proposal

[Emmanuel Lecharny <el...@apache.org>]

Hi guys,

as we discussed with Niklas about how to test MINA perfs, we agreed that 
it would be interesting to start a subproject to conduct performance 
tests. This is also something Julien mentionned as 'to  be done for MINA 
3.0'.

So here is the proposal :

Mina-Perf subproject

We need a way to test a MINA based application. We also need a way to 
test MINA itself.

There are many dimension we want to test, namely :
- throughput (number of message per second, depending on the message size)
- number of client we can support (scalability)
- CPU consumption
- Memory consumption
- reliability on the long term (in other words, do we have memory leaks)

In order to build this tool, we need to define its architecture. The 
main issue is to determinate which part of MINA sucks CPU and Memory, 
excluded the application (depending on the number of filters we add, we 
will have different results).

We also need a platform to run those tests, and we probably want them 
running often, a bit like a CI system.

wdyt ?

-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org

Re: [perf tests] proposal

[Julien Vermillard <jv...@archean.fr>]

We need to test latency too.

Julien

Le Mon, 15 Jun 2009 14:34:23 +0200,
Emmanuel Lecharny <el...@apache.org> a écrit :

> Hi guys,
> 
> as we discussed with Niklas about how to test MINA perfs, we agreed
> that it would be interesting to start a subproject to conduct
> performance tests. This is also something Julien mentionned as 'to
> be done for MINA 3.0'.
> 
> So here is the proposal :
> 
> Mina-Perf subproject
> 
> We need a way to test a MINA based application. We also need a way to 
> test MINA itself.
> 
> There are many dimension we want to test, namely :
> - throughput (number of message per second, depending on the message
> size)
> - number of client we can support (scalability)
> - CPU consumption
> - Memory consumption
> - reliability on the long term (in other words, do we have memory
> leaks)
> 
> In order to build this tool, we need to define its architecture. The 
> main issue is to determinate which part of MINA sucks CPU and Memory, 
> excluded the application (depending on the number of filters we add,
> we will have different results).
> 
> We also need a platform to run those tests, and we probably want them 
> running often, a bit like a CI system.
> 
> wdyt ?
> 

Re: [perf tests] proposal

[Niklas Gustavsson <ni...@protocol7.com>]

On Mon, Jun 15, 2009 at 2:34 PM, Emmanuel Lecharny<el...@apache.org> wrote:
> We also need a platform to run those tests, and we probably want them
> running often, a bit like a CI system.

How about aiming for triggering the tests from Hudson (where we run
our CI builds). Of course, the current Hudson boxes are to loaded to
actually run the tests, but we could set up a Hudson slave (like
Lucene has) for this purpose. This would allow these tests to be
trigged by SVN commits and we would get our results as part of the
existing build reports. I would be happy to code on Hudson
integration. Of course, the big question is on what box to run.

/niklas

Re: [perf tests] proposal

[Ashish <pa...@gmail.com>]

On Mon, Jun 15, 2009 at 6:04 PM, Emmanuel Lecharny<el...@apache.org> wrote:
> Hi guys,
>
> as we discussed with Niklas about how to test MINA perfs, we agreed that it
> would be interesting to start a subproject to conduct performance tests.
> This is also something Julien mentionned as 'to  be done for MINA 3.0'.
>
> So here is the proposal :
>
> Mina-Perf subproject

+1
probably generic enough to be used by other projects. I guess we did
discussed Protocol Testing Framework
at length in one of discussions :-)

>
> We need a way to test a MINA based application. We also need a way to test
> MINA itself.
>
> There are many dimension we want to test, namely :
> - throughput (number of message per second, depending on the message size)

Along with type of decoder's taken into consideration

> - number of client we can support (scalability)
> - CPU consumption
> - Memory consumption
> - reliability on the long term (in other words, do we have memory leaks)
>
> In order to build this tool, we need to define its architecture. The main
> issue is to determinate which part of MINA sucks CPU and Memory, excluded
> the application (depending on the number of filters we add, we will have
> different results).

I think we can we can derive stuff from already existing platforms,
and customize it to fit our needs or
we can write from scratch ??

> We also need a platform to run those tests, and we probably want them
> running often, a bit like a CI system.

So if I understood it correctly, it will be more like functional load
testing, like FtpServer code running (with only Codecs and mocked
functionality) and benchmarking the code.

>
> wdyt ?

Lets work out the architecture :-)

-- 
thanks
ashish