Speed Tests

[Emmanuel Lecharny <el...@gmail.com>]

Hi all,

I have done some speed tests lately on all the versions of ApacheDS :
 * 0.8
 * 0.9
 * 0.9.2
 * 0.9.3
 * 1.0-RC1
 * 2.0-RC1 + patch

This is a very simple test, done with Jmeter : Bind, search N times the 
same user, and unbind. Usually, N * nbthreads = 5 000

Here are what I get :
* 0.8 : unstable. I was able to do around 236 search request per second, 
with one thread, and I hve got many error.
* 0.9 : Strange enough, the fastest of all the different versions : I 
reached 744 searches per second with 5 threads.
* 0.9.1 : not tested yet
* 0.9.2 : Impossible to do some tests with more than one thread. Peak at 
555 searches per second.
* 0.9.3 : Same problem than 0.9.2. I got 469 req/s
* 1.0-RC1 : a peak at 566 req/s. Performance degradation after a few 
seconds : there is a memory leak.
* 1.0-RC1 + patch : more than 780 req/s with one thread. No more memory 
leak. *But* : just freeze with more than one thread, after a few 
thousand requests. There is a concurrent access problem in this version...

Ok,  it was just to inform you, developpers and users,  about which kind 
of performance we can expect from ADS, and also which kind of problems 
we are facing. Obviously, 1.0-RC1 is *not* production ready. And 1.0-RC2 
is still in its pre-infancy.

Those tests will be improved, in order to be reproduced when needed, for 
each new versions. Some numbers will be stored somewhere to be able to 
compare the results of newer versions with older results.

If goodwill people can help us to define better load tests, this will be 
appreciate !

Hope it helps,
Emmanuel Lécharny

Re: Speed Tests

[Jörg Henne <j....@levigo.de>]

Emmanuel Lecharny schrieb:
>
> Ok,  it was just to inform you, developpers and users,  about which 
> kind of performance we can expect from ADS, and also which kind of 
> problems we are facing. Obviously, 1.0-RC1 is *not* production ready. 
> And 1.0-RC2 is still in its pre-infancy.
I can confirm that there is a deadlock problem with concurrent access. I 
had to work around the problem by artificially serializing access and 
using JNDI connection pools.

Joerg Henne

Re: Speed Tests

[Emmanuel Lecharny <el...@gmail.com>]

Alex Karasulu a écrit :

> Emmanuel Lecharny wrote:
> ...
>
>> * 2.0-RC1 + patch
>
> You probably meant RC2 here.

Not really RC2 yet :) We need to find a fix for the concurrent problem.
 

>> * 0.9 : Strange enough, the fastest of all the different versions : I 
>> reached 744 searches per second with 5 threads.
>
> With correctness comes a cost ... various schema checks and such are 
> slowing down the server.

Exactly. I was surprized when I saw the results, but they are biased. 
First, we have added a lot of fixes and functionalities since 0.9. 
Second, starting with 0.9.2, we have this concurrent problem that forbid 
us to go to higher numbers.

>> * 1.0-RC1 + patch : more than 780 req/s with one thread. No more 
>> memory leak. *But* : just freeze with more than one thread, after a 
>> few thousand requests. There is a concurrent access problem in this 
>> version...
>
> Shooot!  We have to find this concurrency issue.  :(

Yeah. Hard work though.

>> Obviously, 1.0-RC1 is *not* production ready. And 1.0-RC2 is still in 
>> its pre-infancy.
>
> Aahhha!

Baby need new diapers ? Let's provide some !!!

I think all those results are just good news. There are bugs, we know 
that, and this is good, because a known bug is a almost fixed bug.

Emmanuel

Re: Speed Tests

[Alex Karasulu <ao...@bellsouth.net>]

Ole Ersoy wrote:
> I'm sure everyone on this list has already seen / know
> about this new eclipse TPTP article, but I figured I'd
> send it out just in case ...
>
> http://www.eclipse.org/articles/Article-TPTP-Profiling-Tool/tptpProfilingArticle.html
>
> I skimmed it once and it seems like a really powerful
> tool for performance testing.  I'll give it a go on
> Apache DS as soon as I get a few of these other
> projects out of the way.
>
>   
Cool we'd love to see any stats you can produce with ApacheDS.  Also 
looking at bottlenecks of deadlocks etc.

Thanks,
Alex

Re: Speed Tests

[Ole Ersoy <ol...@yahoo.com>]

I'm sure everyone on this list has already seen / know
about this new eclipse TPTP article, but I figured I'd
send it out just in case ...

http://www.eclipse.org/articles/Article-TPTP-Profiling-Tool/tptpProfilingArticle.html

I skimmed it once and it seems like a really powerful
tool for performance testing.  I'll give it a go on
Apache DS as soon as I get a few of these other
projects out of the way.

--- Alex Karasulu <ao...@bellsouth.net> wrote:

> Emmanuel Lecharny wrote:
> ...
> > * 2.0-RC1 + patch
> You probably meant RC2 here.
> ...
> > This is a very simple test, done with Jmeter :
> Bind, search N times 
> > the same user, and unbind. Usually, N * nbthreads
> = 5 000
> >
> > Here are what I get :
> > * 0.8 : unstable. I was able to do around 236
> search request per 
> > second, with one thread, and I hve got many error.
> > * 0.9 : Strange enough, the fastest of all the
> different versions : I 
> > reached 744 searches per second with 5 threads.
> With correctness comes a cost ... various schema
> checks and such are 
> slowing down the server.
> > * 0.9.1 : not tested yet
> > * 0.9.2 : Impossible to do some tests with more
> than one thread. Peak 
> > at 555 searches per second.
> > * 0.9.3 : Same problem than 0.9.2. I got 469 req/s
> > * 1.0-RC1 : a peak at 566 req/s. Performance
> degradation after a few 
> > seconds : there is a memory leak.
> > * 1.0-RC1 + patch : more than 780 req/s with one
> thread. No more 
> > memory leak. *But* : just freeze with more than
> one thread, after a 
> > few thousand requests. There is a concurrent
> access problem in this 
> > version...
> Shooot!  We have to find this concurrency issue.  :(
> > Ok,  it was just to inform you, developpers and
> users,  about which 
> > kind of performance we can expect from ADS, and
> also which kind of 
> > problems we are facing. Obviously, 1.0-RC1 is
> *not* production ready. 
> > And 1.0-RC2 is still in its pre-infancy.
> Aahhha!
> 
> Alex
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Speed Tests

[Alex Karasulu <ao...@bellsouth.net>]

Emmanuel Lecharny wrote:
...
> * 2.0-RC1 + patch
You probably meant RC2 here.
...
> This is a very simple test, done with Jmeter : Bind, search N times 
> the same user, and unbind. Usually, N * nbthreads = 5 000
>
> Here are what I get :
> * 0.8 : unstable. I was able to do around 236 search request per 
> second, with one thread, and I hve got many error.
> * 0.9 : Strange enough, the fastest of all the different versions : I 
> reached 744 searches per second with 5 threads.
With correctness comes a cost ... various schema checks and such are 
slowing down the server.
> * 0.9.1 : not tested yet
> * 0.9.2 : Impossible to do some tests with more than one thread. Peak 
> at 555 searches per second.
> * 0.9.3 : Same problem than 0.9.2. I got 469 req/s
> * 1.0-RC1 : a peak at 566 req/s. Performance degradation after a few 
> seconds : there is a memory leak.
> * 1.0-RC1 + patch : more than 780 req/s with one thread. No more 
> memory leak. *But* : just freeze with more than one thread, after a 
> few thousand requests. There is a concurrent access problem in this 
> version...
Shooot!  We have to find this concurrency issue.  :(
> Ok,  it was just to inform you, developpers and users,  about which 
> kind of performance we can expect from ADS, and also which kind of 
> problems we are facing. Obviously, 1.0-RC1 is *not* production ready. 
> And 1.0-RC2 is still in its pre-infancy.
Aahhha!

Alex