Some more speed improvement for basic operations

[Emmanuel Lecharny <el...@gmail.com>]

Hi guys,

today I finished to clean the rename() operation, and I just have the 
three last operations remaining to be clean, ie move, moveAndRename and 
compare.

In the meantime, I was able to avoid an Entry clone to be done, and this 
has a huge impact on some operations. Here are the result I get with or 
without the clone :

add    : 578/s(with) / 607/s (without) / +5%
lookup : 19 568/s(with) / 26 542/s (without) / +36%
search : 19 727/s(with) / 19 560/s (without) / ---
modify : 1 991/s(with) / 2103/s (without) / +5%
delete : 248/s(with) / 248/s (without) / ---

As we can see, the lookup is really faster with such a modif. The other 
operations aren't that much impacted, the cost of writing on disk kills 
the gain we could have.

One more thing : this is a test done with one single thread, directly on 
top of the core-session.

However, it demonstrates that with enough cache, and a good network 
layer, we should be able to get some good performances out of the server.

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.nextury.com

Re: Some more speed improvement for basic operations

[Kiran Ayyagari <ka...@apache.org>]

On Wed, Jul 28, 2010 at 8:35 PM, Emmanuel Lecharny <el...@apache.org> wrote:
> An update :
>
> as of today (July 28th), I'm able to run the search perf test and obtain
> those values :
>
> June, 8th :
> search : 19 727 search/s
>
> July 28th : 35 812 search/s
this is great :)
>
> NOTE : this is for a simple search (ie, OBJECT scope) and no entry in the
> base. The test does not go through any network (though no BER
> encoding/decoding). It just check the speed of the internal server.
>
> Also it's interesting as it prove there is a wide margin of improvement in
> the server itself.
>
> From the last test I did, I think we can gain 30 to 40% more by using a DN
> cache (DN parsing and normalization count for 26% of the whole CPU), and we
> have a spurious call to the hasEntry() method which costs 13,5% of the CPU.
> The log system costs 2.75%, something we can also improve.
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>


Kiran Ayyagari

Re: Some more speed improvement for basic operations

[Emmanuel Lecharny <el...@apache.org>]

An update :

as of today (July 28th), I'm able to run the search perf test and obtain
those values :

June, 8th :
search : 19 727 search/s

July 28th : 35 812 search/s

NOTE : this is for a simple search (ie, OBJECT scope) and no entry in the
base. The test does not go through any network (though no BER
encoding/decoding). It just check the speed of the internal server.

Also it's interesting as it prove there is a wide margin of improvement in
the server itself.

>From the last test I did, I think we can gain 30 to 40% more by using a DN
cache (DN parsing and normalization count for 26% of the whole CPU), and we
have a spurious call to the hasEntry() method which costs 13,5% of the CPU.
The log system costs 2.75%, something we can also improve.

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Some more speed improvement for basic operations

[Emmanuel Lecharny <el...@gmail.com>]

On 6/8/10 1:18 AM, Quanah Gibson-Mount wrote:
> --On Tuesday, June 08, 2010 1:01 AM +0200 Emmanuel Lecharny 
> <el...@gmail.com> wrote:
>
>> Hi guys,
>>
>> today I finished to clean the rename() operation, and I just have the
>> three last operations remaining to be clean, ie move, moveAndRename and
>> compare.
>>
>> In the meantime, I was able to avoid an Entry clone to be done, and this
>> has a huge impact on some operations. Here are the result I get with or
>> without the clone :
>>
>> add    : 578/s(with) / 607/s (without) / +5%
>> lookup : 19 568/s(with) / 26 542/s (without) / +36%
>> search : 19 727/s(with) / 19 560/s (without) / ---
>> modify : 1 991/s(with) / 2103/s (without) / +5%
>> delete : 248/s(with) / 248/s (without) / ---
>>
>> As we can see, the lookup is really faster with such a modif. The other
>> operations aren't that much impacted, the cost of writing on disk kills
>> the gain we could have.
>>
>> One more thing : this is a test done with one single thread, directly on
>> top of the core-session.
>>
>> However, it demonstrates that with enough cache, and a good network
>> layer, we should be able to get some good performances out of the 
>> server.
>
> Exciting. :)  One of these days soon, I'll have a slamd perf lab again 
> (HW exists, waiting on installation), so I can draw up some numbers 
> between OpenLDAP, ApacheDS, and maybe some others.

We still have a long way to go to get numbers that compares to what we 
can get with OpenLDAP :)

However, we know where to squeeze performance out of the parts that need 
it :
- the network ayer that may be sub-optimal at this point,
- definitively the backend, as we have some serious contention when 
working with more than one client.

Considering using the server embedded, ie without dealing with the 
network layer, I must say I like the performance we now get :)

However, we are still cleaning the house at the moment, considering that 
having a stable, robust and compliant server is more important than 
having good performances. Right now, the fnny part of the on going work 
is that while working on making some operation atomic (we had some issue 
on ModDN operations), we get a side benefit of having better performance 
by removing useless processing.

Thanks Quanah !

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.nextury.com

Re: Some more speed improvement for basic operations

[Quanah Gibson-Mount <qu...@zimbra.com>]

--On Tuesday, June 08, 2010 1:01 AM +0200 Emmanuel Lecharny 
<el...@gmail.com> wrote:

> Hi guys,
>
> today I finished to clean the rename() operation, and I just have the
> three last operations remaining to be clean, ie move, moveAndRename and
> compare.
>
> In the meantime, I was able to avoid an Entry clone to be done, and this
> has a huge impact on some operations. Here are the result I get with or
> without the clone :
>
> add    : 578/s(with) / 607/s (without) / +5%
> lookup : 19 568/s(with) / 26 542/s (without) / +36%
> search : 19 727/s(with) / 19 560/s (without) / ---
> modify : 1 991/s(with) / 2103/s (without) / +5%
> delete : 248/s(with) / 248/s (without) / ---
>
> As we can see, the lookup is really faster with such a modif. The other
> operations aren't that much impacted, the cost of writing on disk kills
> the gain we could have.
>
> One more thing : this is a test done with one single thread, directly on
> top of the core-session.
>
> However, it demonstrates that with enough cache, and a good network
> layer, we should be able to get some good performances out of the server.

Exciting. :)  One of these days soon, I'll have a slamd perf lab again (HW 
exists, waiting on installation), so I can draw up some numbers between 
OpenLDAP, ApacheDS, and maybe some others.

--Quanah


--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra ::  the leader in open source messaging and collaboration