ApacheDS possible improvements

[Emmanuel Lécharny <el...@gmail.com>]

Hi guys,

now that I'm done with this damn Mavibot bug, and as I conducted some
profiling sessions to see how good the server is going with Mavibot, I
can provide a list of possible improvements we can make in the server.
Not all of them are urgent nor needed, but it's always good to know what
we can do better. here is the list :

- check the findPos calls to limit the number of time we use them
- don't store the UP value as a String, keep the Up byte[] (this saves 2
x String -> UTF-8 and UTF-8 -> String)
- use hash of Keys, instead of plain keys, for indexes
- review the way we return entries, by avoiding a clone for each entry
(use a proxy entry for that). Removed Attribute are just marked as removed,
we add new attributes into a special set of added Attributes. When a
value is removed (Authz), we mark the value as deleted in the proxy
attribute
- remove the locks and use only Mavibot with revisions
- MINA 3 should be used
- use the messageSent event to send the next entry
- don't allocate a new ByteBuffer for each entry to be sent, use a
Thread Local storage to store a big direct buffer to be used during the
encoding.
If the ByteBuffer is not big enough, catch the exception and compute the
length. We save the length computation, which is a costly operation.
- use a cache for aliases (implemented)
- improve the Mavibot BTree cache : we currently cache only entries, not
indexes
- we should not drop/add elements when we just want to update them (this
is typically what we do with the RdnIndex)

I can't tell how much each of those improvements can bring, but there is
at least a few that are critical. For instance, storing the UP values as
byte[] without converting them to String and back to byte[] when sending
them will save a lot of String and byte[] creation and String ->
byte[]/byte[] -> String conversions. The replace() method in index is
also extremelly important (we call drop/add as many times as we have
RDNs in an entry).

But the most important one would be the clone removal. It eats around
25% of the search time.

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com 

Re: ApacheDS possible improvements

[Kiran Ayyagari <ka...@apache.org>]

On Tue, Nov 12, 2013 at 4:04 PM, Emmanuel Lécharny <el...@gmail.com>wrote:

> Hi guys,
>
> now that I'm done with this damn Mavibot bug, and as I conducted some
> profiling sessions to see how good the server is going with Mavibot, I
> can provide a list of possible improvements we can make in the server.
> Not all of them are urgent nor needed, but it's always good to know what
> we can do better. here is the list :
>
> - check the findPos calls to limit the number of time we use them
> - don't store the UP value as a String, keep the Up byte[] (this saves 2
> x String -> UTF-8 and UTF-8 -> String)
> - use hash of Keys, instead of plain keys, for indexes
> - review the way we return entries, by avoiding a clone for each entry
> (use a proxy entry for that). Removed Attribute are just marked as removed,
> we add new attributes into a special set of added Attributes. When a
> value is removed (Authz), we mark the value as deleted in the proxy
> attribute
> - remove the locks and use only Mavibot with revisions
> - MINA 3 should be used
> - use the messageSent event to send the next entry
> - don't allocate a new ByteBuffer for each entry to be sent, use a
> Thread Local storage to store a big direct buffer to be used during the
> encoding.
> If the ByteBuffer is not big enough, catch the exception and compute the
> length. We save the length computation, which is a costly operation.
> - use a cache for aliases (implemented)
> - improve the Mavibot BTree cache : we currently cache only entries, not
> indexes
>
yeah, no cache is used for any of the indices atm, this is worth trying out
immediately, let us try this next week

> - we should not drop/add elements when we just want to update them (this
> is typically what we do with the RdnIndex)
>
> I can't tell how much each of those improvements can bring, but there is
> at least a few that are critical. For instance, storing the UP values as
> byte[] without converting them to String and back to byte[] when sending
> them will save a lot of String and byte[] creation and String ->
> byte[]/byte[] -> String conversions. The replace() method in index is
> also extremelly important (we call drop/add as many times as we have
> RDNs in an entry).
>
> But the most important one would be the clone removal. It eats around
> 25% of the search time.
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>


-- 
Kiran Ayyagari
http://keydap.com