You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Emmanuel Lécharny <el...@gmail.com> on 2012/05/01 03:08:37 UTC
Inex branch merged into trunk...
Hi,
just to inform you that the index branch has been merged with no harm
today. I just had to fix 3 conflicts, and two bugs I introduced in the
branch before the commit.
The server performance is way better for searches, with a few
improvements I did those last 4 days. It was impressive how easy it was
to improve the speed with little modifications. The global result is
that the server is now :
o Object scope search (lookup) : 49 880 req/s compared to 23 081 on the
previous trunk
o One Level scope search (5 entries returned) : 68 715 entries returned
per second, compared to 33 120/s
o Sub Level scope search (10 entries returned ) : 70 830 entries
returned per second, compared to 18 910/s
There is room for more improvement, but it will be more complex. The
area that can be improved are :
o get rid of the extra getSearchControls() call in intercepotrs. This is
the easiest fix
o review the way we handle entries modification before we return them.
Currently, we clone the entry, and remove the attributes the user has
not required. See DIRSERVER-1719 for more explaination on this subject.
Note that the filtering of attributes represent around 9% of the global
CPU time.
o getting back the ID from a Dn is a very costly operation (19% of the
global CPU time), and the longer the DN, the longer the operation. For
each RDN, we have to do a lookup in the RdnIndex. The only solution
would be to have a Dn -> ID cache somewhere. This would boost the server
performance, that's for sure.
o fetching an entry from the backend cost 38% of the global time, out of
which 29% represent the cost to clone the entry. If we could avoid doing
this clone (see upper), we may have some major performances increase.
o when evaluating an entry to see if it fits the filter, we use the
reverseIndex, which is also a costly operation. We shoudl re-evaluate if
it wouldn't be better to use the MatchingRules comparator to do that
instead (reverse lookups account for 4% of the used CPU time)
One interesting result is that the LRUCache.get() operation represent
13% of the used time. This is definitively not small. There is probably
some room for some improvement here, but this is way more complex...
All those numbers have been collected using YourKit on a Lookup test
(150 000 lookups on one single element have been done)
There are also some improvements to expect on the Add/Delete/Move
operation, as we have to delete/add the keys on the RdnIndex. This is
something Im going to work on tomorrow.
One more thing : the number I get when running the server-integ search
perf are way below (from 2900 to 5400 per second). This is plain normal.
When going through th network, we pay some extra price :
o the client code eats 57% of all the time it takes to run the test
o On the server, normalizing the incoming Dn costs 7% of the processing time
o the entries encoding is very expensive
All in all, on the server, unless we test it on a different machine than
the injectors, all the measures are pretty impossible to do. There is
too much noise...
I'd be interested to conduct largest tests on a multi-core server, with
lots of memory, and a lot of entries, with external injectors, to see
what kind of performances we can get...
In the next few days, I will probably fix some pending bugs. I think we
can cut a M7 release by the end of this week, and make it available by
next week.
Thanks !
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Inex branch merged into trunk...
Posted by Emmanuel Lécharny <el...@gmail.com>.
Hi,
for the fun, I did a perf test against 1.5.0 to see how this 5 years old
version was holding the load...
o Object scope search (lookup) : 32 993/s
o One Level scope search (5 entries returned) : 156 935/s
o Sub Level scope search (11 entries returned ) : 137 225/s
So the latest version is faster when it comes to process the initial search request, but way slower when it comes to deliver entries...
There are reasons why 2.0.0 trunk is slower :
- we don't store the name in the Entry, so we have to rebuild it, which is a costly operation
- in 1.5.0, we don't clone the entry, we create a new one, and we inject into it the ATs which have been selected... Sounds like what I'm proposing to do :)
- 1.5.0 was not schema aware, so a lot less checks are done
- the backend is way mor ecomplex in 2.0.0
I'm confident we can double the current performances on 2.0.0, it's just a matter to do that in the right timing...
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Inex branch merged into trunk...
Posted by Alex Karasulu <ak...@apache.org>.
On Wed, May 2, 2012 at 4:44 PM, Emmanuel Lécharny <el...@gmail.com>wrote:
> Le 5/2/12 3:21 PM, Alex Karasulu a écrit :
>
>> On Wed, May 2, 2012 at 1:54 PM, Emmanuel Lécharny<el...@gmail.com>**
>> wrote:
>>
>> Le 5/2/12 12:33 PM, Alex Karasulu a écrit :
>>>
>>> On Wed, May 2, 2012 at 12:49 PM, Emmanuel Lécharny<elecharny@gmail.com
>>>> >****
>>>>
>>>> wrote:
>>>>
>>>> Le 5/2/12 9:53 AM, Alex Karasulu a écrit :
>>>>
>>>>> On Wed, May 2, 2012 at 2:43 AM, Emmanuel Lécharny<elecharny@gmail.com
>>>>> >*
>>>>>
>>>>>> ***
>>>>>>
>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>> Le 5/1/12 3:05 PM, Alex Karasulu a écrit :
>>>>>>
>>>>>> On Tue, May 1, 2012 at 4:08 AM, Emmanuel Lécharny<
>>>>>>> elecharny@gmail.com
>>>>>>>
>>>>>>>> *
>>>>>>>> ***
>>>>>>>>
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> o Object scope search (lookup) : 49 880 req/s compared to 23 081 on
>>>>>>>> the
>>>>>>>> previous trunk
>>>>>>>> o One Level scope search (5 entries returned) : 68 715 entries
>>>>>>>> returned
>>>>>>>> per second, compared to 33 120/s
>>>>>>>> o Sub Level scope search (10 entries returned ) : 70 830 entries
>>>>>>>> returned
>>>>>>>> per second, compared to 18 910/s
>>>>>>>>
>>>>>>>>
>>>>>>>> This is great work Emmanuel. Nicely done!
>>>>>>>>
>>>>>>>> I have some even better results, as of today :
>>>>>>>>
>>>>>>>> o Object scope search (lookup) : 52 712 req/s compared to 23 081
>>>>>>> on the
>>>>>>> previous trunk
>>>>>>> o One Level scope search (5 entries returned) : 72 635 entries
>>>>>>> returned
>>>>>>> per second, compared to 33 120/s
>>>>>>> o Sub Level scope search (10 entries returned ) : 75 100 entries
>>>>>>> returned
>>>>>>> per second, compared to 18 910/s
>>>>>>>
>>>>>>>
>>>>>>> This is just sick man! You've more than doubled the performance.
>>>>>>>
>>>>>>> Some new idea this morning :
>>>>>>
>>>>> atm, we do clone the entries we fetch from the server, then we filter
>>>>> the
>>>>> Attributes and the values, modifying the cloned entries. This leads to
>>>>> useless create of the removed Attributes and Values. We suggested to
>>>>> accumulate the modifications and to apply them at the end, avoiding the
>>>>> cloning of AT which will not be returned.
>>>>>
>>>>> First of all, we can avoid cloning the Values. The Value implementation
>>>>> are immutable classes. This save around 7% of the time.
>>>>>
>>>>> But this is not all we can do : we can simply avoid the accumulation of
>>>>> modifications *and* avoid cloning the entry !
>>>>>
>>>>> The idea is simple : when we get an entry in the cursor we have got
>>>>> back,
>>>>> we create a new empty entry, then we iterate over all the original
>>>>> entry's
>>>>> attributes and values, and for each one of those attributes and values,
>>>>> we
>>>>> check the filters, which will just tell if the Attribute/Value must be
>>>>> ditched or kept. This way, we don't do anything useless, like storing
>>>>> the
>>>>> modification or creating useless Attributes.
>>>>>
>>>>> It will work to the extent we deal with the CollectiveAttributes which
>>>>> must be injected somewhere, before we enter the loop (a
>>>>> connectiveAttribute
>>>>> might perfectly be removed by the ACI filter...). But we can also
>>>>> inject
>>>>> those added collective attributes into the loop of filters.
>>>>>
>>>>> I may miss something, but I do think that this solution is a clear
>>>>> winner,
>>>>> even in term of implementation...
>>>>>
>>>>> thoughts ?
>>>>>
>>>>>
>>>>> We talked about using a wrapper around the entry to encapsulate these
>>>>>
>>>> matters making it happen automatically behind the scenes. This does not
>>>> affect the surrounding code.
>>>>
>>>> How is this proposal now different? Why would you not use a wrapper?
>>>>
>>>> Because the wrapper is useless in this case !
>>>
>>> The beauty of the solution is that we either create a new entry with all
>>> the requested AT and values, accordingly to the filters (if the user
>>> embeds
>>> the server), or if this is a network request, we directly generates the
>>> encoded message without having to create the intermediate entry at all !
>>>
>>>
>>> I don't know how you'll pull that off considering that the interceptors
>> which cause side effects are expecting an entry to alter or from which to
>> read information from to do their thang.
>>
>> This is why I'm a bit confused. Maybe its a matter of description and the
>> language where I'm failing to understand.
>>
> Probably...
>
> Just forgot to mention that the filters have to be modified : they won't
> take an Entry as a parameter, but an Attribute, and the accept( Attribute )
> method will just return something telling if the Attribute should be
> removed or not. If at least one filter says 'remove this atttribute', then
> it will not be added to the resulting entry.
>
>
These are the entry filters used by interceptors I gather, right? Could you
provide a link to the code where you propose these changes?
> For values (ie, the ACI filter), as we will proceed with some fine
> granilarity (ie values), we can consider that the passed Attribute will be
> cloned and modified internally, so we can fill the entry with the correct
> Attribute (or the original one)
>
> For the CollectiveAttribute attribute, the filter will require the full
> entry to be able to return the added attributes, but the mechanism is a bit
> different there. I was thinking that we could also get all the added
> ConnectiveAttributes, and inject them in the other filters.
>
>
Have no idea what ConnectiveAttributes are ... guess you must mean
CollectiveAttribute?
> At this point, it's more ideas than implementation, but I definitively see
> how it could work well.
>
>
Understood. Experimentation might help a lot more so than discussions but I
think we should tread carefully here (not that you are not careful). I
think we can be surprised by the number of painful side effects this change
for a 10-15% optimization could produce.
Generally I get really nervous when you try to optimize. I then see your
results then I'm super happy. It's like riding on the back of Evel
Knievel's motorcycle as he jumps over 10 busses. You shit yourself before
and during but then when you make it to the other side you feel like, f**k
that was awesome ... let's do it again.
> Btw, some more perfs have been added into the server :
>
> o Object scope search (lookup) : 57 191/s compared to 23 081 on the
> previous trunk (148% more)
>
That looks like 248% ---- Ahh you say 148% more meaning 100% + 148% = 248%.
> o One Level scope search (5 entries returned) : 76 250 entries returned
> per second, compared to 33 120/s (130% more)
> o Sub Level scope search (10 entries returned ) : 77 200/s entries
> returned per second, compared to 18 910/s (308% more)
>
>
See this where I say AWESOME!
--
Best Regards,
-- Alex
Re: Inex branch merged into trunk...
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 5/2/12 3:21 PM, Alex Karasulu a écrit :
> On Wed, May 2, 2012 at 1:54 PM, Emmanuel Lécharny<el...@gmail.com>wrote:
>
>> Le 5/2/12 12:33 PM, Alex Karasulu a écrit :
>>
>>> On Wed, May 2, 2012 at 12:49 PM, Emmanuel Lécharny<el...@gmail.com>**
>>> wrote:
>>>
>>> Le 5/2/12 9:53 AM, Alex Karasulu a écrit :
>>>> On Wed, May 2, 2012 at 2:43 AM, Emmanuel Lécharny<el...@gmail.com>*
>>>>> ***
>>>>>
>>>>> wrote:
>>>>>
>>>>> Le 5/1/12 3:05 PM, Alex Karasulu a écrit :
>>>>>
>>>>>> On Tue, May 1, 2012 at 4:08 AM, Emmanuel Lécharny<elecharny@gmail.com
>>>>>>> *
>>>>>>> ***
>>>>>>>
>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>> o Object scope search (lookup) : 49 880 req/s compared to 23 081 on
>>>>>>> the
>>>>>>> previous trunk
>>>>>>> o One Level scope search (5 entries returned) : 68 715 entries
>>>>>>> returned
>>>>>>> per second, compared to 33 120/s
>>>>>>> o Sub Level scope search (10 entries returned ) : 70 830 entries
>>>>>>> returned
>>>>>>> per second, compared to 18 910/s
>>>>>>>
>>>>>>>
>>>>>>> This is great work Emmanuel. Nicely done!
>>>>>>>
>>>>>>> I have some even better results, as of today :
>>>>>>>
>>>>>> o Object scope search (lookup) : 52 712 req/s compared to 23 081 on the
>>>>>> previous trunk
>>>>>> o One Level scope search (5 entries returned) : 72 635 entries returned
>>>>>> per second, compared to 33 120/s
>>>>>> o Sub Level scope search (10 entries returned ) : 75 100 entries
>>>>>> returned
>>>>>> per second, compared to 18 910/s
>>>>>>
>>>>>>
>>>>>> This is just sick man! You've more than doubled the performance.
>>>>>>
>>>>> Some new idea this morning :
>>>> atm, we do clone the entries we fetch from the server, then we filter the
>>>> Attributes and the values, modifying the cloned entries. This leads to
>>>> useless create of the removed Attributes and Values. We suggested to
>>>> accumulate the modifications and to apply them at the end, avoiding the
>>>> cloning of AT which will not be returned.
>>>>
>>>> First of all, we can avoid cloning the Values. The Value implementation
>>>> are immutable classes. This save around 7% of the time.
>>>>
>>>> But this is not all we can do : we can simply avoid the accumulation of
>>>> modifications *and* avoid cloning the entry !
>>>>
>>>> The idea is simple : when we get an entry in the cursor we have got back,
>>>> we create a new empty entry, then we iterate over all the original
>>>> entry's
>>>> attributes and values, and for each one of those attributes and values,
>>>> we
>>>> check the filters, which will just tell if the Attribute/Value must be
>>>> ditched or kept. This way, we don't do anything useless, like storing the
>>>> modification or creating useless Attributes.
>>>>
>>>> It will work to the extent we deal with the CollectiveAttributes which
>>>> must be injected somewhere, before we enter the loop (a
>>>> connectiveAttribute
>>>> might perfectly be removed by the ACI filter...). But we can also inject
>>>> those added collective attributes into the loop of filters.
>>>>
>>>> I may miss something, but I do think that this solution is a clear
>>>> winner,
>>>> even in term of implementation...
>>>>
>>>> thoughts ?
>>>>
>>>>
>>>> We talked about using a wrapper around the entry to encapsulate these
>>> matters making it happen automatically behind the scenes. This does not
>>> affect the surrounding code.
>>>
>>> How is this proposal now different? Why would you not use a wrapper?
>>>
>> Because the wrapper is useless in this case !
>>
>> The beauty of the solution is that we either create a new entry with all
>> the requested AT and values, accordingly to the filters (if the user embeds
>> the server), or if this is a network request, we directly generates the
>> encoded message without having to create the intermediate entry at all !
>>
>>
> I don't know how you'll pull that off considering that the interceptors
> which cause side effects are expecting an entry to alter or from which to
> read information from to do their thang.
>
> This is why I'm a bit confused. Maybe its a matter of description and the
> language where I'm failing to understand.
Probably...
Just forgot to mention that the filters have to be modified : they won't
take an Entry as a parameter, but an Attribute, and the accept(
Attribute ) method will just return something telling if the Attribute
should be removed or not. If at least one filter says 'remove this
atttribute', then it will not be added to the resulting entry.
For values (ie, the ACI filter), as we will proceed with some fine
granilarity (ie values), we can consider that the passed Attribute will
be cloned and modified internally, so we can fill the entry with the
correct Attribute (or the original one)
For the CollectiveAttribute attribute, the filter will require the full
entry to be able to return the added attributes, but the mechanism is a
bit different there. I was thinking that we could also get all the added
ConnectiveAttributes, and inject them in the other filters.
At this point, it's more ideas than implementation, but I definitively
see how it could work well.
Btw, some more perfs have been added into the server :
o Object scope search (lookup) : 57 191/s compared to 23 081 on the previous trunk (148% more)
o One Level scope search (5 entries returned) : 76 250 entries returned per second, compared to 33 120/s (130% more)
o Sub Level scope search (10 entries returned ) : 77 200/s entries returned per second, compared to 18 910/s (308% more)
The values are not any more cloned (the Value classes are immutable)
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Inex branch merged into trunk...
Posted by Alex Karasulu <ak...@apache.org>.
On Wed, May 2, 2012 at 1:54 PM, Emmanuel Lécharny <el...@gmail.com>wrote:
> Le 5/2/12 12:33 PM, Alex Karasulu a écrit :
>
>> On Wed, May 2, 2012 at 12:49 PM, Emmanuel Lécharny<el...@gmail.com>**
>> wrote:
>>
>> Le 5/2/12 9:53 AM, Alex Karasulu a écrit :
>>>
>>> On Wed, May 2, 2012 at 2:43 AM, Emmanuel Lécharny<el...@gmail.com>*
>>>> ***
>>>>
>>>> wrote:
>>>>
>>>> Le 5/1/12 3:05 PM, Alex Karasulu a écrit :
>>>>
>>>>> On Tue, May 1, 2012 at 4:08 AM, Emmanuel Lécharny<elecharny@gmail.com
>>>>> >*
>>>>>
>>>>>> ***
>>>>>>
>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>> o Object scope search (lookup) : 49 880 req/s compared to 23 081 on
>>>>>> the
>>>>>> previous trunk
>>>>>> o One Level scope search (5 entries returned) : 68 715 entries
>>>>>> returned
>>>>>> per second, compared to 33 120/s
>>>>>> o Sub Level scope search (10 entries returned ) : 70 830 entries
>>>>>> returned
>>>>>> per second, compared to 18 910/s
>>>>>>
>>>>>>
>>>>>> This is great work Emmanuel. Nicely done!
>>>>>>
>>>>>> I have some even better results, as of today :
>>>>>>
>>>>> o Object scope search (lookup) : 52 712 req/s compared to 23 081 on the
>>>>> previous trunk
>>>>> o One Level scope search (5 entries returned) : 72 635 entries returned
>>>>> per second, compared to 33 120/s
>>>>> o Sub Level scope search (10 entries returned ) : 75 100 entries
>>>>> returned
>>>>> per second, compared to 18 910/s
>>>>>
>>>>>
>>>>> This is just sick man! You've more than doubled the performance.
>>>>>
>>>> Some new idea this morning :
>>>
>>> atm, we do clone the entries we fetch from the server, then we filter the
>>> Attributes and the values, modifying the cloned entries. This leads to
>>> useless create of the removed Attributes and Values. We suggested to
>>> accumulate the modifications and to apply them at the end, avoiding the
>>> cloning of AT which will not be returned.
>>>
>>> First of all, we can avoid cloning the Values. The Value implementation
>>> are immutable classes. This save around 7% of the time.
>>>
>>> But this is not all we can do : we can simply avoid the accumulation of
>>> modifications *and* avoid cloning the entry !
>>>
>>> The idea is simple : when we get an entry in the cursor we have got back,
>>> we create a new empty entry, then we iterate over all the original
>>> entry's
>>> attributes and values, and for each one of those attributes and values,
>>> we
>>> check the filters, which will just tell if the Attribute/Value must be
>>> ditched or kept. This way, we don't do anything useless, like storing the
>>> modification or creating useless Attributes.
>>>
>>> It will work to the extent we deal with the CollectiveAttributes which
>>> must be injected somewhere, before we enter the loop (a
>>> connectiveAttribute
>>> might perfectly be removed by the ACI filter...). But we can also inject
>>> those added collective attributes into the loop of filters.
>>>
>>> I may miss something, but I do think that this solution is a clear
>>> winner,
>>> even in term of implementation...
>>>
>>> thoughts ?
>>>
>>>
>>> We talked about using a wrapper around the entry to encapsulate these
>> matters making it happen automatically behind the scenes. This does not
>> affect the surrounding code.
>>
>> How is this proposal now different? Why would you not use a wrapper?
>>
> Because the wrapper is useless in this case !
>
> The beauty of the solution is that we either create a new entry with all
> the requested AT and values, accordingly to the filters (if the user embeds
> the server), or if this is a network request, we directly generates the
> encoded message without having to create the intermediate entry at all !
>
>
I don't know how you'll pull that off considering that the interceptors
which cause side effects are expecting an entry to alter or from which to
read information from to do their thang.
This is why I'm a bit confused. Maybe its a matter of description and the
language where I'm failing to understand.
--
Best Regards,
-- Alex
Re: Inex branch merged into trunk...
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 5/2/12 12:33 PM, Alex Karasulu a écrit :
> On Wed, May 2, 2012 at 12:49 PM, Emmanuel Lécharny<el...@gmail.com>wrote:
>
>> Le 5/2/12 9:53 AM, Alex Karasulu a écrit :
>>
>>> On Wed, May 2, 2012 at 2:43 AM, Emmanuel Lécharny<el...@gmail.com>**
>>> wrote:
>>>
>>> Le 5/1/12 3:05 PM, Alex Karasulu a écrit :
>>>> On Tue, May 1, 2012 at 4:08 AM, Emmanuel Lécharny<el...@gmail.com>*
>>>>> ***
>>>>>
>>>>> wrote:
>>>>>
>>>>> o Object scope search (lookup) : 49 880 req/s compared to 23 081 on the
>>>>> previous trunk
>>>>> o One Level scope search (5 entries returned) : 68 715 entries returned
>>>>> per second, compared to 33 120/s
>>>>> o Sub Level scope search (10 entries returned ) : 70 830 entries
>>>>> returned
>>>>> per second, compared to 18 910/s
>>>>>
>>>>>
>>>>> This is great work Emmanuel. Nicely done!
>>>>>
>>>>> I have some even better results, as of today :
>>>> o Object scope search (lookup) : 52 712 req/s compared to 23 081 on the
>>>> previous trunk
>>>> o One Level scope search (5 entries returned) : 72 635 entries returned
>>>> per second, compared to 33 120/s
>>>> o Sub Level scope search (10 entries returned ) : 75 100 entries returned
>>>> per second, compared to 18 910/s
>>>>
>>>>
>>>> This is just sick man! You've more than doubled the performance.
>> Some new idea this morning :
>>
>> atm, we do clone the entries we fetch from the server, then we filter the
>> Attributes and the values, modifying the cloned entries. This leads to
>> useless create of the removed Attributes and Values. We suggested to
>> accumulate the modifications and to apply them at the end, avoiding the
>> cloning of AT which will not be returned.
>>
>> First of all, we can avoid cloning the Values. The Value implementation
>> are immutable classes. This save around 7% of the time.
>>
>> But this is not all we can do : we can simply avoid the accumulation of
>> modifications *and* avoid cloning the entry !
>>
>> The idea is simple : when we get an entry in the cursor we have got back,
>> we create a new empty entry, then we iterate over all the original entry's
>> attributes and values, and for each one of those attributes and values, we
>> check the filters, which will just tell if the Attribute/Value must be
>> ditched or kept. This way, we don't do anything useless, like storing the
>> modification or creating useless Attributes.
>>
>> It will work to the extent we deal with the CollectiveAttributes which
>> must be injected somewhere, before we enter the loop (a connectiveAttribute
>> might perfectly be removed by the ACI filter...). But we can also inject
>> those added collective attributes into the loop of filters.
>>
>> I may miss something, but I do think that this solution is a clear winner,
>> even in term of implementation...
>>
>> thoughts ?
>>
>>
> We talked about using a wrapper around the entry to encapsulate these
> matters making it happen automatically behind the scenes. This does not
> affect the surrounding code.
>
> How is this proposal now different? Why would you not use a wrapper?
Because the wrapper is useless in this case !
The beauty of the solution is that we either create a new entry with all
the requested AT and values, accordingly to the filters (if the user
embeds the server), or if this is a network request, we directly
generates the encoded message without having to create the intermediate
entry at all !
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Inex branch merged into trunk...
Posted by Alex Karasulu <ak...@apache.org>.
On Wed, May 2, 2012 at 12:49 PM, Emmanuel Lécharny <el...@gmail.com>wrote:
> Le 5/2/12 9:53 AM, Alex Karasulu a écrit :
>
>> On Wed, May 2, 2012 at 2:43 AM, Emmanuel Lécharny<el...@gmail.com>**
>> wrote:
>>
>> Le 5/1/12 3:05 PM, Alex Karasulu a écrit :
>>>
>>> On Tue, May 1, 2012 at 4:08 AM, Emmanuel Lécharny<el...@gmail.com>*
>>>> ***
>>>>
>>>> wrote:
>>>>
>>>> o Object scope search (lookup) : 49 880 req/s compared to 23 081 on the
>>>> previous trunk
>>>> o One Level scope search (5 entries returned) : 68 715 entries returned
>>>> per second, compared to 33 120/s
>>>> o Sub Level scope search (10 entries returned ) : 70 830 entries
>>>> returned
>>>> per second, compared to 18 910/s
>>>>
>>>>
>>>> This is great work Emmanuel. Nicely done!
>>>>
>>>> I have some even better results, as of today :
>>>
>>> o Object scope search (lookup) : 52 712 req/s compared to 23 081 on the
>>> previous trunk
>>> o One Level scope search (5 entries returned) : 72 635 entries returned
>>> per second, compared to 33 120/s
>>> o Sub Level scope search (10 entries returned ) : 75 100 entries returned
>>> per second, compared to 18 910/s
>>>
>>>
>>> This is just sick man! You've more than doubled the performance.
>>
>
> Some new idea this morning :
>
> atm, we do clone the entries we fetch from the server, then we filter the
> Attributes and the values, modifying the cloned entries. This leads to
> useless create of the removed Attributes and Values. We suggested to
> accumulate the modifications and to apply them at the end, avoiding the
> cloning of AT which will not be returned.
>
> First of all, we can avoid cloning the Values. The Value implementation
> are immutable classes. This save around 7% of the time.
>
> But this is not all we can do : we can simply avoid the accumulation of
> modifications *and* avoid cloning the entry !
>
> The idea is simple : when we get an entry in the cursor we have got back,
> we create a new empty entry, then we iterate over all the original entry's
> attributes and values, and for each one of those attributes and values, we
> check the filters, which will just tell if the Attribute/Value must be
> ditched or kept. This way, we don't do anything useless, like storing the
> modification or creating useless Attributes.
>
> It will work to the extent we deal with the CollectiveAttributes which
> must be injected somewhere, before we enter the loop (a connectiveAttribute
> might perfectly be removed by the ACI filter...). But we can also inject
> those added collective attributes into the loop of filters.
>
> I may miss something, but I do think that this solution is a clear winner,
> even in term of implementation...
>
> thoughts ?
>
>
We talked about using a wrapper around the entry to encapsulate these
matters making it happen automatically behind the scenes. This does not
affect the surrounding code.
How is this proposal now different? Why would you not use a wrapper?
--
Best Regards,
-- Alex
Re: Inex branch merged into trunk...
Posted by Pierre-Arnaud Marcelot <pa...@marcelot.net>.
On 2 mai 2012, at 11:49, Emmanuel Lécharny wrote:
> Le 5/2/12 9:53 AM, Alex Karasulu a écrit :
>> On Wed, May 2, 2012 at 2:43 AM, Emmanuel Lécharny<el...@gmail.com>wrote:
>>
>>> Le 5/1/12 3:05 PM, Alex Karasulu a écrit :
>>>
>>>> On Tue, May 1, 2012 at 4:08 AM, Emmanuel Lécharny<el...@gmail.com>**
>>>> wrote:
>>>>
>>>> o Object scope search (lookup) : 49 880 req/s compared to 23 081 on the
>>>> previous trunk
>>>> o One Level scope search (5 entries returned) : 68 715 entries returned
>>>> per second, compared to 33 120/s
>>>> o Sub Level scope search (10 entries returned ) : 70 830 entries returned
>>>> per second, compared to 18 910/s
>>>>
>>>>
>>>> This is great work Emmanuel. Nicely done!
>>>>
>>> I have some even better results, as of today :
>>>
>>> o Object scope search (lookup) : 52 712 req/s compared to 23 081 on the
>>> previous trunk
>>> o One Level scope search (5 entries returned) : 72 635 entries returned
>>> per second, compared to 33 120/s
>>> o Sub Level scope search (10 entries returned ) : 75 100 entries returned
>>> per second, compared to 18 910/s
>>>
>>>
>> This is just sick man! You've more than doubled the performance.
>
> Some new idea this morning :
>
> atm, we do clone the entries we fetch from the server, then we filter the Attributes and the values, modifying the cloned entries. This leads to useless create of the removed Attributes and Values. We suggested to accumulate the modifications and to apply them at the end, avoiding the cloning of AT which will not be returned.
>
> First of all, we can avoid cloning the Values. The Value implementation are immutable classes. This save around 7% of the time.
>
> But this is not all we can do : we can simply avoid the accumulation of modifications *and* avoid cloning the entry !
>
> The idea is simple : when we get an entry in the cursor we have got back, we create a new empty entry, then we iterate over all the original entry's attributes and values, and for each one of those attributes and values, we check the filters, which will just tell if the Attribute/Value must be ditched or kept. This way, we don't do anything useless, like storing the modification or creating useless Attributes.
>
> It will work to the extent we deal with the CollectiveAttributes which must be injected somewhere, before we enter the loop (a connectiveAttribute might perfectly be removed by the ACI filter...). But we can also inject those added collective attributes into the loop of filters.
>
> I may miss something, but I do think that this solution is a clear winner, even in term of implementation...
>
> thoughts ?
Since, cloning the entire entry (including all attributes and values) is awfully costly.
I think it makes much sense, indeed, to start from an empty entry and add the appropriate attributes and values, rather than cloning the whole entry and then get rid of almost everything to return only a sub-set of the attributes (sometimes none).
It seems much more optimized this way.
I'd say it deserves to be tested at least...
Regards,
Pierre-Arnaud
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
Re: Inex branch merged into trunk...
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 5/2/12 2:57 PM, Emmanuel Lécharny a écrit :
>
> This should defintively save a large amout of CPU !
I have done a small test, where I didn't copied the entry, but instead
created a new one and injected into it only the required AT. the gain is
significant ;
Object : 67 385/s against 57 191/s
OneLevel : 89 650/s against 76 250/s
SubLevel : 93 500/s against 77 200/s
Ok, those sounds cool, but the thing is that such a modification is
impacting the whole server, and even if it works for some tests, it
breaks many others. I must say I have been a bit brutal too ;)
But anyway, this is clearly the way to go.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Inex branch merged into trunk...
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 5/2/12 12:55 PM, Kiran Ayyagari a écrit :
> On Wed, May 2, 2012 at 4:20 PM, Pierre-Arnaud Marcelot<pa...@marcelot.net> wrote:
>> On 2 mai 2012, at 12:24, Kiran Ayyagari wrote:
>>> this method has two extreme cases,
>>> 1. best performance when no attributes are requested
>>> 2. same overhead as cloning when all attributes are requested
>>>
>>> cause in real world scenario 1 will never be used,
>> Actually in Studio, scenario 1 is used a lot.
>> That's exactly what happens do when opening a node and browsing the DIT in the UI (almost, because we do ask for the objectClass attribute).
>>
> yeap, I know, but this is a very limited usage, or shall I say just
> for tooling (essentially helpful for paging etc, else dOOM is certain
> the moment I expand a 100k node tree (provided my limit is set to>
> 100k ;))
Regarding a standard directory usage, I would say that even if most of
the users are doing simple searches requested all the user attributes to
be returned (regardless the netwrok traffic it generates), again, they
usually don't request operational attributes.
But if they do limit the number of attributes they want, then this makes
a huge difference from the server perspective not to clone the useless
attributes.
Otherwise, with the second solution I proposed (ie, not cloning nor
storing the removed ATs), we just evaluate on the fly the elements to
encode, without having to create an entry at all.
This should defintively save a large amout of CPU !
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Inex branch merged into trunk...
Posted by Pierre-Arnaud Marcelot <pa...@marcelot.net>.
On 2 mai 2012, at 12:55, Kiran Ayyagari wrote:
> On Wed, May 2, 2012 at 4:20 PM, Pierre-Arnaud Marcelot <pa...@marcelot.net> wrote:
>> On 2 mai 2012, at 12:24, Kiran Ayyagari wrote:
>>> this method has two extreme cases,
>>> 1. best performance when no attributes are requested
>>> 2. same overhead as cloning when all attributes are requested
>>>
>>> cause in real world scenario 1 will never be used,
>>
>> Actually in Studio, scenario 1 is used a lot.
>> That's exactly what happens do when opening a node and browsing the DIT in the UI (almost, because we do ask for the objectClass attribute).
>>
> yeap, I know, but this is a very limited usage, or shall I say just
> for tooling (essentially helpful for paging etc, else dOOM is certain
> the moment I expand a 100k node tree (provided my limit is set to >
> 100k ;))
Yeah, that's for sure.
We agree that directories are not meant to be browse by GUI as primary usage… ;)
Regards,
Pierre-Arnaud
>> Regards,
>> Pierre-Arnaud
>>
>>> we might end up
>>> copying some attributes anyway
>>> but still better than cloning all
>>> On Wed, May 2, 2012 at 3:19 PM, Emmanuel Lécharny <el...@gmail.com> wrote:
>>>> Le 5/2/12 9:53 AM, Alex Karasulu a écrit :
>>>>>
>>>>> On Wed, May 2, 2012 at 2:43 AM, Emmanuel
>>>>> Lécharny<el...@gmail.com>wrote:
>>>>>
>>>>>> Le 5/1/12 3:05 PM, Alex Karasulu a écrit :
>>>>>>
>>>>>>> On Tue, May 1, 2012 at 4:08 AM, Emmanuel Lécharny<el...@gmail.com>**
>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>> o Object scope search (lookup) : 49 880 req/s compared to 23 081 on the
>>>>>>> previous trunk
>>>>>>> o One Level scope search (5 entries returned) : 68 715 entries returned
>>>>>>> per second, compared to 33 120/s
>>>>>>> o Sub Level scope search (10 entries returned ) : 70 830 entries
>>>>>>> returned
>>>>>>> per second, compared to 18 910/s
>>>>>>>
>>>>>>>
>>>>>>> This is great work Emmanuel. Nicely done!
>>>>>>>
>>>>>> I have some even better results, as of today :
>>>>>>
>>>>>> o Object scope search (lookup) : 52 712 req/s compared to 23 081 on the
>>>>>> previous trunk
>>>>>> o One Level scope search (5 entries returned) : 72 635 entries returned
>>>>>> per second, compared to 33 120/s
>>>>>> o Sub Level scope search (10 entries returned ) : 75 100 entries returned
>>>>>> per second, compared to 18 910/s
>>>>>>
>>>>>>
>>>>> This is just sick man! You've more than doubled the performance.
>>>>
>>>>
>>>> Some new idea this morning :
>>>>
>>>> atm, we do clone the entries we fetch from the server, then we filter the
>>>> Attributes and the values, modifying the cloned entries. This leads to
>>>> useless create of the removed Attributes and Values. We suggested to
>>>> accumulate the modifications and to apply them at the end, avoiding the
>>>> cloning of AT which will not be returned.
>>>>
>>>> First of all, we can avoid cloning the Values. The Value implementation are
>>>> immutable classes. This save around 7% of the time.
>>>>
>>>> But this is not all we can do : we can simply avoid the accumulation of
>>>> modifications *and* avoid cloning the entry !
>>>>
>>>> The idea is simple : when we get an entry in the cursor we have got back, we
>>>> create a new empty entry, then we iterate over all the original entry's
>>>> attributes and values, and for each one of those attributes and values, we
>>>> check the filters, which will just tell if the Attribute/Value must be
>>>> ditched or kept. This way, we don't do anything useless, like storing the
>>>> modification or creating useless Attributes.
>>>>
>>>> It will work to the extent we deal with the CollectiveAttributes which must
>>>> be injected somewhere, before we enter the loop (a connectiveAttribute might
>>>> perfectly be removed by the ACI filter...). But we can also inject those
>>>> added collective attributes into the loop of filters.
>>>>
>>>> I may miss something, but I do think that this solution is a clear winner,
>>>> even in term of implementation...
>>>>
>>>> thoughts ?
>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Cordialement,
>>>> Emmanuel Lécharny
>>>> www.iktek.com
>>>>
>>>
>>>
>>>
>>> --
>>> Kiran Ayyagari
>>
>
>
>
> --
> Kiran Ayyagari
Re: Inex branch merged into trunk...
Posted by Kiran Ayyagari <ka...@apache.org>.
On Wed, May 2, 2012 at 4:20 PM, Pierre-Arnaud Marcelot <pa...@marcelot.net> wrote:
> On 2 mai 2012, at 12:24, Kiran Ayyagari wrote:
>> this method has two extreme cases,
>> 1. best performance when no attributes are requested
>> 2. same overhead as cloning when all attributes are requested
>>
>> cause in real world scenario 1 will never be used,
>
> Actually in Studio, scenario 1 is used a lot.
> That's exactly what happens do when opening a node and browsing the DIT in the UI (almost, because we do ask for the objectClass attribute).
>
yeap, I know, but this is a very limited usage, or shall I say just
for tooling (essentially helpful for paging etc, else dOOM is certain
the moment I expand a 100k node tree (provided my limit is set to >
100k ;))
> Regards,
> Pierre-Arnaud
>
>> we might end up
>> copying some attributes anyway
>> but still better than cloning all
>> On Wed, May 2, 2012 at 3:19 PM, Emmanuel Lécharny <el...@gmail.com> wrote:
>>> Le 5/2/12 9:53 AM, Alex Karasulu a écrit :
>>>>
>>>> On Wed, May 2, 2012 at 2:43 AM, Emmanuel
>>>> Lécharny<el...@gmail.com>wrote:
>>>>
>>>>> Le 5/1/12 3:05 PM, Alex Karasulu a écrit :
>>>>>
>>>>>> On Tue, May 1, 2012 at 4:08 AM, Emmanuel Lécharny<el...@gmail.com>**
>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>> o Object scope search (lookup) : 49 880 req/s compared to 23 081 on the
>>>>>> previous trunk
>>>>>> o One Level scope search (5 entries returned) : 68 715 entries returned
>>>>>> per second, compared to 33 120/s
>>>>>> o Sub Level scope search (10 entries returned ) : 70 830 entries
>>>>>> returned
>>>>>> per second, compared to 18 910/s
>>>>>>
>>>>>>
>>>>>> This is great work Emmanuel. Nicely done!
>>>>>>
>>>>> I have some even better results, as of today :
>>>>>
>>>>> o Object scope search (lookup) : 52 712 req/s compared to 23 081 on the
>>>>> previous trunk
>>>>> o One Level scope search (5 entries returned) : 72 635 entries returned
>>>>> per second, compared to 33 120/s
>>>>> o Sub Level scope search (10 entries returned ) : 75 100 entries returned
>>>>> per second, compared to 18 910/s
>>>>>
>>>>>
>>>> This is just sick man! You've more than doubled the performance.
>>>
>>>
>>> Some new idea this morning :
>>>
>>> atm, we do clone the entries we fetch from the server, then we filter the
>>> Attributes and the values, modifying the cloned entries. This leads to
>>> useless create of the removed Attributes and Values. We suggested to
>>> accumulate the modifications and to apply them at the end, avoiding the
>>> cloning of AT which will not be returned.
>>>
>>> First of all, we can avoid cloning the Values. The Value implementation are
>>> immutable classes. This save around 7% of the time.
>>>
>>> But this is not all we can do : we can simply avoid the accumulation of
>>> modifications *and* avoid cloning the entry !
>>>
>>> The idea is simple : when we get an entry in the cursor we have got back, we
>>> create a new empty entry, then we iterate over all the original entry's
>>> attributes and values, and for each one of those attributes and values, we
>>> check the filters, which will just tell if the Attribute/Value must be
>>> ditched or kept. This way, we don't do anything useless, like storing the
>>> modification or creating useless Attributes.
>>>
>>> It will work to the extent we deal with the CollectiveAttributes which must
>>> be injected somewhere, before we enter the loop (a connectiveAttribute might
>>> perfectly be removed by the ACI filter...). But we can also inject those
>>> added collective attributes into the loop of filters.
>>>
>>> I may miss something, but I do think that this solution is a clear winner,
>>> even in term of implementation...
>>>
>>> thoughts ?
>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Cordialement,
>>> Emmanuel Lécharny
>>> www.iktek.com
>>>
>>
>>
>>
>> --
>> Kiran Ayyagari
>
--
Kiran Ayyagari
Re: Inex branch merged into trunk...
Posted by Pierre-Arnaud Marcelot <pa...@marcelot.net>.
On 2 mai 2012, at 12:24, Kiran Ayyagari wrote:
> this method has two extreme cases,
> 1. best performance when no attributes are requested
> 2. same overhead as cloning when all attributes are requested
>
> cause in real world scenario 1 will never be used,
Actually in Studio, scenario 1 is used a lot.
That's exactly what happens do when opening a node and browsing the DIT in the UI (almost, because we do ask for the objectClass attribute).
Regards,
Pierre-Arnaud
> we might end up
> copying some attributes anyway
> but still better than cloning all
> On Wed, May 2, 2012 at 3:19 PM, Emmanuel Lécharny <el...@gmail.com> wrote:
>> Le 5/2/12 9:53 AM, Alex Karasulu a écrit :
>>>
>>> On Wed, May 2, 2012 at 2:43 AM, Emmanuel
>>> Lécharny<el...@gmail.com>wrote:
>>>
>>>> Le 5/1/12 3:05 PM, Alex Karasulu a écrit :
>>>>
>>>>> On Tue, May 1, 2012 at 4:08 AM, Emmanuel Lécharny<el...@gmail.com>**
>>>>>
>>>>> wrote:
>>>>>
>>>>> o Object scope search (lookup) : 49 880 req/s compared to 23 081 on the
>>>>> previous trunk
>>>>> o One Level scope search (5 entries returned) : 68 715 entries returned
>>>>> per second, compared to 33 120/s
>>>>> o Sub Level scope search (10 entries returned ) : 70 830 entries
>>>>> returned
>>>>> per second, compared to 18 910/s
>>>>>
>>>>>
>>>>> This is great work Emmanuel. Nicely done!
>>>>>
>>>> I have some even better results, as of today :
>>>>
>>>> o Object scope search (lookup) : 52 712 req/s compared to 23 081 on the
>>>> previous trunk
>>>> o One Level scope search (5 entries returned) : 72 635 entries returned
>>>> per second, compared to 33 120/s
>>>> o Sub Level scope search (10 entries returned ) : 75 100 entries returned
>>>> per second, compared to 18 910/s
>>>>
>>>>
>>> This is just sick man! You've more than doubled the performance.
>>
>>
>> Some new idea this morning :
>>
>> atm, we do clone the entries we fetch from the server, then we filter the
>> Attributes and the values, modifying the cloned entries. This leads to
>> useless create of the removed Attributes and Values. We suggested to
>> accumulate the modifications and to apply them at the end, avoiding the
>> cloning of AT which will not be returned.
>>
>> First of all, we can avoid cloning the Values. The Value implementation are
>> immutable classes. This save around 7% of the time.
>>
>> But this is not all we can do : we can simply avoid the accumulation of
>> modifications *and* avoid cloning the entry !
>>
>> The idea is simple : when we get an entry in the cursor we have got back, we
>> create a new empty entry, then we iterate over all the original entry's
>> attributes and values, and for each one of those attributes and values, we
>> check the filters, which will just tell if the Attribute/Value must be
>> ditched or kept. This way, we don't do anything useless, like storing the
>> modification or creating useless Attributes.
>>
>> It will work to the extent we deal with the CollectiveAttributes which must
>> be injected somewhere, before we enter the loop (a connectiveAttribute might
>> perfectly be removed by the ACI filter...). But we can also inject those
>> added collective attributes into the loop of filters.
>>
>> I may miss something, but I do think that this solution is a clear winner,
>> even in term of implementation...
>>
>> thoughts ?
>>
>>>
>>
>>
>> --
>> Regards,
>> Cordialement,
>> Emmanuel Lécharny
>> www.iktek.com
>>
>
>
>
> --
> Kiran Ayyagari
Re: Inex branch merged into trunk...
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 5/2/12 12:24 PM, Kiran Ayyagari a écrit :
> this method has two extreme cases,
> 1. best performance when no attributes are requested
> 2. same overhead as cloning when all attributes are requested
>
> cause in real world scenario 1 will never be used, we might end up
> copying some attributes anyway
> but still better than cloning all
generally speaking, users never request the Operational attributes. We
have quite a lot of them (at least 4), so there is a net gain here.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Inex branch merged into trunk...
Posted by Kiran Ayyagari <ka...@apache.org>.
this method has two extreme cases,
1. best performance when no attributes are requested
2. same overhead as cloning when all attributes are requested
cause in real world scenario 1 will never be used, we might end up
copying some attributes anyway
but still better than cloning all
On Wed, May 2, 2012 at 3:19 PM, Emmanuel Lécharny <el...@gmail.com> wrote:
> Le 5/2/12 9:53 AM, Alex Karasulu a écrit :
>>
>> On Wed, May 2, 2012 at 2:43 AM, Emmanuel
>> Lécharny<el...@gmail.com>wrote:
>>
>>> Le 5/1/12 3:05 PM, Alex Karasulu a écrit :
>>>
>>>> On Tue, May 1, 2012 at 4:08 AM, Emmanuel Lécharny<el...@gmail.com>**
>>>>
>>>> wrote:
>>>>
>>>> o Object scope search (lookup) : 49 880 req/s compared to 23 081 on the
>>>> previous trunk
>>>> o One Level scope search (5 entries returned) : 68 715 entries returned
>>>> per second, compared to 33 120/s
>>>> o Sub Level scope search (10 entries returned ) : 70 830 entries
>>>> returned
>>>> per second, compared to 18 910/s
>>>>
>>>>
>>>> This is great work Emmanuel. Nicely done!
>>>>
>>> I have some even better results, as of today :
>>>
>>> o Object scope search (lookup) : 52 712 req/s compared to 23 081 on the
>>> previous trunk
>>> o One Level scope search (5 entries returned) : 72 635 entries returned
>>> per second, compared to 33 120/s
>>> o Sub Level scope search (10 entries returned ) : 75 100 entries returned
>>> per second, compared to 18 910/s
>>>
>>>
>> This is just sick man! You've more than doubled the performance.
>
>
> Some new idea this morning :
>
> atm, we do clone the entries we fetch from the server, then we filter the
> Attributes and the values, modifying the cloned entries. This leads to
> useless create of the removed Attributes and Values. We suggested to
> accumulate the modifications and to apply them at the end, avoiding the
> cloning of AT which will not be returned.
>
> First of all, we can avoid cloning the Values. The Value implementation are
> immutable classes. This save around 7% of the time.
>
> But this is not all we can do : we can simply avoid the accumulation of
> modifications *and* avoid cloning the entry !
>
> The idea is simple : when we get an entry in the cursor we have got back, we
> create a new empty entry, then we iterate over all the original entry's
> attributes and values, and for each one of those attributes and values, we
> check the filters, which will just tell if the Attribute/Value must be
> ditched or kept. This way, we don't do anything useless, like storing the
> modification or creating useless Attributes.
>
> It will work to the extent we deal with the CollectiveAttributes which must
> be injected somewhere, before we enter the loop (a connectiveAttribute might
> perfectly be removed by the ACI filter...). But we can also inject those
> added collective attributes into the loop of filters.
>
> I may miss something, but I do think that this solution is a clear winner,
> even in term of implementation...
>
> thoughts ?
>
>>
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
--
Kiran Ayyagari
Re: Inex branch merged into trunk...
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 5/2/12 9:53 AM, Alex Karasulu a écrit :
> On Wed, May 2, 2012 at 2:43 AM, Emmanuel Lécharny<el...@gmail.com>wrote:
>
>> Le 5/1/12 3:05 PM, Alex Karasulu a écrit :
>>
>>> On Tue, May 1, 2012 at 4:08 AM, Emmanuel Lécharny<el...@gmail.com>**
>>> wrote:
>>>
>>> o Object scope search (lookup) : 49 880 req/s compared to 23 081 on the
>>> previous trunk
>>> o One Level scope search (5 entries returned) : 68 715 entries returned
>>> per second, compared to 33 120/s
>>> o Sub Level scope search (10 entries returned ) : 70 830 entries returned
>>> per second, compared to 18 910/s
>>>
>>>
>>> This is great work Emmanuel. Nicely done!
>>>
>> I have some even better results, as of today :
>>
>> o Object scope search (lookup) : 52 712 req/s compared to 23 081 on the
>> previous trunk
>> o One Level scope search (5 entries returned) : 72 635 entries returned
>> per second, compared to 33 120/s
>> o Sub Level scope search (10 entries returned ) : 75 100 entries returned
>> per second, compared to 18 910/s
>>
>>
> This is just sick man! You've more than doubled the performance.
Some new idea this morning :
atm, we do clone the entries we fetch from the server, then we filter
the Attributes and the values, modifying the cloned entries. This leads
to useless create of the removed Attributes and Values. We suggested to
accumulate the modifications and to apply them at the end, avoiding the
cloning of AT which will not be returned.
First of all, we can avoid cloning the Values. The Value implementation
are immutable classes. This save around 7% of the time.
But this is not all we can do : we can simply avoid the accumulation of
modifications *and* avoid cloning the entry !
The idea is simple : when we get an entry in the cursor we have got
back, we create a new empty entry, then we iterate over all the original
entry's attributes and values, and for each one of those attributes and
values, we check the filters, which will just tell if the
Attribute/Value must be ditched or kept. This way, we don't do anything
useless, like storing the modification or creating useless Attributes.
It will work to the extent we deal with the CollectiveAttributes which
must be injected somewhere, before we enter the loop (a
connectiveAttribute might perfectly be removed by the ACI filter...).
But we can also inject those added collective attributes into the loop
of filters.
I may miss something, but I do think that this solution is a clear
winner, even in term of implementation...
thoughts ?
>
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Inex branch merged into trunk...
Posted by Alex Karasulu <ak...@apache.org>.
On Wed, May 2, 2012 at 2:43 AM, Emmanuel Lécharny <el...@gmail.com>wrote:
> Le 5/1/12 3:05 PM, Alex Karasulu a écrit :
>
>> On Tue, May 1, 2012 at 4:08 AM, Emmanuel Lécharny<el...@gmail.com>**
>> wrote:
>>
>> o Object scope search (lookup) : 49 880 req/s compared to 23 081 on the
>> previous trunk
>> o One Level scope search (5 entries returned) : 68 715 entries returned
>> per second, compared to 33 120/s
>> o Sub Level scope search (10 entries returned ) : 70 830 entries returned
>> per second, compared to 18 910/s
>>
>>
>> This is great work Emmanuel. Nicely done!
>>
>
> I have some even better results, as of today :
>
> o Object scope search (lookup) : 52 712 req/s compared to 23 081 on the
> previous trunk
> o One Level scope search (5 entries returned) : 72 635 entries returned
> per second, compared to 33 120/s
> o Sub Level scope search (10 entries returned ) : 75 100 entries returned
> per second, compared to 18 910/s
>
>
This is just sick man! You've more than doubled the performance.
--
Best Regards,
-- Alex
Re: Inex branch merged into trunk...
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 5/1/12 3:05 PM, Alex Karasulu a écrit :
> On Tue, May 1, 2012 at 4:08 AM, Emmanuel Lécharny<el...@gmail.com>wrote:
>
> o Object scope search (lookup) : 49 880 req/s compared to 23 081 on the
> previous trunk
> o One Level scope search (5 entries returned) : 68 715 entries returned
> per second, compared to 33 120/s
> o Sub Level scope search (10 entries returned ) : 70 830 entries returned
> per second, compared to 18 910/s
>
>
> This is great work Emmanuel. Nicely done!
I have some even better results, as of today :
o Object scope search (lookup) : 52 712 req/s compared to 23 081 on the previous trunk
o One Level scope search (5 entries returned) : 72 635 entries returned per second, compared to 33 120/s
o Sub Level scope search (10 entries returned ) : 75 100 entries returned per second, compared to 18 910/s
I have just removed the use of the getSearchControls() call, gaining 10%
in perfs.
>
>
>> There is room for more improvement, but it will be more complex. The area
>> that can be improved are :
>> o get rid of the extra getSearchControls() call in intercepotrs. This is
>> the easiest fix
Done.
>> o review the way we handle entries modification before we return them.
>> Currently, we clone the entry, and remove the attributes the user has not
>> required. See DIRSERVER-1719 for more explaination on this subject. Note
>> that the filtering of attributes represent around 9% of the global CPU time.
>> o getting back the ID from a Dn is a very costly operation (19% of the
>> global CPU time), and the longer the DN, the longer the operation. For each
>> RDN, we have to do a lookup in the RdnIndex. The only solution would be to
>> have a Dn -> ID cache somewhere. This would boost the server performance,
>> that's for sure.
>> o fetching an entry from the backend cost 38% of the global time, out of
>> which 29% represent the cost to clone the entry. If we could avoid doing
>> this clone (see upper), we may have some major performances increase.
>> o when evaluating an entry to see if it fits the filter, we use the
>> reverseIndex, which is also a costly operation. We shoudl re-evaluate if it
>> wouldn't be better to use the MatchingRules comparator to do that instead
>> (reverse lookups account for 4% of the used CPU time)
>>
>>
> I guess we have these in JIRA?
No, not all of them. I will add the missing one (reverseIndex evaluation)
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Inex branch merged into trunk...
Posted by Alex Karasulu <ak...@apache.org>.
On Tue, May 1, 2012 at 4:08 AM, Emmanuel Lécharny <el...@gmail.com>wrote:
> Hi,
>
> just to inform you that the index branch has been merged with no harm
> today. I just had to fix 3 conflicts, and two bugs I introduced in the
> branch before the commit.
>
> The server performance is way better for searches, with a few improvements
> I did those last 4 days. It was impressive how easy it was to improve the
> speed with little modifications. The global result is that the server is
> now :
> o Object scope search (lookup) : 49 880 req/s compared to 23 081 on the
> previous trunk
> o One Level scope search (5 entries returned) : 68 715 entries returned
> per second, compared to 33 120/s
> o Sub Level scope search (10 entries returned ) : 70 830 entries returned
> per second, compared to 18 910/s
>
>
This is great work Emmanuel. Nicely done!
> There is room for more improvement, but it will be more complex. The area
> that can be improved are :
> o get rid of the extra getSearchControls() call in intercepotrs. This is
> the easiest fix
> o review the way we handle entries modification before we return them.
> Currently, we clone the entry, and remove the attributes the user has not
> required. See DIRSERVER-1719 for more explaination on this subject. Note
> that the filtering of attributes represent around 9% of the global CPU time.
> o getting back the ID from a Dn is a very costly operation (19% of the
> global CPU time), and the longer the DN, the longer the operation. For each
> RDN, we have to do a lookup in the RdnIndex. The only solution would be to
> have a Dn -> ID cache somewhere. This would boost the server performance,
> that's for sure.
> o fetching an entry from the backend cost 38% of the global time, out of
> which 29% represent the cost to clone the entry. If we could avoid doing
> this clone (see upper), we may have some major performances increase.
> o when evaluating an entry to see if it fits the filter, we use the
> reverseIndex, which is also a costly operation. We shoudl re-evaluate if it
> wouldn't be better to use the MatchingRules comparator to do that instead
> (reverse lookups account for 4% of the used CPU time)
>
>
I guess we have these in JIRA?
> One interesting result is that the LRUCache.get() operation represent 13%
> of the used time. This is definitively not small. There is probably some
> room for some improvement here, but this is way more complex...
>
> All those numbers have been collected using YourKit on a Lookup test (150
> 000 lookups on one single element have been done)
>
>
I wonder what the over the network stats are with a client machine separate
from the server machine. Oh and with multiple clients. It's too bad we
never got a chance to setup such an environment :( .
>
> There are also some improvements to expect on the Add/Delete/Move
> operation, as we have to delete/add the keys on the RdnIndex. This is
> something Im going to work on tomorrow.
>
>
Cool.
>
> One more thing : the number I get when running the server-integ search
> perf are way below (from 2900 to 5400 per second). This is plain normal.
> When going through th network, we pay some extra price :
> o the client code eats 57% of all the time it takes to run the test
> o On the server, normalizing the incoming Dn costs 7% of the processing
> time
> o the entries encoding is very expensive
>
> All in all, on the server, unless we test it on a different machine than
> the injectors, all the measures are pretty impossible to do. There is too
> much noise...
>
> I'd be interested to conduct largest tests on a multi-core server, with
> lots of memory, and a lot of entries, with external injectors, to see what
> kind of performances we can get...
>
>
Ditto.
>
> In the next few days, I will probably fix some pending bugs. I think we
> can cut a M7 release by the end of this week, and make it available by next
> week.
>
>
That sounds great. Thanks!
--
Best Regards,
-- Alex