Indexing performance

[Craig Setera <cr...@mfoundry.com>]

In my recent conversations about indexing, I was having problems with
indexing.  I've managed to get it to work locally, but others are having
trouble replicating my working instance.

My question to the group isn't related to that, but to the performance I'm
seeing when the indexes are working locally for me.  What should I expect to
see in terms of performance difference with the indexing in place?  I've
created a script that uses ldapsearch to search for 10 entries 10 times and
I'm actually seeing mixed results where sometimes it is actually *slower*
with the indexing in place:

ldapsearch -H ldap://localhost:10389 -x -D "uid=admin,ou=system" -b
"dc=mfoundry,dc=com" -w secret -s sub -a always "(mfname=$name)" "mfname"
"mfuid" "objectClass" >> ~/temp/apachedstests/searchresults.txt

With indexing disabled:

real 0m8.626s
user 0m0.387s
sys 0m0.406s

With indexing enabled, it is inconsistent, but averages no better:

real 0m9.161s
user 0m0.381s
sys 0m0.391s

real 0m8.329s
user 0m0.382s
sys 0m0.391s

real 0m8.357s
user 0m0.381s
sys 0m0.390s

Does that mean that the indexes are not being used?  Or does it just mean
that my use case isn't something helped by the indexes?

Thanks for any insights.
Craig



-- 
Craig Setera
Director, Product Engineering
mFoundry
p 415.324.5801
craig@mfoundry.com

Re: Indexing performance

[Alex Karasulu <ak...@apache.org>]

On Wed, Oct 5, 2011 at 2:41 PM, Emmanuel Lecharny <el...@gmail.com>wrote:

> On 10/5/11 1:29 PM, Craig Setera wrote:
>
>> In my recent conversations about indexing, I was having problems with
>> indexing.  I've managed to get it to work locally, but others are having
>> trouble replicating my working instance.
>>
>> My question to the group isn't related to that, but to the performance I'm
>> seeing when the indexes are working locally for me.  What should I expect
>> to
>> see in terms of performance difference with the indexing in place?  I've
>> created a script that uses ldapsearch to search for 10 entries 10 times
>> and
>> I'm actually seeing mixed results where sometimes it is actually *slower*
>> with the indexing in place:
>>
>>
Also the entry cache to DIB size ratio is important as well. If you're
keeping every entry cached yet your index is not cached then performance can
theoretically decrease with an index that has to keep hitting the disk.
 Right now the way we deal with caching there is no clear way to tune this
optimally: it's a mater of experimentation with tweaks to the entry cache
and index caches.

-- 
Best Regards,
-- Alex

Re: Indexing performance

[Emmanuel Lecharny <el...@gmail.com>]

On 10/5/11 1:29 PM, Craig Setera wrote:
> In my recent conversations about indexing, I was having problems with
> indexing.  I've managed to get it to work locally, but others are having
> trouble replicating my working instance.
>
> My question to the group isn't related to that, but to the performance I'm
> seeing when the indexes are working locally for me.  What should I expect to
> see in terms of performance difference with the indexing in place?  I've
> created a script that uses ldapsearch to search for 10 entries 10 times and
> I'm actually seeing mixed results where sometimes it is actually *slower*
> with the indexing in place:
>
> ldapsearch -H ldap://localhost:10389 -x -D "uid=admin,ou=system" -b
> "dc=mfoundry,dc=com" -w secret -s sub -a always "(mfname=$name)" "mfname"
> "mfuid" "objectClass">>  ~/temp/apachedstests/searchresults.txt
>
> With indexing disabled:
>
> real 0m8.626s
> user 0m0.387s
> sys 0m0.406s
>
> With indexing enabled, it is inconsistent, but averages no better:
>
> real 0m9.161s
> user 0m0.381s
> sys 0m0.391s
>
> real 0m8.329s
> user 0m0.382s
> sys 0m0.391s
>
> real 0m8.357s
> user 0m0.381s
> sys 0m0.390s
>
> Does that mean that the indexes are not being used?  Or does it just mean
> that my use case isn't something helped by the indexes?
It depends on the number of entries you have in the base.

With only a few, indexes won't be useful. If you have millions of 
entries, trust me, you will see the difference.


-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com