You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@directory.apache.org by Ca...@ibs-ag.com on 2012/10/15 23:52:46 UTC
Possible regression for substring search on indexes?
Hi, we're using M9 built from the trunk on Wednesday of last week. We have our db setup as it has been for the past several months. There are 80k users.
(sn=Thompson) returns in miliseconds
(sn=Thompso*) returns after 3-4 minutes.
This seems very much like an issue we had back and in June (and was fixed). Any help is most appreciated. We need this build for all the good work Karin did on the password policy.
Thank you, Carlo Accorsi
Re: Possible regression for substring search on indexes?
Posted by Emmanuel Lécharny <el...@gmail.com>.
I have done some tests with 80 000 entries in the server (ok not a big
entry but anyway).
The entry is like :
dn: cn=testNNNNN,dc=example,dc=com
ObjectClass: top
ObjectClass: person
sn: testNNNN
cn: test
where NNNN varies from 1 to 80000
Doing such a search :
baseDN : dc=example,dc=com
filter : "(sn=test123*)"
scope : SUBTREE
returns 111 entries in 166 milliseconds once I have set the SN index.
If the SN index is not set, then the search does take several minutes too.
I would suggest you reinject all the entries in the server with the new
jars.
I'm investigating the OOM error on my side.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Possible regression for substring search on indexes?
Posted by Kiran Ayyagari <ka...@apache.org>.
On Thu, Oct 18, 2012 at 3:24 AM, Emmanuel Lécharny <el...@gmail.com> wrote:
> Le 10/17/12 11:10 PM, Carlo.Accorsi@ibs-ag.com a écrit :
>
>> Hi, All - I've rebuilt from the trunk this morning twice and loaded our
>> users from scratch.
>>
>> The out of memory errors are gone, thank you but now we don't get results
>> of any kind, seemingly under any circumstances when doing a substring
>> search.
>> I get instant results when doing an exact match but the substring searches
>> are not working.
>>
>> sn=Smith
>>
>> However,
>> sn=Smit*
>>
>> The result comes back (0, filtered) from Studio.
>>
>> At one point there were some corrections to normalize data (to lowercase)
>> in the indexes. I tested searching against
>>
>> sn=smit*
>>
>> after saving the field back to 'smith' and still no results.
>>
>> I check out from here. Is this where I should? Thanks!
>> svn checkout
>> http://svn.apache.org/repos/asf/directory/apacheds/trunk-with-dependencies
>
> This link or http://svn.apache.org/repos/asf/directory/trunks/
>
> but you will get the same sources.
>
> Ths is quite strange, as the tests I conducted shown that the substring
> filter was correctly returning the right values...
>
> I have added a test that demonstrate the subsctring search works on indexed
> attributes :
>
> http://svn.apache.org/viewvc?rev=1399450&view=rev
>
> Can you test that this test works for you ?
>
> Sady, I'm MIA for a few days, with little internet connection and time to
> check that. Kiran or Pierre-Arnaud, you may give me an hand while I'm not
> around ?
>
sure, count on us :)
> Thanks !
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
--
Kiran Ayyagari
http://keydap.com
Re: Possible regression for substring search on indexes?
Posted by Kiran Ayyagari <ka...@apache.org>.
On Thu, Oct 18, 2012 at 3:34 AM, <Ca...@ibs-ag.com> wrote:
> Just realized my partition doesn't have a ads-contextentry.. Is this critical? Could this be a factor? We didn't have it in our M8 instances.
>
no this is not critical
> Ok, I'll try and run the tests but it won't be until later and I know you are leaving. Thanks for your help. If it's not working, we can sort it out when you get back.
>
can you turn logging on at debug level in the server and report any
unexpected behavior (exceptions etc.)
below is a sample log4j configuration I use
log4j.rootCategory=OFF, stdout, file
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d{HH:mm:ss}] %p [%c] - %m%n
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=[%d{HH:mm:ss}] %p [%c] - %m%n
log4j.appender.file.File=/tmp/mmr.log
log4j.appender.file.MaxFileSize=5120KB
log4j.appender.file.MaxBackupIndex=5
log4j.logger.org.apache.directory.server.core.partition=FATAL
log4j.logger.org.apache.directory.shared.ldap.model=FATAL
log4j.logger.org.apache.directory.shared.ldap.codec=FATAL
log4j.logger.org.apache.directory.server.core.partition.ldif=FATAL
#please keep the below levels at FATAL or WARNING only to reduce the noise
log4j.logger.org.apache.directory.shared.util.Strings=FATAL
log4j.logger.ATTRIBUTE_TYPE=FATAL
log4j.logger.SYNTAX_CHECKER=FATAL
log4j.logger.MATCHING_RULE=FATAL
log4j.logger.LDAP_SYNTAX=FATAL
log4j.logger.org.apache.directory.shared.ldap.model.schema=FATAL
log4j.logger.JdbmTable=FATAL
log4j.logger.JdbmIndex=FATAL
log4j.logger.BaseRecordManager=FATAL
log4j.logger.LOG_CHANGES=FATAL
log4j.logger.net=FATAL
log4j.logger.aci-logger=FATAL
log4j.logger.DupsCursor=FATAL
log4j.logger.PROVIDER_LOG=FATAL
log4j.logger.CONSUMER_LOG=FATAL
log4j.logger.CURSOR=FATAL
log4j.logger.org.apache.directory.shared.asn1.ber=FATAL
> Regards,
> Carlo Accorsi
>
> -----Original Message-----
> From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
> Sent: Wednesday, October 17, 2012 5:55 PM
> To: users@directory.apache.org
> Subject: Re: Possible regression for substring search on indexes?
>
> Le 10/17/12 11:10 PM, Carlo.Accorsi@ibs-ag.com a écrit :
>> Hi, All - I've rebuilt from the trunk this morning twice and loaded our users from scratch.
>>
>> The out of memory errors are gone, thank you but now we don't get results of any kind, seemingly under any circumstances when doing a substring search.
>> I get instant results when doing an exact match but the substring searches are not working.
>>
>> sn=Smith
>>
>> However,
>> sn=Smit*
>>
>> The result comes back (0, filtered) from Studio.
>>
>> At one point there were some corrections to normalize data (to
>> lowercase) in the indexes. I tested searching against
>>
>> sn=smit*
>>
>> after saving the field back to 'smith' and still no results.
>>
>> I check out from here. Is this where I should? Thanks!
>> svn checkout
>> http://svn.apache.org/repos/asf/directory/apacheds/trunk-with-dependen
>> cies
> This link or http://svn.apache.org/repos/asf/directory/trunks/
>
> but you will get the same sources.
>
> Ths is quite strange, as the tests I conducted shown that the substring filter was correctly returning the right values...
>
> I have added a test that demonstrate the subsctring search works on indexed attributes :
>
> http://svn.apache.org/viewvc?rev=1399450&view=rev
>
> Can you test that this test works for you ?
>
> Sady, I'm MIA for a few days, with little internet connection and time to check that. Kiran or Pierre-Arnaud, you may give me an hand while I'm not around ?
>
> Thanks !
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
--
Kiran Ayyagari
http://keydap.com
RE: Possible regression for substring search on indexes?
Posted by Ca...@ibs-ag.com.
Just realized my partition doesn't have a ads-contextentry.. Is this critical? Could this be a factor? We didn't have it in our M8 instances.
Ok, I'll try and run the tests but it won't be until later and I know you are leaving. Thanks for your help. If it's not working, we can sort it out when you get back.
Regards,
Carlo Accorsi
-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
Sent: Wednesday, October 17, 2012 5:55 PM
To: users@directory.apache.org
Subject: Re: Possible regression for substring search on indexes?
Le 10/17/12 11:10 PM, Carlo.Accorsi@ibs-ag.com a écrit :
> Hi, All - I've rebuilt from the trunk this morning twice and loaded our users from scratch.
>
> The out of memory errors are gone, thank you but now we don't get results of any kind, seemingly under any circumstances when doing a substring search.
> I get instant results when doing an exact match but the substring searches are not working.
>
> sn=Smith
>
> However,
> sn=Smit*
>
> The result comes back (0, filtered) from Studio.
>
> At one point there were some corrections to normalize data (to
> lowercase) in the indexes. I tested searching against
>
> sn=smit*
>
> after saving the field back to 'smith' and still no results.
>
> I check out from here. Is this where I should? Thanks!
> svn checkout
> http://svn.apache.org/repos/asf/directory/apacheds/trunk-with-dependen
> cies
This link or http://svn.apache.org/repos/asf/directory/trunks/
but you will get the same sources.
Ths is quite strange, as the tests I conducted shown that the substring filter was correctly returning the right values...
I have added a test that demonstrate the subsctring search works on indexed attributes :
http://svn.apache.org/viewvc?rev=1399450&view=rev
Can you test that this test works for you ?
Sady, I'm MIA for a few days, with little internet connection and time to check that. Kiran or Pierre-Arnaud, you may give me an hand while I'm not around ?
Thanks !
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Possible regression for substring search on indexes?
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 10/17/12 11:10 PM, Carlo.Accorsi@ibs-ag.com a écrit :
> Hi, All - I've rebuilt from the trunk this morning twice and loaded our users from scratch.
>
> The out of memory errors are gone, thank you but now we don't get results of any kind, seemingly under any circumstances when doing a substring search.
> I get instant results when doing an exact match but the substring searches are not working.
>
> sn=Smith
>
> However,
> sn=Smit*
>
> The result comes back (0, filtered) from Studio.
>
> At one point there were some corrections to normalize data (to lowercase) in the indexes. I tested searching against
>
> sn=smit*
>
> after saving the field back to 'smith' and still no results.
>
> I check out from here. Is this where I should? Thanks!
> svn checkout http://svn.apache.org/repos/asf/directory/apacheds/trunk-with-dependencies
This link or http://svn.apache.org/repos/asf/directory/trunks/
but you will get the same sources.
Ths is quite strange, as the tests I conducted shown that the substring
filter was correctly returning the right values...
I have added a test that demonstrate the subsctring search works on
indexed attributes :
http://svn.apache.org/viewvc?rev=1399450&view=rev
Can you test that this test works for you ?
Sady, I'm MIA for a few days, with little internet connection and time
to check that. Kiran or Pierre-Arnaud, you may give me an hand while I'm
not around ?
Thanks !
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
RE: Possible regression for substring search on indexes?
Posted by Ca...@ibs-ag.com.
Hi, All - I've rebuilt from the trunk this morning twice and loaded our users from scratch.
The out of memory errors are gone, thank you but now we don't get results of any kind, seemingly under any circumstances when doing a substring search.
I get instant results when doing an exact match but the substring searches are not working.
sn=Smith
However,
sn=Smit*
The result comes back (0, filtered) from Studio.
At one point there were some corrections to normalize data (to lowercase) in the indexes. I tested searching against
sn=smit*
after saving the field back to 'smith' and still no results.
I check out from here. Is this where I should? Thanks!
svn checkout http://svn.apache.org/repos/asf/directory/apacheds/trunk-with-dependencies
Regards,
Carlo Accorsi
-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
Sent: Wednesday, October 17, 2012 1:40 AM
To: users@directory.apache.org
Subject: Re: Possible regression for substring search on indexes?
Le 10/16/12 5:03 PM, Emmanuel Lécharny a écrit :
> Le 10/16/12 4:51 PM, Carlo.Accorsi@ibs-ag.com a écrit :
>> OK thanks!
> I think I found the reason why we have an OOM in such a case.
>
> Testing the fix atm.
I have committed some more fix for the OOM. Tests are passing.
I suggest you test the server with those last patches.
FYI, I'll be MIA for the next 5 days.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
RE: Possible regression for substring search on indexes?
Posted by Ca...@ibs-ag.com.
OK thank you!!!
Regards,
Carlo Accorsi
-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
Sent: Wednesday, October 17, 2012 1:40 AM
To: users@directory.apache.org
Subject: Re: Possible regression for substring search on indexes?
Le 10/16/12 5:03 PM, Emmanuel Lécharny a écrit :
> Le 10/16/12 4:51 PM, Carlo.Accorsi@ibs-ag.com a écrit :
>> OK thanks!
> I think I found the reason why we have an OOM in such a case.
>
> Testing the fix atm.
I have committed some more fix for the OOM. Tests are passing.
I suggest you test the server with those last patches.
FYI, I'll be MIA for the next 5 days.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Possible regression for substring search on indexes?
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 10/16/12 5:03 PM, Emmanuel Lécharny a écrit :
> Le 10/16/12 4:51 PM, Carlo.Accorsi@ibs-ag.com a écrit :
>> OK thanks!
> I think I found the reason why we have an OOM in such a case.
>
> Testing the fix atm.
I have committed some more fix for the OOM. Tests are passing.
I suggest you test the server with those last patches.
FYI, I'll be MIA for the next 5 days.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Possible regression for substring search on indexes?
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 10/16/12 4:51 PM, Carlo.Accorsi@ibs-ag.com a écrit :
> OK thanks!
I think I found the reason why we have an OOM in such a case.
Testing the fix atm.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
RE: Possible regression for substring search on indexes?
Posted by Ca...@ibs-ag.com.
OK thanks!
-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
Sent: Tuesday, October 16, 2012 10:30 AM
To: users@directory.apache.org
Subject: Re: Possible regression for substring search on indexes?
Le 10/16/12 4:02 PM, Carlo.Accorsi@ibs-ag.com a écrit :
> Oops forgot to mention. We have 1GB assigned to the JVM. It's the same arrangement we've had for months. This is the first time I've seen an out of memory error.
> I'm the only one on the machine so there's no load other than mine. Seeing the method below the stack trace made me wonder if it's related to the recent changes.
> I'm rebuilding everything from scratch again. Thanks.
If there is no index set for the substring filtered attribte, with 80
000 entries, I also get an OOM.
Investigating...
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Possible regression for substring search on indexes?
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 10/16/12 4:02 PM, Carlo.Accorsi@ibs-ag.com a écrit :
> Oops forgot to mention. We have 1GB assigned to the JVM. It's the same arrangement we've had for months. This is the first time I've seen an out of memory error.
> I'm the only one on the machine so there's no load other than mine. Seeing the method below the stack trace made me wonder if it's related to the recent changes.
> I'm rebuilding everything from scratch again. Thanks.
If there is no index set for the substring filtered attribte, with 80
000 entries, I also get an OOM.
Investigating...
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
RE: Possible regression for substring search on indexes?
Posted by Ca...@ibs-ag.com.
Oops forgot to mention. We have 1GB assigned to the JVM. It's the same arrangement we've had for months. This is the first time I've seen an out of memory error.
I'm the only one on the machine so there's no load other than mine. Seeing the method below the stack trace made me wonder if it's related to the recent changes.
I'm rebuilding everything from scratch again. Thanks.
at org.apache.directory.server.xdbm.search.evaluator.SubstringEvaluator.evaluate(SubstringEvaluator.java:135)
Regards,
Carlo Accorsi
-----Original Message-----
From: ayyagarikiran@gmail.com [mailto:ayyagarikiran@gmail.com] On Behalf Of Kiran Ayyagari
Sent: Tuesday, October 16, 2012 9:58 AM
To: users@directory.apache.org
Subject: Re: Possible regression for substring search on indexes?
On Tue, Oct 16, 2012 at 7:24 PM, <Ca...@ibs-ag.com> wrote:
> Here are my files.. no, doesn't appear that I have them. Do these look right ?
>
> 0.9.2342.19200300.100.1.1.db
> 0.9.2342.19200300.100.1.1.lg
> 0.9.2342.19200300.100.1.1-uid.txt
> 0.9.2342.19200300.100.1.3.db
> 0.9.2342.19200300.100.1.3.lg
> 0.9.2342.19200300.100.1.3-mail.txt
> 0.9.2342.19200300.100.1.6.db
> 0.9.2342.19200300.100.1.6.lg
> 0.9.2342.19200300.100.1.6-roomNumber.txt
> 1.3.6.1.4.1.18060.0.4.1.2.3.db
> 1.3.6.1.4.1.18060.0.4.1.2.3.lg
> 1.3.6.1.4.1.18060.0.4.1.2.3-apachePresence.txt
> 1.3.6.1.4.1.18060.0.4.1.2.5.db
> 1.3.6.1.4.1.18060.0.4.1.2.5.lg
> 1.3.6.1.4.1.18060.0.4.1.2.50.db
> 1.3.6.1.4.1.18060.0.4.1.2.50-apacheRdn.txt
> 1.3.6.1.4.1.18060.0.4.1.2.5-apacheOneAlias.txt
> 1.3.6.1.4.1.18060.0.4.1.2.6.db
> 1.3.6.1.4.1.18060.0.4.1.2.6.lg
> 1.3.6.1.4.1.18060.0.4.1.2.6-apacheSubAlias.txt
> 1.3.6.1.4.1.18060.0.4.1.2.7.db
> 1.3.6.1.4.1.18060.0.4.1.2.7.lg
> 1.3.6.1.4.1.18060.0.4.1.2.7-apacheAlias.txt
> 1.3.6.1.4.1.42.2.27.8.1.23.db
> 1.3.6.1.4.1.42.2.27.8.1.23.lg
> 1.3.6.1.4.1.42.2.27.8.1.23-pwdPolicySubentry.txt
> 1.3.6.1.4.1.4203.666.1.7.db
> 1.3.6.1.4.1.4203.666.1.7.lg
> 1.3.6.1.4.1.4203.666.1.7-entryCSN.txt
> 1.3.6.1.4.1.5322.10.1.1.db
> 1.3.6.1.4.1.5322.10.1.1.lg
> 1.3.6.1.4.1.5322.10.1.1-krb5PrincipalName.txt
> 2.16.840.1.113730.3.1.241.db
> 2.16.840.1.113730.3.1.241.lg
> 2.16.840.1.113730.3.1.241-displayName.txt
> 2.16.840.1.113730.3.1.3.db
> 2.16.840.1.113730.3.1.3.lg
> 2.16.840.1.113730.3.1.3-employeeNumber.txt
> 2.5.18.5.db
> 2.5.18.5.lg
> 2.5.18.5-administrativeRole.txt
> 2.5.4.0.db
> 2.5.4.0.lg
> 2.5.4.0-objectClass.txt
> 2.5.4.10.db
> 2.5.4.10.lg
> 2.5.4.10-o.txt
> 2.5.4.11.db
> 2.5.4.11.lg
> 2.5.4.11-ou.txt
> 2.5.4.13.db
> 2.5.4.13.lg
> 2.5.4.13-description.txt
> 2.5.4.3.db
> 2.5.4.3.lg
> 2.5.4.31.db
> 2.5.4.31.lg
> 2.5.4.31-member.txt
> 2.5.4.3-cn.txt
> 2.5.4.4.db
> 2.5.4.4.lg
> 2.5.4.42.db
> 2.5.4.42.lg
> 2.5.4.42-givenName.txt
> 2.5.4.4-sn.txt
> master.db
>
looks fine
> Out of memory when searching for sn=Thomps*
>
ahhh, what is the size of heap memory allocated to JVM?
> jvm 1 | Server daemon died!
> jvm 1 | java.lang.OutOfMemoryError: Java heap space
> jvm 1 | at java.nio.CharBuffer.allocate(CharBuffer.java:312)
> jvm 1 | at java.nio.charset.CharsetEncoder.isLegalReplacement(CharsetEncoder.java:319)
> jvm 1 | at java.nio.charset.CharsetEncoder.replaceWith(CharsetEncoder.java:267)
> jvm 1 | at java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:186)
> jvm 1 | at java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:209)
> jvm 1 | at sun.nio.cs.SingleByteEncoder.<init>(SingleByteEncoder.java:39)
> jvm 1 | at sun.nio.cs.MS1252$Encoder.<init>(MS1252.java:115)
> jvm 1 | at sun.nio.cs.MS1252.newEncoder(MS1252.java:43)
> jvm 1 | at java.lang.StringCoding$StringEncoder.<init>(StringCoding.java:215)
> jvm 1 | at java.lang.StringCoding$StringEncoder.<init>(StringCoding.java:207)
> jvm 1 | at java.lang.StringCoding.encode(StringCoding.java:266)
> jvm 1 | at java.lang.StringCoding.encode(StringCoding.java:284)
> jvm 1 | at java.lang.String.getBytes(String.java:986)
> jvm 1 | at org.tanukisoftware.wrapper.WrapperManager.sendCommand(WrapperManager.java:3685)
> jvm 1 | at org.tanukisoftware.wrapper.WrapperManager.handleSocket(WrapperManager.java:3804)
> jvm 1 | at org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java:4084)
> jvm 1 | at java.lang.Thread.run(Thread.java:662)
> jvm 1 | [09:31:26] WARN [org.apache.directory.server.ldap.LdapProtocolHandler] - Unexpected exception forcing session to close: sending disconnect notice to client.
> jvm 1 | java.lang.OutOfMemoryError: Java heap space
> jvm 1 | at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:232)
> jvm 1 | at java.lang.StringCoding.encode(StringCoding.java:272)
> jvm 1 | at java.lang.String.getBytes(String.java:946)
> jvm 1 | at org.apache.directory.shared.util.Strings.getBytesUtf8(Strings.java:1587)
> jvm 1 | at org.apache.directory.shared.ldap.model.entry.StringValue.readExternal(StringValue.java:448)
> jvm 1 | at org.apache.directory.shared.ldap.model.entry.DefaultAttribute.readExternal(DefaultAttribute.java:2064)
> jvm 1 | at org.apache.directory.server.core.partition.impl.btree.jdbm.EntrySerializer.deserialize(EntrySerializer.java:219)
> jvm 1 | at jdbm.btree.BPage.deserialize(BPage.java:1255)
> jvm 1 | at jdbm.btree.BPage.deserialize(BPage.java:84)
> jvm 1 | at jdbm.recman.BaseRecordManager.fetch(BaseRecordManager.java:451)
> jvm 1 | at jdbm.recman.CacheRecordManager.fetch(CacheRecordManager.java:264)
> jvm 1 | at jdbm.btree.BPage.loadBPage(BPage.java:1017)
> jvm 1 | at jdbm.btree.BPage.find(BPage.java:315)
> jvm 1 | at jdbm.btree.BTree.browse(BTree.java:706)
> jvm 1 | at jdbm.btree.BTree.find(BTree.java:548)
> jvm 1 | at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.get(JdbmTable.java:312)
> jvm 1 | at org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.lookup(AbstractBTreePartition.java:1183)
> jvm 1 | at org.apache.directory.server.xdbm.search.evaluator.SubstringEvaluator.evaluate(SubstringEvaluator.java:135)
> jvm 1 | at org.apache.directory.server.xdbm.search.evaluator.AndEvaluator.evaluate(AndEvaluator.java:109)
> jvm 1 | at org.apache.directory.server.core.partition.impl.btree.EntryCursorAdaptor.get(EntryCursorAdaptor.java:146)
> jvm 1 | at org.apache.directory.server.core.partition.impl.btree.EntryCursorAdaptor.get(EntryCursorAdaptor.java:40)
> jvm 1 | at org.apache.directory.server.core.api.filtering.BaseEntryFilteringCursor.next(BaseEntryFilteringCursor.java:475)
> jvm 1 | at org.apache.directory.server.ldap.handlers.SearchHandler.readResults(SearchHandler.java:377)
> jvm 1 | at org.apache.directory.server.ldap.handlers.SearchHandler.doSimpleSearch(SearchHandler.java:830)
> jvm 1 | at org.apache.directory.server.ldap.handlers.SearchHandler.handleIgnoringReferrals(SearchHandler.java:1137)
> jvm 1 | at org.apache.directory.server.ldap.handlers.SearchHandler.handleWithReferrals(SearchHandler.java:1220)
> jvm 1 | at org.apache.directory.server.ldap.handlers.SearchHandler.handle(SearchHandler.java:227)
> jvm 1 | at org.apache.directory.server.ldap.handlers.SearchHandler.handle(SearchHandler.java:87)
> jvm 1 | at org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:209)
> jvm 1 | at org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:56)
> jvm 1 | at org.apache.mina.handler.demux.DemuxingIoHandler.messageReceived(DemuxingIoHandler.java:232)
> jvm 1 | at org.apache.directory.server.ldap.LdapProtocolHandler.messageReceived(LdapProtocolHandler.java:209)
>
>
>
>
> Regards,
> Carlo Accorsi
>
>
>
> -----Original Message-----
> From: ayyagarikiran@gmail.com [mailto:ayyagarikiran@gmail.com] On Behalf Of Kiran Ayyagari
> Sent: Tuesday, October 16, 2012 9:46 AM
> To: users@directory.apache.org
> Subject: Re: Possible regression for substring search on indexes?
>
> On Tue, Oct 16, 2012 at 6:35 PM, <Ca...@ibs-ag.com> wrote:
>> OK thanks for the quick response. I've built and tested (revision
>> 1398757) but is still taking several minutes to return substring search. I'm doing my searching via Studio and have exclusive access To the directory. Interestingly, it somehow knows that count right away but doesn't return anything for several mintues.
>
>>
>> I just swapped out the jar (with the one from a week ago) in this case bc I was hoping to validate the test without needing to rebuild and re-import all the users.
>> I would rebuild everything before for production but to test this correction, do I need to do this?
>>
> better to rebuild the server and overwrite the existing jars
>> Also, months ago we had an explicit index on entryUUID but we removed it when we didn't see it in the stock config.ldif.
>> entryCSN does have an index.
>>
> it is not needed anymore, cause we now use entryUUID as the *ID* of entry and master table contains these IDs as keys
>> These are the indexed attrs on the partition. After the ldif import they all had grown in size (on disk) so I'm fairly confident they're being populated.
>> Are we missing any that would work in conjunction with index searches? Thanks!
>>
>> apacheRdn
>> apacheSubLevel
>> apachePresence
>> apacheOneLevel
>> apacheOneAlias
>> apacheSubAlias
>> apacheAlias
>> entryCSN
>> o
>> krb5PrincipalName
>> objectClass
>> ou
>> uid
>> employeeNumber
>> displayName
>> cn
>> mail
>> roomNumber
>> pwdPolicySubEntry
>> member
>> description
>> givenName
>> sn
>> administrativeRole
>>
>>
> no, nothing is missing, otoh, do you really have the .db files related to ONE and SUB level indices (not ONE and SUB level *aliases*) curious cause they are no longer created/used by the server
>> Regards,
>> Carlo Accorsi
>>
>> -----Original Message-----
>> From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
>> Sent: Tuesday, October 16, 2012 7:33 AM
>> To: users@directory.apache.org
>> Subject: Re: Possible regression for substring search on indexes?
>>
>> Le 10/15/12 11:52 PM, Carlo.Accorsi@ibs-ag.com a écrit :
>>> Hi, we're using M9 built from the trunk on Wednesday of last week. We have our db setup as it has been for the past several months. There are 80k users.
>>>
>>> (sn=Thompson) returns in miliseconds
>>> (sn=Thompso*) returns after 3-4 minutes.
>>>
>>> This seems very much like an issue we had back and in June (and was fixed). Any help is most appreciated. We need this build for all the good work Karin did on the password policy.
>>>
>>> Thank you, Carlo Accorsi
>>>
>> It should be fixed with
>> http://svn.apache.org/viewvc?rev=1398737&view=rev
>>
>> Can you give it a try ?
>>
>> --
>> Regards,
>> Cordialement,
>> Emmanuel Lécharny
>> www.iktek.com
>>
>
>
>
> --
> Kiran Ayyagari
> http://keydap.com
--
Kiran Ayyagari
http://keydap.com
Re: Possible regression for substring search on indexes?
Posted by Kiran Ayyagari <ka...@apache.org>.
On Tue, Oct 16, 2012 at 7:24 PM, <Ca...@ibs-ag.com> wrote:
> Here are my files.. no, doesn't appear that I have them. Do these look right ?
>
> 0.9.2342.19200300.100.1.1.db
> 0.9.2342.19200300.100.1.1.lg
> 0.9.2342.19200300.100.1.1-uid.txt
> 0.9.2342.19200300.100.1.3.db
> 0.9.2342.19200300.100.1.3.lg
> 0.9.2342.19200300.100.1.3-mail.txt
> 0.9.2342.19200300.100.1.6.db
> 0.9.2342.19200300.100.1.6.lg
> 0.9.2342.19200300.100.1.6-roomNumber.txt
> 1.3.6.1.4.1.18060.0.4.1.2.3.db
> 1.3.6.1.4.1.18060.0.4.1.2.3.lg
> 1.3.6.1.4.1.18060.0.4.1.2.3-apachePresence.txt
> 1.3.6.1.4.1.18060.0.4.1.2.5.db
> 1.3.6.1.4.1.18060.0.4.1.2.5.lg
> 1.3.6.1.4.1.18060.0.4.1.2.50.db
> 1.3.6.1.4.1.18060.0.4.1.2.50-apacheRdn.txt
> 1.3.6.1.4.1.18060.0.4.1.2.5-apacheOneAlias.txt
> 1.3.6.1.4.1.18060.0.4.1.2.6.db
> 1.3.6.1.4.1.18060.0.4.1.2.6.lg
> 1.3.6.1.4.1.18060.0.4.1.2.6-apacheSubAlias.txt
> 1.3.6.1.4.1.18060.0.4.1.2.7.db
> 1.3.6.1.4.1.18060.0.4.1.2.7.lg
> 1.3.6.1.4.1.18060.0.4.1.2.7-apacheAlias.txt
> 1.3.6.1.4.1.42.2.27.8.1.23.db
> 1.3.6.1.4.1.42.2.27.8.1.23.lg
> 1.3.6.1.4.1.42.2.27.8.1.23-pwdPolicySubentry.txt
> 1.3.6.1.4.1.4203.666.1.7.db
> 1.3.6.1.4.1.4203.666.1.7.lg
> 1.3.6.1.4.1.4203.666.1.7-entryCSN.txt
> 1.3.6.1.4.1.5322.10.1.1.db
> 1.3.6.1.4.1.5322.10.1.1.lg
> 1.3.6.1.4.1.5322.10.1.1-krb5PrincipalName.txt
> 2.16.840.1.113730.3.1.241.db
> 2.16.840.1.113730.3.1.241.lg
> 2.16.840.1.113730.3.1.241-displayName.txt
> 2.16.840.1.113730.3.1.3.db
> 2.16.840.1.113730.3.1.3.lg
> 2.16.840.1.113730.3.1.3-employeeNumber.txt
> 2.5.18.5.db
> 2.5.18.5.lg
> 2.5.18.5-administrativeRole.txt
> 2.5.4.0.db
> 2.5.4.0.lg
> 2.5.4.0-objectClass.txt
> 2.5.4.10.db
> 2.5.4.10.lg
> 2.5.4.10-o.txt
> 2.5.4.11.db
> 2.5.4.11.lg
> 2.5.4.11-ou.txt
> 2.5.4.13.db
> 2.5.4.13.lg
> 2.5.4.13-description.txt
> 2.5.4.3.db
> 2.5.4.3.lg
> 2.5.4.31.db
> 2.5.4.31.lg
> 2.5.4.31-member.txt
> 2.5.4.3-cn.txt
> 2.5.4.4.db
> 2.5.4.4.lg
> 2.5.4.42.db
> 2.5.4.42.lg
> 2.5.4.42-givenName.txt
> 2.5.4.4-sn.txt
> master.db
>
looks fine
> Out of memory when searching for sn=Thomps*
>
ahhh, what is the size of heap memory allocated to JVM?
> jvm 1 | Server daemon died!
> jvm 1 | java.lang.OutOfMemoryError: Java heap space
> jvm 1 | at java.nio.CharBuffer.allocate(CharBuffer.java:312)
> jvm 1 | at java.nio.charset.CharsetEncoder.isLegalReplacement(CharsetEncoder.java:319)
> jvm 1 | at java.nio.charset.CharsetEncoder.replaceWith(CharsetEncoder.java:267)
> jvm 1 | at java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:186)
> jvm 1 | at java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:209)
> jvm 1 | at sun.nio.cs.SingleByteEncoder.<init>(SingleByteEncoder.java:39)
> jvm 1 | at sun.nio.cs.MS1252$Encoder.<init>(MS1252.java:115)
> jvm 1 | at sun.nio.cs.MS1252.newEncoder(MS1252.java:43)
> jvm 1 | at java.lang.StringCoding$StringEncoder.<init>(StringCoding.java:215)
> jvm 1 | at java.lang.StringCoding$StringEncoder.<init>(StringCoding.java:207)
> jvm 1 | at java.lang.StringCoding.encode(StringCoding.java:266)
> jvm 1 | at java.lang.StringCoding.encode(StringCoding.java:284)
> jvm 1 | at java.lang.String.getBytes(String.java:986)
> jvm 1 | at org.tanukisoftware.wrapper.WrapperManager.sendCommand(WrapperManager.java:3685)
> jvm 1 | at org.tanukisoftware.wrapper.WrapperManager.handleSocket(WrapperManager.java:3804)
> jvm 1 | at org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java:4084)
> jvm 1 | at java.lang.Thread.run(Thread.java:662)
> jvm 1 | [09:31:26] WARN [org.apache.directory.server.ldap.LdapProtocolHandler] - Unexpected exception forcing session to close: sending disconnect notice to client.
> jvm 1 | java.lang.OutOfMemoryError: Java heap space
> jvm 1 | at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:232)
> jvm 1 | at java.lang.StringCoding.encode(StringCoding.java:272)
> jvm 1 | at java.lang.String.getBytes(String.java:946)
> jvm 1 | at org.apache.directory.shared.util.Strings.getBytesUtf8(Strings.java:1587)
> jvm 1 | at org.apache.directory.shared.ldap.model.entry.StringValue.readExternal(StringValue.java:448)
> jvm 1 | at org.apache.directory.shared.ldap.model.entry.DefaultAttribute.readExternal(DefaultAttribute.java:2064)
> jvm 1 | at org.apache.directory.server.core.partition.impl.btree.jdbm.EntrySerializer.deserialize(EntrySerializer.java:219)
> jvm 1 | at jdbm.btree.BPage.deserialize(BPage.java:1255)
> jvm 1 | at jdbm.btree.BPage.deserialize(BPage.java:84)
> jvm 1 | at jdbm.recman.BaseRecordManager.fetch(BaseRecordManager.java:451)
> jvm 1 | at jdbm.recman.CacheRecordManager.fetch(CacheRecordManager.java:264)
> jvm 1 | at jdbm.btree.BPage.loadBPage(BPage.java:1017)
> jvm 1 | at jdbm.btree.BPage.find(BPage.java:315)
> jvm 1 | at jdbm.btree.BTree.browse(BTree.java:706)
> jvm 1 | at jdbm.btree.BTree.find(BTree.java:548)
> jvm 1 | at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.get(JdbmTable.java:312)
> jvm 1 | at org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.lookup(AbstractBTreePartition.java:1183)
> jvm 1 | at org.apache.directory.server.xdbm.search.evaluator.SubstringEvaluator.evaluate(SubstringEvaluator.java:135)
> jvm 1 | at org.apache.directory.server.xdbm.search.evaluator.AndEvaluator.evaluate(AndEvaluator.java:109)
> jvm 1 | at org.apache.directory.server.core.partition.impl.btree.EntryCursorAdaptor.get(EntryCursorAdaptor.java:146)
> jvm 1 | at org.apache.directory.server.core.partition.impl.btree.EntryCursorAdaptor.get(EntryCursorAdaptor.java:40)
> jvm 1 | at org.apache.directory.server.core.api.filtering.BaseEntryFilteringCursor.next(BaseEntryFilteringCursor.java:475)
> jvm 1 | at org.apache.directory.server.ldap.handlers.SearchHandler.readResults(SearchHandler.java:377)
> jvm 1 | at org.apache.directory.server.ldap.handlers.SearchHandler.doSimpleSearch(SearchHandler.java:830)
> jvm 1 | at org.apache.directory.server.ldap.handlers.SearchHandler.handleIgnoringReferrals(SearchHandler.java:1137)
> jvm 1 | at org.apache.directory.server.ldap.handlers.SearchHandler.handleWithReferrals(SearchHandler.java:1220)
> jvm 1 | at org.apache.directory.server.ldap.handlers.SearchHandler.handle(SearchHandler.java:227)
> jvm 1 | at org.apache.directory.server.ldap.handlers.SearchHandler.handle(SearchHandler.java:87)
> jvm 1 | at org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:209)
> jvm 1 | at org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:56)
> jvm 1 | at org.apache.mina.handler.demux.DemuxingIoHandler.messageReceived(DemuxingIoHandler.java:232)
> jvm 1 | at org.apache.directory.server.ldap.LdapProtocolHandler.messageReceived(LdapProtocolHandler.java:209)
>
>
>
>
> Regards,
> Carlo Accorsi
>
>
>
> -----Original Message-----
> From: ayyagarikiran@gmail.com [mailto:ayyagarikiran@gmail.com] On Behalf Of Kiran Ayyagari
> Sent: Tuesday, October 16, 2012 9:46 AM
> To: users@directory.apache.org
> Subject: Re: Possible regression for substring search on indexes?
>
> On Tue, Oct 16, 2012 at 6:35 PM, <Ca...@ibs-ag.com> wrote:
>> OK thanks for the quick response. I've built and tested (revision
>> 1398757) but is still taking several minutes to return substring search. I'm doing my searching via Studio and have exclusive access To the directory. Interestingly, it somehow knows that count right away but doesn't return anything for several mintues.
>
>>
>> I just swapped out the jar (with the one from a week ago) in this case bc I was hoping to validate the test without needing to rebuild and re-import all the users.
>> I would rebuild everything before for production but to test this correction, do I need to do this?
>>
> better to rebuild the server and overwrite the existing jars
>> Also, months ago we had an explicit index on entryUUID but we removed it when we didn't see it in the stock config.ldif.
>> entryCSN does have an index.
>>
> it is not needed anymore, cause we now use entryUUID as the *ID* of entry and master table contains these IDs as keys
>> These are the indexed attrs on the partition. After the ldif import they all had grown in size (on disk) so I'm fairly confident they're being populated.
>> Are we missing any that would work in conjunction with index searches? Thanks!
>>
>> apacheRdn
>> apacheSubLevel
>> apachePresence
>> apacheOneLevel
>> apacheOneAlias
>> apacheSubAlias
>> apacheAlias
>> entryCSN
>> o
>> krb5PrincipalName
>> objectClass
>> ou
>> uid
>> employeeNumber
>> displayName
>> cn
>> mail
>> roomNumber
>> pwdPolicySubEntry
>> member
>> description
>> givenName
>> sn
>> administrativeRole
>>
>>
> no, nothing is missing, otoh, do you really have the .db files related to ONE and SUB level indices (not ONE and SUB level *aliases*) curious cause they are no longer created/used by the server
>> Regards,
>> Carlo Accorsi
>>
>> -----Original Message-----
>> From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
>> Sent: Tuesday, October 16, 2012 7:33 AM
>> To: users@directory.apache.org
>> Subject: Re: Possible regression for substring search on indexes?
>>
>> Le 10/15/12 11:52 PM, Carlo.Accorsi@ibs-ag.com a écrit :
>>> Hi, we're using M9 built from the trunk on Wednesday of last week. We have our db setup as it has been for the past several months. There are 80k users.
>>>
>>> (sn=Thompson) returns in miliseconds
>>> (sn=Thompso*) returns after 3-4 minutes.
>>>
>>> This seems very much like an issue we had back and in June (and was fixed). Any help is most appreciated. We need this build for all the good work Karin did on the password policy.
>>>
>>> Thank you, Carlo Accorsi
>>>
>> It should be fixed with
>> http://svn.apache.org/viewvc?rev=1398737&view=rev
>>
>> Can you give it a try ?
>>
>> --
>> Regards,
>> Cordialement,
>> Emmanuel Lécharny
>> www.iktek.com
>>
>
>
>
> --
> Kiran Ayyagari
> http://keydap.com
--
Kiran Ayyagari
http://keydap.com
RE: Possible regression for substring search on indexes?
Posted by Ca...@ibs-ag.com.
Here are my files.. no, doesn't appear that I have them. Do these look right ?
0.9.2342.19200300.100.1.1.db
0.9.2342.19200300.100.1.1.lg
0.9.2342.19200300.100.1.1-uid.txt
0.9.2342.19200300.100.1.3.db
0.9.2342.19200300.100.1.3.lg
0.9.2342.19200300.100.1.3-mail.txt
0.9.2342.19200300.100.1.6.db
0.9.2342.19200300.100.1.6.lg
0.9.2342.19200300.100.1.6-roomNumber.txt
1.3.6.1.4.1.18060.0.4.1.2.3.db
1.3.6.1.4.1.18060.0.4.1.2.3.lg
1.3.6.1.4.1.18060.0.4.1.2.3-apachePresence.txt
1.3.6.1.4.1.18060.0.4.1.2.5.db
1.3.6.1.4.1.18060.0.4.1.2.5.lg
1.3.6.1.4.1.18060.0.4.1.2.50.db
1.3.6.1.4.1.18060.0.4.1.2.50-apacheRdn.txt
1.3.6.1.4.1.18060.0.4.1.2.5-apacheOneAlias.txt
1.3.6.1.4.1.18060.0.4.1.2.6.db
1.3.6.1.4.1.18060.0.4.1.2.6.lg
1.3.6.1.4.1.18060.0.4.1.2.6-apacheSubAlias.txt
1.3.6.1.4.1.18060.0.4.1.2.7.db
1.3.6.1.4.1.18060.0.4.1.2.7.lg
1.3.6.1.4.1.18060.0.4.1.2.7-apacheAlias.txt
1.3.6.1.4.1.42.2.27.8.1.23.db
1.3.6.1.4.1.42.2.27.8.1.23.lg
1.3.6.1.4.1.42.2.27.8.1.23-pwdPolicySubentry.txt
1.3.6.1.4.1.4203.666.1.7.db
1.3.6.1.4.1.4203.666.1.7.lg
1.3.6.1.4.1.4203.666.1.7-entryCSN.txt
1.3.6.1.4.1.5322.10.1.1.db
1.3.6.1.4.1.5322.10.1.1.lg
1.3.6.1.4.1.5322.10.1.1-krb5PrincipalName.txt
2.16.840.1.113730.3.1.241.db
2.16.840.1.113730.3.1.241.lg
2.16.840.1.113730.3.1.241-displayName.txt
2.16.840.1.113730.3.1.3.db
2.16.840.1.113730.3.1.3.lg
2.16.840.1.113730.3.1.3-employeeNumber.txt
2.5.18.5.db
2.5.18.5.lg
2.5.18.5-administrativeRole.txt
2.5.4.0.db
2.5.4.0.lg
2.5.4.0-objectClass.txt
2.5.4.10.db
2.5.4.10.lg
2.5.4.10-o.txt
2.5.4.11.db
2.5.4.11.lg
2.5.4.11-ou.txt
2.5.4.13.db
2.5.4.13.lg
2.5.4.13-description.txt
2.5.4.3.db
2.5.4.3.lg
2.5.4.31.db
2.5.4.31.lg
2.5.4.31-member.txt
2.5.4.3-cn.txt
2.5.4.4.db
2.5.4.4.lg
2.5.4.42.db
2.5.4.42.lg
2.5.4.42-givenName.txt
2.5.4.4-sn.txt
master.db
Out of memory when searching for sn=Thomps*
jvm 1 | Server daemon died!
jvm 1 | java.lang.OutOfMemoryError: Java heap space
jvm 1 | at java.nio.CharBuffer.allocate(CharBuffer.java:312)
jvm 1 | at java.nio.charset.CharsetEncoder.isLegalReplacement(CharsetEncoder.java:319)
jvm 1 | at java.nio.charset.CharsetEncoder.replaceWith(CharsetEncoder.java:267)
jvm 1 | at java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:186)
jvm 1 | at java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:209)
jvm 1 | at sun.nio.cs.SingleByteEncoder.<init>(SingleByteEncoder.java:39)
jvm 1 | at sun.nio.cs.MS1252$Encoder.<init>(MS1252.java:115)
jvm 1 | at sun.nio.cs.MS1252.newEncoder(MS1252.java:43)
jvm 1 | at java.lang.StringCoding$StringEncoder.<init>(StringCoding.java:215)
jvm 1 | at java.lang.StringCoding$StringEncoder.<init>(StringCoding.java:207)
jvm 1 | at java.lang.StringCoding.encode(StringCoding.java:266)
jvm 1 | at java.lang.StringCoding.encode(StringCoding.java:284)
jvm 1 | at java.lang.String.getBytes(String.java:986)
jvm 1 | at org.tanukisoftware.wrapper.WrapperManager.sendCommand(WrapperManager.java:3685)
jvm 1 | at org.tanukisoftware.wrapper.WrapperManager.handleSocket(WrapperManager.java:3804)
jvm 1 | at org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java:4084)
jvm 1 | at java.lang.Thread.run(Thread.java:662)
jvm 1 | [09:31:26] WARN [org.apache.directory.server.ldap.LdapProtocolHandler] - Unexpected exception forcing session to close: sending disconnect notice to client.
jvm 1 | java.lang.OutOfMemoryError: Java heap space
jvm 1 | at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:232)
jvm 1 | at java.lang.StringCoding.encode(StringCoding.java:272)
jvm 1 | at java.lang.String.getBytes(String.java:946)
jvm 1 | at org.apache.directory.shared.util.Strings.getBytesUtf8(Strings.java:1587)
jvm 1 | at org.apache.directory.shared.ldap.model.entry.StringValue.readExternal(StringValue.java:448)
jvm 1 | at org.apache.directory.shared.ldap.model.entry.DefaultAttribute.readExternal(DefaultAttribute.java:2064)
jvm 1 | at org.apache.directory.server.core.partition.impl.btree.jdbm.EntrySerializer.deserialize(EntrySerializer.java:219)
jvm 1 | at jdbm.btree.BPage.deserialize(BPage.java:1255)
jvm 1 | at jdbm.btree.BPage.deserialize(BPage.java:84)
jvm 1 | at jdbm.recman.BaseRecordManager.fetch(BaseRecordManager.java:451)
jvm 1 | at jdbm.recman.CacheRecordManager.fetch(CacheRecordManager.java:264)
jvm 1 | at jdbm.btree.BPage.loadBPage(BPage.java:1017)
jvm 1 | at jdbm.btree.BPage.find(BPage.java:315)
jvm 1 | at jdbm.btree.BTree.browse(BTree.java:706)
jvm 1 | at jdbm.btree.BTree.find(BTree.java:548)
jvm 1 | at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.get(JdbmTable.java:312)
jvm 1 | at org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.lookup(AbstractBTreePartition.java:1183)
jvm 1 | at org.apache.directory.server.xdbm.search.evaluator.SubstringEvaluator.evaluate(SubstringEvaluator.java:135)
jvm 1 | at org.apache.directory.server.xdbm.search.evaluator.AndEvaluator.evaluate(AndEvaluator.java:109)
jvm 1 | at org.apache.directory.server.core.partition.impl.btree.EntryCursorAdaptor.get(EntryCursorAdaptor.java:146)
jvm 1 | at org.apache.directory.server.core.partition.impl.btree.EntryCursorAdaptor.get(EntryCursorAdaptor.java:40)
jvm 1 | at org.apache.directory.server.core.api.filtering.BaseEntryFilteringCursor.next(BaseEntryFilteringCursor.java:475)
jvm 1 | at org.apache.directory.server.ldap.handlers.SearchHandler.readResults(SearchHandler.java:377)
jvm 1 | at org.apache.directory.server.ldap.handlers.SearchHandler.doSimpleSearch(SearchHandler.java:830)
jvm 1 | at org.apache.directory.server.ldap.handlers.SearchHandler.handleIgnoringReferrals(SearchHandler.java:1137)
jvm 1 | at org.apache.directory.server.ldap.handlers.SearchHandler.handleWithReferrals(SearchHandler.java:1220)
jvm 1 | at org.apache.directory.server.ldap.handlers.SearchHandler.handle(SearchHandler.java:227)
jvm 1 | at org.apache.directory.server.ldap.handlers.SearchHandler.handle(SearchHandler.java:87)
jvm 1 | at org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:209)
jvm 1 | at org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:56)
jvm 1 | at org.apache.mina.handler.demux.DemuxingIoHandler.messageReceived(DemuxingIoHandler.java:232)
jvm 1 | at org.apache.directory.server.ldap.LdapProtocolHandler.messageReceived(LdapProtocolHandler.java:209)
Regards,
Carlo Accorsi
-----Original Message-----
From: ayyagarikiran@gmail.com [mailto:ayyagarikiran@gmail.com] On Behalf Of Kiran Ayyagari
Sent: Tuesday, October 16, 2012 9:46 AM
To: users@directory.apache.org
Subject: Re: Possible regression for substring search on indexes?
On Tue, Oct 16, 2012 at 6:35 PM, <Ca...@ibs-ag.com> wrote:
> OK thanks for the quick response. I've built and tested (revision
> 1398757) but is still taking several minutes to return substring search. I'm doing my searching via Studio and have exclusive access To the directory. Interestingly, it somehow knows that count right away but doesn't return anything for several mintues.
>
> I just swapped out the jar (with the one from a week ago) in this case bc I was hoping to validate the test without needing to rebuild and re-import all the users.
> I would rebuild everything before for production but to test this correction, do I need to do this?
>
better to rebuild the server and overwrite the existing jars
> Also, months ago we had an explicit index on entryUUID but we removed it when we didn't see it in the stock config.ldif.
> entryCSN does have an index.
>
it is not needed anymore, cause we now use entryUUID as the *ID* of entry and master table contains these IDs as keys
> These are the indexed attrs on the partition. After the ldif import they all had grown in size (on disk) so I'm fairly confident they're being populated.
> Are we missing any that would work in conjunction with index searches? Thanks!
>
> apacheRdn
> apacheSubLevel
> apachePresence
> apacheOneLevel
> apacheOneAlias
> apacheSubAlias
> apacheAlias
> entryCSN
> o
> krb5PrincipalName
> objectClass
> ou
> uid
> employeeNumber
> displayName
> cn
> mail
> roomNumber
> pwdPolicySubEntry
> member
> description
> givenName
> sn
> administrativeRole
>
>
no, nothing is missing, otoh, do you really have the .db files related to ONE and SUB level indices (not ONE and SUB level *aliases*) curious cause they are no longer created/used by the server
> Regards,
> Carlo Accorsi
>
> -----Original Message-----
> From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
> Sent: Tuesday, October 16, 2012 7:33 AM
> To: users@directory.apache.org
> Subject: Re: Possible regression for substring search on indexes?
>
> Le 10/15/12 11:52 PM, Carlo.Accorsi@ibs-ag.com a écrit :
>> Hi, we're using M9 built from the trunk on Wednesday of last week. We have our db setup as it has been for the past several months. There are 80k users.
>>
>> (sn=Thompson) returns in miliseconds
>> (sn=Thompso*) returns after 3-4 minutes.
>>
>> This seems very much like an issue we had back and in June (and was fixed). Any help is most appreciated. We need this build for all the good work Karin did on the password policy.
>>
>> Thank you, Carlo Accorsi
>>
> It should be fixed with
> http://svn.apache.org/viewvc?rev=1398737&view=rev
>
> Can you give it a try ?
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
--
Kiran Ayyagari
http://keydap.com
Re: Possible regression for substring search on indexes?
Posted by Kiran Ayyagari <ka...@apache.org>.
On Tue, Oct 16, 2012 at 6:35 PM, <Ca...@ibs-ag.com> wrote:
> OK thanks for the quick response. I've built and tested (revision 1398757) but is still taking several minutes to return substring search. I'm doing my searching via Studio and have exclusive access
> To the directory. Interestingly, it somehow knows that count right away but doesn't return anything for several mintues.
>
> I just swapped out the jar (with the one from a week ago) in this case bc I was hoping to validate the test without needing to rebuild and re-import all the users.
> I would rebuild everything before for production but to test this correction, do I need to do this?
>
better to rebuild the server and overwrite the existing jars
> Also, months ago we had an explicit index on entryUUID but we removed it when we didn't see it in the stock config.ldif.
> entryCSN does have an index.
>
it is not needed anymore, cause we now use entryUUID as the *ID* of
entry and master table contains these IDs as keys
> These are the indexed attrs on the partition. After the ldif import they all had grown in size (on disk) so I'm fairly confident they're being populated.
> Are we missing any that would work in conjunction with index searches? Thanks!
>
> apacheRdn
> apacheSubLevel
> apachePresence
> apacheOneLevel
> apacheOneAlias
> apacheSubAlias
> apacheAlias
> entryCSN
> o
> krb5PrincipalName
> objectClass
> ou
> uid
> employeeNumber
> displayName
> cn
> mail
> roomNumber
> pwdPolicySubEntry
> member
> description
> givenName
> sn
> administrativeRole
>
>
no, nothing is missing, otoh, do you really have the .db files related
to ONE and SUB level indices (not ONE and SUB level *aliases*)
curious cause they are no longer created/used by the server
> Regards,
> Carlo Accorsi
>
> -----Original Message-----
> From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
> Sent: Tuesday, October 16, 2012 7:33 AM
> To: users@directory.apache.org
> Subject: Re: Possible regression for substring search on indexes?
>
> Le 10/15/12 11:52 PM, Carlo.Accorsi@ibs-ag.com a écrit :
>> Hi, we're using M9 built from the trunk on Wednesday of last week. We have our db setup as it has been for the past several months. There are 80k users.
>>
>> (sn=Thompson) returns in miliseconds
>> (sn=Thompso*) returns after 3-4 minutes.
>>
>> This seems very much like an issue we had back and in June (and was fixed). Any help is most appreciated. We need this build for all the good work Karin did on the password policy.
>>
>> Thank you, Carlo Accorsi
>>
> It should be fixed with http://svn.apache.org/viewvc?rev=1398737&view=rev
>
> Can you give it a try ?
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
--
Kiran Ayyagari
http://keydap.com
RE: Possible regression for substring search on indexes?
Posted by Ca...@ibs-ag.com.
OK thanks for the quick response. I've built and tested (revision 1398757) but is still taking several minutes to return substring search. I'm doing my searching via Studio and have exclusive access
To the directory. Interestingly, it somehow knows that count right away but doesn't return anything for several mintues.
I just swapped out the jar (with the one from a week ago) in this case bc I was hoping to validate the test without needing to rebuild and re-import all the users.
I would rebuild everything before for production but to test this correction, do I need to do this?
Also, months ago we had an explicit index on entryUUID but we removed it when we didn't see it in the stock config.ldif.
entryCSN does have an index.
These are the indexed attrs on the partition. After the ldif import they all had grown in size (on disk) so I'm fairly confident they're being populated.
Are we missing any that would work in conjunction with index searches? Thanks!
apacheRdn
apacheSubLevel
apachePresence
apacheOneLevel
apacheOneAlias
apacheSubAlias
apacheAlias
entryCSN
o
krb5PrincipalName
objectClass
ou
uid
employeeNumber
displayName
cn
mail
roomNumber
pwdPolicySubEntry
member
description
givenName
sn
administrativeRole
Regards,
Carlo Accorsi
-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
Sent: Tuesday, October 16, 2012 7:33 AM
To: users@directory.apache.org
Subject: Re: Possible regression for substring search on indexes?
Le 10/15/12 11:52 PM, Carlo.Accorsi@ibs-ag.com a écrit :
> Hi, we're using M9 built from the trunk on Wednesday of last week. We have our db setup as it has been for the past several months. There are 80k users.
>
> (sn=Thompson) returns in miliseconds
> (sn=Thompso*) returns after 3-4 minutes.
>
> This seems very much like an issue we had back and in June (and was fixed). Any help is most appreciated. We need this build for all the good work Karin did on the password policy.
>
> Thank you, Carlo Accorsi
>
It should be fixed with http://svn.apache.org/viewvc?rev=1398737&view=rev
Can you give it a try ?
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Possible regression for substring search on indexes?
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 10/15/12 11:52 PM, Carlo.Accorsi@ibs-ag.com a écrit :
> Hi, we're using M9 built from the trunk on Wednesday of last week. We have our db setup as it has been for the past several months. There are 80k users.
>
> (sn=Thompson) returns in miliseconds
> (sn=Thompso*) returns after 3-4 minutes.
>
> This seems very much like an issue we had back and in June (and was fixed). Any help is most appreciated. We need this build for all the good work Karin did on the password policy.
>
> Thank you, Carlo Accorsi
>
It should be fixed with http://svn.apache.org/viewvc?rev=1398737&view=rev
Can you give it a try ?
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Possible regression for substring search on indexes?
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 10/15/12 11:52 PM, Carlo.Accorsi@ibs-ag.com a écrit :
> Hi, we're using M9 built from the trunk on Wednesday of last week. We have our db setup as it has been for the past several months. There are 80k users.
>
> (sn=Thompson) returns in miliseconds
> (sn=Thompso*) returns after 3-4 minutes.
>
> This seems very much like an issue we had back and in June (and was fixed). Any help is most appreciated. We need this build for all the good work Karin did on the password policy.
I'll investigate this.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Possible regression for substring search on indexes?
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 10/15/12 11:52 PM, Carlo.Accorsi@ibs-ag.com a écrit :
> Hi, we're using M9 built from the trunk on Wednesday of last week. We have our db setup as it has been for the past several months. There are 80k users.
>
> (sn=Thompson) returns in miliseconds
> (sn=Thompso*) returns after 3-4 minutes.
>
> This seems very much like an issue we had back and in June (and was fixed). Any help is most appreciated. We need this build for all the good work Karin did on the password policy.
>
> Thank you, Carlo Accorsi
>
I have created a JIRA for this problem :
https://issues.apache.org/jira/browse/DIRSERVER-1751
Since the extensive modifications I have made last month, I confirm that
substring filters are not used during the evaluation and that leads to a
full scan. This is my mistake, I'm working on fixing it atm.
Thanks for the report !
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com