You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@directory.apache.org by Gabriel Albano <ga...@pilixo.com> on 2014/05/05 22:06:00 UTC
Re: Problem with indexes...
I am having a similar problem. Search before adding the index:
#!SEARCH REQUEST (161) OK
#!CONNECTION ldap://stage-ds.pilixo.com:10389
#!DATE 2014-05-05T19:59:01.677
# LDAP URL : ldap://
stage-ds.pilixo.com:10389/ou=organizations,dc=pilixo?*?sub?(displayName=gabriel*)
# command line : ldapsearch -H ldap://stage-ds.pilixo.com:10389 -x -D
"uid=admin,ou=system" -W -b "ou=organizations,dc=pilixo" -s sub -a always
-z 100000 "(displayName=gabriel*)" "*"
# baseObject : ou=organizations,dc=pilixo
# scope : wholeSubtree (2)
# derefAliases : derefAlways (3)
# sizeLimit : 100000
# timeLimit : 0
# typesOnly : False
# filter : (displayName=gabriel*)
# attributes : *
#!SEARCH RESULT DONE (161) OK
#!CONNECTION ldap://stage-ds.pilixo.com:10389
#!DATE 2014-05-05T19:59:01.825
# numEntries : 2
Added index:
dn: ads-indexAttributeId=displayName,ou=indexes,ads-partitionId=pilixo,ou=pa
rtitions,ads-directoryServiceId=default,ou=config
objectclass: top
objectclass: ads-jdbmIndex
objectclass: ads-base
objectclass: ads-index
ads-indexattributeid: displayName
ads-indexhasreverse: FALSE
ads-enabled: TRUE
Restarted Server and searched again:
#!SEARCH REQUEST (173) OK
#!CONNECTION ldap://stage-ds.pilixo.com:10389
#!DATE 2014-05-05T20:02:16.850
# LDAP URL : ldap://
stage-ds.pilixo.com:10389/ou=organizations,dc=pilixo?*?sub?(displayName=gabriel*)
# command line : ldapsearch -H ldap://stage-ds.pilixo.com:10389 -x -D
"uid=admin,ou=system" -W -b "ou=organizations,dc=pilixo" -s sub -a always
-z 100000 "(displayName=gabriel*)" "*"
# baseObject : ou=organizations,dc=pilixo
# scope : wholeSubtree (2)
# derefAliases : derefAlways (3)
# sizeLimit : 100000
# timeLimit : 0
# typesOnly : False
# filter : (displayName=gabriel*)
# attributes : *
#!SEARCH RESULT DONE (173) OK
#!CONNECTION ldap://stage-ds.pilixo.com:10389
#!DATE 2014-05-05T20:02:16.985
# numEntries : 0
Running 2.0.0-M16, no errors in log as far as I can tell. Furthermore, if
I modify existing or add new entries, they get picked up by the new index,
but existing entries are not.
Thanks,
Gabriel
On Wed, Apr 23, 2014 at 3:09 AM, Kiran Ayyagari <ka...@apache.org>wrote:
> On Wed, Apr 23, 2014 at 12:29 PM, Chris Betts <pe...@gmail.com> wrote:
>
> > Hi Kirin,
> >
> > I get exactly the same behaviour with the 'cn' attribute, so I
> > suspect the problem is something to do with creating the indexes after
> the
> > partition has been created rather than the attribute.
> >
> > Curiously, using the same data I successfully created a directory
> > with working indices on an AWS linux image, but in that case I created
> the
> > indices at the same time I created the partition.
> >
> > Here's my current 'not working' config from my laptop (OSX / java 6 /
> > apacheds 2) which returns 0 entries when searching for (cn=*):
> >
> you need to restart the server after adding the new index/indices in
> configuration.
> Can you try the search after restarting? note that server may take a while
> to start due to indexing the data
> (btw, configuration is looking good)
>
> otoh, this is strange, the search shouldn't get impacted before restart,
> what version of the server are you using?
>
> >
> > #!SEARCH REQUEST (146) OK
> >
> > #!CONNECTION ldap://localhost:10389
> >
> > #!DATE 2014-04-23T06:43:15.774
> >
> > # LDAP URL :
> > ldap://localhost:10389/o=groupmind,c=au?objectClass?sub?(cn=*)
> >
> > # command line : ldapsearch -H ldap://localhost:10389 -x -D
> > "uid=admin,ou=system" -W -b "o=groupmind,c=au" -s sub -a always -z 1000
> > "(cn=*)" "objectClass"
> >
> > # baseObject : o=groupmind,c=au
> >
> > # scope : wholeSubtree (2)
> >
> > # derefAliases : derefAlways (3)
> >
> > # sizeLimit : 1000
> >
> > # timeLimit : 0
> >
> > # typesOnly : False
> >
> > # filter : (cn=*)
> >
> > # attributes : objectClass
> >
> >
> > #!SEARCH RESULT DONE (146) OK
> >
> > #!CONNECTION ldap://localhost:10389
> >
> > #!DATE 2014-04-23T06:43:15.776
> >
> > # numEntries : 0
> >
> >
> > ... hope this helps; however since the AWS version is working fine it's
> no
> > longer super-important to solve - I'll try blowing away the partition and
> > recreating it along with the indexes at the same time, and see if that
> > solves the problem :-).
> >
> sure, but see if restarting the server fixes this problem and let me know.
>
> > cheers,
> >
> > - Chris
> >
> >
> > -----
> > *Dr Christopher Betts*
> > Australian Cloud Identity
> > http://cloudidentity.com.au
> > m: 0408 533 456
> >
> >
> > On 23 April 2014 15:13, Kiran Ayyagari <ka...@apache.org> wrote:
> >
> >> can you share the partition configuration and the schema definition of
> >> the attribute 'groupmindId'?
> >>
> >>
> >> On Wed, Apr 23, 2014 at 5:57 AM, Chris Betts <pe...@gmail.com> wrote:
> >>
> >>> Hi Folks,
> >>>
> >>> I'm trying to migrate a project from openldap to apacheds, and I've
> >>> hit
> >>> a weird problem with indexing... I suspect I'm doing something wrong,
> and
> >>> I'm hoping someone can help :-).
> >>>
> >>> Basically, I have a DIT with a subtree of entries containing a
> custom
> >>> schema attribute 'groupmindID', which is a case sensitive ID code. The
> >>> application uses this groupmindID for searching entries.
> >>>
> >>> After importing the schema and starting things up everything worked,
> >>> but
> >>> it was pretty slow, so I thought I'd index various attributes, which I
> >>> did
> >>> using the 'Partitions->Configuration' pane, at which point apacheds
> >>> stopped
> >>> finding them at all!
> >>>
> >>> I've replicated the problem in apacheds studio; here's some output.
> >>>
> >>> #!SEARCH REQUEST (212) OK
> >>> #!CONNECTION ldap://localhost:10389
> >>> #!DATE 2014-04-23T00:04:42.092
> >>> # LDAP URL :
> >>>
> >>>
> ldap://localhost:10389/cn=topics,o=groupmind,c=au?objectClass?sub?(groupmindID=*)
> >>> # command line : ldapsearch -H ldap://localhost:10389 -x -D
> >>> "uid=admin,ou=system" -W -b "cn=topics,o=groupmind,c=au" -s sub -a
> always
> >>> -z 1000 "(groupmindID=*)" "objectClass"
> >>> # baseObject : cn=topics,o=groupmind,c=au
> >>> # scope : wholeSubtree (2)
> >>> # derefAliases : derefAlways (3)
> >>> # sizeLimit : 1000
> >>> # timeLimit : 0
> >>> # typesOnly : False
> >>> # filter : (groupmindID=*)
> >>> # attributes : objectClass
> >>>
> >>> #!SEARCH RESULT DONE (212) OK
> >>> #!CONNECTION ldap://localhost:10389
> >>> #!DATE 2014-04-23T00:04:42.102
> >>> # numEntries : 0
> >>>
> >>> ***
> >>> *** Delete index on 'groupmindID' and restart server, rerun test:
> >>> ***
> >>>
> >>> #!SEARCH REQUEST (253) OK
> >>> #!CONNECTION ldap://localhost:10389
> >>> #!DATE 2014-04-23T00:09:42.999
> >>> # LDAP URL :
> >>>
> >>>
> ldap://localhost:10389/cn=topics,o=groupmind,c=au?objectClass?sub?(groupmindID=*)
> >>> # command line : ldapsearch -H ldap://localhost:10389 -x -D
> >>> "uid=admin,ou=system" -W -b "cn=topics,o=groupmind,c=au" -s sub -a
> always
> >>> -z 1000 "(groupmindID=*)" "objectClass"
> >>> # baseObject : cn=topics,o=groupmind,c=au
> >>> # scope : wholeSubtree (2)
> >>> # derefAliases : derefAlways (3)
> >>> # sizeLimit : 1000
> >>> # timeLimit : 0
> >>> # typesOnly : False
> >>> # filter : (groupmindID=*)
> >>> # attributes : objectClass
> >>>
> >>> #!SEARCH RESULT DONE (253) OK
> >>> #!CONNECTION ldap://localhost:10389
> >>> #!DATE 2014-04-23T00:09:46.856
> >>> # numEntries : 1000
> >>>
> >>>
> >>>
> >>> ... I've had a play around, and I can get the same behaviour even from
> >>> 'normal' attributes like 'cn', so I suspect I'm doing something bone
> >>> headed
> >>> with the way I've set the indexes up - apologies if it's obvious, I'm
> >>> very
> >>> new to apacheds and apacheds studio!
> >>>
> >>> cheers,
> >>>
> >>> - Chris
> >>>
> >>> -----
> >>> *Dr Christopher Betts*
> >>>
> >>> Australian Cloud Identity
> >>> http://cloudidentity.com.au
> >>> m: 0408 533 456
> >>>
> >>
> >>
> >>
> >> --
> >> Kiran Ayyagari
> >> http://keydap.com
> >>
> >
> >
>
>
> --
> Kiran Ayyagari
> http://keydap.com
>
Re: Problem with indexes...
Posted by Kiran Ayyagari <ka...@apache.org>.
On Mon, May 12, 2014 at 9:08 PM, Mark D <te...@debusschere.com> wrote:
> Kiran, I don't have a reliable test. I created a new snapshot build from
> trunk and
> I noticed the following in my logs. I don't recall seeing these before.
> I looked
> at previous logs and didn't find examples but I didn't have a really good
> set to compare with.
>
> [11:10:22] INFO
> [org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmPartition]
> - building the index for attribute type attributetype (
> 0.9.2342.19200300.100.1.1 NAME ( 'uid' 'userid' )
>
> Restarts appear to be working without loosing search, good news.
>
> I would like to figure out what is causing this ? Guessing I have some
> index I auto created that is not defined / used in the partition?
>
> this happens when a new index was configured _after_ the partition was
loaded/added with some data
and the server detects this during next restart and builds the index.
If you are curious to know why the search started working now:
Earlier the index database file was created (after configuring and
restarting the server) but the existing
data was not indexed into this file, so the index scan on say 'uid'
attribute returned zero results (note that
a full table scan is avoided cause the index file is present).
now the old/existing data gets indexed and the index file is appropriately
filled with that data resulting in
correct search results.
[11:10:22] WARN
> [org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmPartition]
> - Failed to delete unused index file
> C:\Users\mdeb\workspace-jimbra\Jaimbra\work\apacheds\jimbra\partitions\jimbra\2.5.18.5.db
>
>
> > this is not related to the above issue (cause once data is in the index
> it
> will survive restarts),
> do you have a test case that I can run to reproduce?
>
> Thanks, Makr.
>
--
Kiran Ayyagari
http://keydap.com
Re: Problem with indexes...
Posted by Mark D <te...@debusschere.com>.
Kiran, I don't have a reliable test. I created a new snapshot build from trunk and
I noticed the following in my logs. I don't recall seeing these before. I looked
at previous logs and didn't find examples but I didn't have a really good set to compare with.
[11:10:22] INFO [org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmPartition] - building the index for attribute type attributetype ( 0.9.2342.19200300.100.1.1 NAME ( 'uid' 'userid' )
Restarts appear to be working without loosing search, good news.
I would like to figure out what is causing this ? Guessing I have some index I auto created that is not defined / used in the partition?
[11:10:22] WARN [org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmPartition] - Failed to delete unused index file C:\Users\mdeb\workspace-jimbra\Jaimbra\work\apacheds\jimbra\partitions\jimbra\2.5.18.5.db
> this is not related to the above issue (cause once data is in the index it
will survive restarts),
do you have a test case that I can run to reproduce?
Thanks, Makr.
Re: Problem with indexes...
Posted by Kiran Ayyagari <ka...@apache.org>.
On Thu, May 8, 2014 at 9:26 PM, Mark D <te...@debusschere.com> wrote:
> Was this issue related to M16 or has it been outstanding?
>
> My app dynamically builds new partitions with index definitions at runtime.
> It adds the partitions configuration definitions then creates the JDMB and
> adds to active service.
> Works well, partition is created, can add data, searches work, but when I
> restart searches don't work at all.
>
> this is not related to the above issue (cause once data is in the index it
will survive restarts),
do you have a test case that I can run to reproduce?
> Trying snapshot build now.
>
> Thanks, Mark.
>
> ----- Original Message -----
> From: "Kiran Ayyagari" <ka...@apache.org>
> To: users@directory.apache.org
> Sent: Tuesday, May 6, 2014 8:43:45 AM
> Subject: Re: Problem with indexes...
>
> I have filed a bug report[1] and working on the fix
>
> [1] https://issues.apache.org/jira/browse/DIRSERVER-1976
>
>
>
> On Tue, May 6, 2014 at 1:36 AM, Gabriel Albano <gabriel.albano@pilixo.com
> >wrote:
>
> > I am having a similar problem. Search before adding the index:
> >
> > #!SEARCH REQUEST (161) OK
> > #!CONNECTION ldap://stage-ds.pilixo.com:10389
> > #!DATE 2014-05-05T19:59:01.677
> > # LDAP URL : ldap://
> >
> >
> stage-ds.pilixo.com:10389/ou=organizations,dc=pilixo?*?sub?(displayName=gabriel*)
> > # command line : ldapsearch -H ldap://stage-ds.pilixo.com:10389 -x -D
> > "uid=admin,ou=system" -W -b "ou=organizations,dc=pilixo" -s sub -a always
> > -z 100000 "(displayName=gabriel*)" "*"
> > # baseObject : ou=organizations,dc=pilixo
> > # scope : wholeSubtree (2)
> > # derefAliases : derefAlways (3)
> > # sizeLimit : 100000
> > # timeLimit : 0
> > # typesOnly : False
> > # filter : (displayName=gabriel*)
> > # attributes : *
> >
> > #!SEARCH RESULT DONE (161) OK
> > #!CONNECTION ldap://stage-ds.pilixo.com:10389
> > #!DATE 2014-05-05T19:59:01.825
> > # numEntries : 2
> >
> >
> > Added index:
> >
> > dn:
> > ads-indexAttributeId=displayName,ou=indexes,ads-partitionId=pilixo,ou=pa
> > rtitions,ads-directoryServiceId=default,ou=config
> > objectclass: top
> > objectclass: ads-jdbmIndex
> > objectclass: ads-base
> > objectclass: ads-index
> > ads-indexattributeid: displayName
> > ads-indexhasreverse: FALSE
> > ads-enabled: TRUE
> >
> >
> > Restarted Server and searched again:
> >
> > #!SEARCH REQUEST (173) OK
> > #!CONNECTION ldap://stage-ds.pilixo.com:10389
> > #!DATE 2014-05-05T20:02:16.850
> > # LDAP URL : ldap://
> >
> >
> stage-ds.pilixo.com:10389/ou=organizations,dc=pilixo?*?sub?(displayName=gabriel*)
> > # command line : ldapsearch -H ldap://stage-ds.pilixo.com:10389 -x -D
> > "uid=admin,ou=system" -W -b "ou=organizations,dc=pilixo" -s sub -a always
> > -z 100000 "(displayName=gabriel*)" "*"
> > # baseObject : ou=organizations,dc=pilixo
> > # scope : wholeSubtree (2)
> > # derefAliases : derefAlways (3)
> > # sizeLimit : 100000
> > # timeLimit : 0
> > # typesOnly : False
> > # filter : (displayName=gabriel*)
> > # attributes : *
> >
> > #!SEARCH RESULT DONE (173) OK
> > #!CONNECTION ldap://stage-ds.pilixo.com:10389
> > #!DATE 2014-05-05T20:02:16.985
> > # numEntries : 0
> >
> > Running 2.0.0-M16, no errors in log as far as I can tell. Furthermore,
> if
> > I modify existing or add new entries, they get picked up by the new
> index,
> > but existing entries are not.
> >
> > Thanks,
> > Gabriel
> >
> >
> >
> > On Wed, Apr 23, 2014 at 3:09 AM, Kiran Ayyagari <kayyagari@apache.org
> > >wrote:
> >
> > > On Wed, Apr 23, 2014 at 12:29 PM, Chris Betts <pe...@gmail.com>
> wrote:
> > >
> > > > Hi Kirin,
> > > >
> > > > I get exactly the same behaviour with the 'cn' attribute, so I
> > > > suspect the problem is something to do with creating the indexes
> after
> > > the
> > > > partition has been created rather than the attribute.
> > > >
> > > > Curiously, using the same data I successfully created a
> directory
> > > > with working indices on an AWS linux image, but in that case I
> created
> > > the
> > > > indices at the same time I created the partition.
> > > >
> > > > Here's my current 'not working' config from my laptop (OSX / java
> > 6 /
> > > > apacheds 2) which returns 0 entries when searching for (cn=*):
> > > >
> > > you need to restart the server after adding the new index/indices in
> > > configuration.
> > > Can you try the search after restarting? note that server may take a
> > while
> > > to start due to indexing the data
> > > (btw, configuration is looking good)
> > >
> > > otoh, this is strange, the search shouldn't get impacted before
> restart,
> > > what version of the server are you using?
> > >
> > > >
> > > > #!SEARCH REQUEST (146) OK
> > > >
> > > > #!CONNECTION ldap://localhost:10389
> > > >
> > > > #!DATE 2014-04-23T06:43:15.774
> > > >
> > > > # LDAP URL :
> > > > ldap://localhost:10389/o=groupmind,c=au?objectClass?sub?(cn=*)
> > > >
> > > > # command line : ldapsearch -H ldap://localhost:10389 -x -D
> > > > "uid=admin,ou=system" -W -b "o=groupmind,c=au" -s sub -a always -z
> 1000
> > > > "(cn=*)" "objectClass"
> > > >
> > > > # baseObject : o=groupmind,c=au
> > > >
> > > > # scope : wholeSubtree (2)
> > > >
> > > > # derefAliases : derefAlways (3)
> > > >
> > > > # sizeLimit : 1000
> > > >
> > > > # timeLimit : 0
> > > >
> > > > # typesOnly : False
> > > >
> > > > # filter : (cn=*)
> > > >
> > > > # attributes : objectClass
> > > >
> > > >
> > > > #!SEARCH RESULT DONE (146) OK
> > > >
> > > > #!CONNECTION ldap://localhost:10389
> > > >
> > > > #!DATE 2014-04-23T06:43:15.776
> > > >
> > > > # numEntries : 0
> > > >
> > > >
> > > > ... hope this helps; however since the AWS version is working fine
> it's
> > > no
> > > > longer super-important to solve - I'll try blowing away the partition
> > and
> > > > recreating it along with the indexes at the same time, and see if
> that
> > > > solves the problem :-).
> > > >
> > > sure, but see if restarting the server fixes this problem and let me
> > know.
> > >
> > > > cheers,
> > > >
> > > > - Chris
> > > >
> > > >
> > > > -----
> > > > *Dr Christopher Betts*
> > > > Australian Cloud Identity
> > > > http://cloudidentity.com.au
> > > > m: 0408 533 456
> > > >
> > > >
> > > > On 23 April 2014 15:13, Kiran Ayyagari <ka...@apache.org> wrote:
> > > >
> > > >> can you share the partition configuration and the schema definition
> of
> > > >> the attribute 'groupmindId'?
> > > >>
> > > >>
> > > >> On Wed, Apr 23, 2014 at 5:57 AM, Chris Betts <pe...@gmail.com>
> > wrote:
> > > >>
> > > >>> Hi Folks,
> > > >>>
> > > >>> I'm trying to migrate a project from openldap to apacheds, and
> > I've
> > > >>> hit
> > > >>> a weird problem with indexing... I suspect I'm doing something
> wrong,
> > > and
> > > >>> I'm hoping someone can help :-).
> > > >>>
> > > >>> Basically, I have a DIT with a subtree of entries containing a
> > > custom
> > > >>> schema attribute 'groupmindID', which is a case sensitive ID code.
> > The
> > > >>> application uses this groupmindID for searching entries.
> > > >>>
> > > >>> After importing the schema and starting things up everything
> > worked,
> > > >>> but
> > > >>> it was pretty slow, so I thought I'd index various attributes,
> which
> > I
> > > >>> did
> > > >>> using the 'Partitions->Configuration' pane, at which point apacheds
> > > >>> stopped
> > > >>> finding them at all!
> > > >>>
> > > >>> I've replicated the problem in apacheds studio; here's some
> > output.
> > > >>>
> > > >>> #!SEARCH REQUEST (212) OK
> > > >>> #!CONNECTION ldap://localhost:10389
> > > >>> #!DATE 2014-04-23T00:04:42.092
> > > >>> # LDAP URL :
> > > >>>
> > > >>>
> > >
> >
> ldap://localhost:10389/cn=topics,o=groupmind,c=au?objectClass?sub?(groupmindID=*)
> > > >>> # command line : ldapsearch -H ldap://localhost:10389 -x -D
> > > >>> "uid=admin,ou=system" -W -b "cn=topics,o=groupmind,c=au" -s sub -a
> > > always
> > > >>> -z 1000 "(groupmindID=*)" "objectClass"
> > > >>> # baseObject : cn=topics,o=groupmind,c=au
> > > >>> # scope : wholeSubtree (2)
> > > >>> # derefAliases : derefAlways (3)
> > > >>> # sizeLimit : 1000
> > > >>> # timeLimit : 0
> > > >>> # typesOnly : False
> > > >>> # filter : (groupmindID=*)
> > > >>> # attributes : objectClass
> > > >>>
> > > >>> #!SEARCH RESULT DONE (212) OK
> > > >>> #!CONNECTION ldap://localhost:10389
> > > >>> #!DATE 2014-04-23T00:04:42.102
> > > >>> # numEntries : 0
> > > >>>
> > > >>> ***
> > > >>> *** Delete index on 'groupmindID' and restart server, rerun test:
> > > >>> ***
> > > >>>
> > > >>> #!SEARCH REQUEST (253) OK
> > > >>> #!CONNECTION ldap://localhost:10389
> > > >>> #!DATE 2014-04-23T00:09:42.999
> > > >>> # LDAP URL :
> > > >>>
> > > >>>
> > >
> >
> ldap://localhost:10389/cn=topics,o=groupmind,c=au?objectClass?sub?(groupmindID=*)
> > > >>> # command line : ldapsearch -H ldap://localhost:10389 -x -D
> > > >>> "uid=admin,ou=system" -W -b "cn=topics,o=groupmind,c=au" -s sub -a
> > > always
> > > >>> -z 1000 "(groupmindID=*)" "objectClass"
> > > >>> # baseObject : cn=topics,o=groupmind,c=au
> > > >>> # scope : wholeSubtree (2)
> > > >>> # derefAliases : derefAlways (3)
> > > >>> # sizeLimit : 1000
> > > >>> # timeLimit : 0
> > > >>> # typesOnly : False
> > > >>> # filter : (groupmindID=*)
> > > >>> # attributes : objectClass
> > > >>>
> > > >>> #!SEARCH RESULT DONE (253) OK
> > > >>> #!CONNECTION ldap://localhost:10389
> > > >>> #!DATE 2014-04-23T00:09:46.856
> > > >>> # numEntries : 1000
> > > >>>
> > > >>>
> > > >>>
> > > >>> ... I've had a play around, and I can get the same behaviour even
> > from
> > > >>> 'normal' attributes like 'cn', so I suspect I'm doing something
> bone
> > > >>> headed
> > > >>> with the way I've set the indexes up - apologies if it's obvious,
> I'm
> > > >>> very
> > > >>> new to apacheds and apacheds studio!
> > > >>>
> > > >>> cheers,
> > > >>>
> > > >>> - Chris
> > > >>>
> > > >>> -----
> > > >>> *Dr Christopher Betts*
> > > >>>
> > > >>> Australian Cloud Identity
> > > >>> http://cloudidentity.com.au
> > > >>> m: 0408 533 456
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Kiran Ayyagari
> > > >> http://keydap.com
> > > >>
> > > >
> > > >
> > >
> > >
> > > --
> > > Kiran Ayyagari
> > > http://keydap.com
> > >
> >
>
>
>
> --
> Kiran Ayyagari
> http://keydap.com
>
--
Kiran Ayyagari
http://keydap.com
Re: Problem with indexes...
Posted by Mark D <te...@debusschere.com>.
Thanks Emmanuel, I will switch to using nanoTime() for measurements.
The performance numbers you quote would far exceed my needs. I need basic
fast reads with occasional small writes.
Is concurrency issue a problem on partition scope, DN scope, is there
any scope to previous discussion of issues with reading / writing.
Re: Problem with indexes...
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 17/05/2014 06:40, Mark D a écrit :
> Emmanuel, thanks for the detailed information. Kiran also had some feedback.
> The build I am using from trunk seems to be working much better. There was serious
> issue with previous build as the method I am using for indexes resulted in searches
> completely failing which obviously breaks everything trying to use it.
It's probably time for a new release...
> The Mavibot update sounds like an exciting change.
Indeed. But it's way more complex that we initially thought it could be,
considering we are doing that on our spare time. At this point, we are
doing progress (and periodic mails are sent to the dev mailing list to
give some status), and it would be extremelly useful to get some
feedback on what we are doing (just because we are likely to make
mistakes, or chose the wrong path...). Checking the existing code would
be as useful.
>
> On the issue of speed, I am using a pool with ldap API 22 release and
> my searches are taking 7-13 msec.
Ar you mesuring one search using System.currentTimeMillis()? Because on
most of the machines, the accuracy of this method is around 10ms (ie,
something that will take sub-ms time will be mesured as taking 10 ms, no
matter what. $
The best way to mesure the time that takes a request would be to use
nanoTime() instead.
> Seems long for single entry result sets,
> granted I have a lot of logging turned on so I can see internals working.
Yes, it's long, and probably you have to consider the network latency,
if you are going through the network. FTR, we have done some benchmarks
last year, with a MacBook Pro as a server (a 2012 MBP), and 5 injectors,
over a 100Mbs network. We were able to sustain a load of 13 000 search
request per second on a 10 000 entries database (random searches,
returning one entry), and the server was peaking at 70% CPU only.
Now, it depends on many parameters, like the network, the disk used in
the server, the memory you have, the number of entries, the log level...
Re: Problem with indexes...
Posted by Mark D <te...@debusschere.com>.
Emmanuel, thanks for the detailed information. Kiran also had some feedback.
The build I am using from trunk seems to be working much better. There was serious
issue with previous build as the method I am using for indexes resulted in searches
completely failing which obviously breaks everything trying to use it.
The Mavibot update sounds like an exciting change.
On the issue of speed, I am using a pool with ldap API 22 release and
my searches are taking 7-13 msec. Seems long for single entry result sets,
granted I have a lot of logging turned on so I can see internals working.
Mark.
Re: Problem with indexes...
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 07/05/2014 18:17, mark debusschere a écrit :
> Was this issue related to M16 or has it been outstanding? I have had countless
> problems with indexes, a few self induced, but recently indexes just stopped working.
> This was after server failed to start due to system schema already present, then
> second attempt server started but app failed to function as all searches were failing.
(first, sorry for the delayed answer, your mail just made it to my inbox
due to heavy issues in The ASF mailers which has stopped working for 5
days...)
There is not one single response to your question. Let me try to give
you some insight.
There are many different use cases, and potantial kind of failures :
- indexes you create before starting the server and injecting data
- indexes you create after having injected data into the server
- we have user indexes (those you create yourself) and system indexes
(those used internally by the server)
The bugs we had was related to the missing initialization of indexes
created after some data were injected. This has been fixed partially in
M16 and in trunk.
Beside this obvious bug, there is a way more critical issue we are
working on for months now : the backend we are using (JDBM) is ok when
it comes to deal with one single BTree (an index is just a BTree, well,
a bit more), but it's not supporting transactions across btrees.
What does it means, and what are the consequences ? Pretty heavy :
- when some user is updating an entry, it will impact tens of indexes
- if at the same time another user is searching the server, it's likely
to use the very same indexes
- *if* this occurs at the very same time, then the updating client might
modify an idex which is used by one of the reader, leading to some
inconsistancies, and an error.
We have added many locks to try to workaround this issue, we modified
the way we fetched the candidates to limit the time it takes to build
this set of candidates - and as a consequence the time during which we
are holding an index -. We also added a journal in the backend to be
able to recover when a crash occurs. All those modifications have
improved the situation, up to a point the server now don't crash when
there are some concurrent reads and writes, but it comes to a price :
all the operations are way longer than they could be.
We are now working on a complete replacement for the JDBM backend :
Mavibot. It will allow us to process an update as a whole, ie all the
indexes will be updated in one shot, or not at all. The searches will be
guaranteed to work as they will *always* proceed on an existing
revision. That will take a bit of time to achieve this new backend, but
we are heavily working on it.
I hope I clarified some aspects of the probelm, and the solution we are
working on...
>
>
Re: Problem with indexes...
Posted by Mark D <te...@debusschere.com>.
Was this issue related to M16 or has it been outstanding?
My app dynamically builds new partitions with index definitions at runtime.
It adds the partitions configuration definitions then creates the JDMB and adds to active service.
Works well, partition is created, can add data, searches work, but when I
restart searches don't work at all.
Trying snapshot build now.
Thanks, Mark.
----- Original Message -----
From: "Kiran Ayyagari" <ka...@apache.org>
To: users@directory.apache.org
Sent: Tuesday, May 6, 2014 8:43:45 AM
Subject: Re: Problem with indexes...
I have filed a bug report[1] and working on the fix
[1] https://issues.apache.org/jira/browse/DIRSERVER-1976
On Tue, May 6, 2014 at 1:36 AM, Gabriel Albano <ga...@pilixo.com>wrote:
> I am having a similar problem. Search before adding the index:
>
> #!SEARCH REQUEST (161) OK
> #!CONNECTION ldap://stage-ds.pilixo.com:10389
> #!DATE 2014-05-05T19:59:01.677
> # LDAP URL : ldap://
>
> stage-ds.pilixo.com:10389/ou=organizations,dc=pilixo?*?sub?(displayName=gabriel*)
> # command line : ldapsearch -H ldap://stage-ds.pilixo.com:10389 -x -D
> "uid=admin,ou=system" -W -b "ou=organizations,dc=pilixo" -s sub -a always
> -z 100000 "(displayName=gabriel*)" "*"
> # baseObject : ou=organizations,dc=pilixo
> # scope : wholeSubtree (2)
> # derefAliases : derefAlways (3)
> # sizeLimit : 100000
> # timeLimit : 0
> # typesOnly : False
> # filter : (displayName=gabriel*)
> # attributes : *
>
> #!SEARCH RESULT DONE (161) OK
> #!CONNECTION ldap://stage-ds.pilixo.com:10389
> #!DATE 2014-05-05T19:59:01.825
> # numEntries : 2
>
>
> Added index:
>
> dn:
> ads-indexAttributeId=displayName,ou=indexes,ads-partitionId=pilixo,ou=pa
> rtitions,ads-directoryServiceId=default,ou=config
> objectclass: top
> objectclass: ads-jdbmIndex
> objectclass: ads-base
> objectclass: ads-index
> ads-indexattributeid: displayName
> ads-indexhasreverse: FALSE
> ads-enabled: TRUE
>
>
> Restarted Server and searched again:
>
> #!SEARCH REQUEST (173) OK
> #!CONNECTION ldap://stage-ds.pilixo.com:10389
> #!DATE 2014-05-05T20:02:16.850
> # LDAP URL : ldap://
>
> stage-ds.pilixo.com:10389/ou=organizations,dc=pilixo?*?sub?(displayName=gabriel*)
> # command line : ldapsearch -H ldap://stage-ds.pilixo.com:10389 -x -D
> "uid=admin,ou=system" -W -b "ou=organizations,dc=pilixo" -s sub -a always
> -z 100000 "(displayName=gabriel*)" "*"
> # baseObject : ou=organizations,dc=pilixo
> # scope : wholeSubtree (2)
> # derefAliases : derefAlways (3)
> # sizeLimit : 100000
> # timeLimit : 0
> # typesOnly : False
> # filter : (displayName=gabriel*)
> # attributes : *
>
> #!SEARCH RESULT DONE (173) OK
> #!CONNECTION ldap://stage-ds.pilixo.com:10389
> #!DATE 2014-05-05T20:02:16.985
> # numEntries : 0
>
> Running 2.0.0-M16, no errors in log as far as I can tell. Furthermore, if
> I modify existing or add new entries, they get picked up by the new index,
> but existing entries are not.
>
> Thanks,
> Gabriel
>
>
>
> On Wed, Apr 23, 2014 at 3:09 AM, Kiran Ayyagari <kayyagari@apache.org
> >wrote:
>
> > On Wed, Apr 23, 2014 at 12:29 PM, Chris Betts <pe...@gmail.com> wrote:
> >
> > > Hi Kirin,
> > >
> > > I get exactly the same behaviour with the 'cn' attribute, so I
> > > suspect the problem is something to do with creating the indexes after
> > the
> > > partition has been created rather than the attribute.
> > >
> > > Curiously, using the same data I successfully created a directory
> > > with working indices on an AWS linux image, but in that case I created
> > the
> > > indices at the same time I created the partition.
> > >
> > > Here's my current 'not working' config from my laptop (OSX / java
> 6 /
> > > apacheds 2) which returns 0 entries when searching for (cn=*):
> > >
> > you need to restart the server after adding the new index/indices in
> > configuration.
> > Can you try the search after restarting? note that server may take a
> while
> > to start due to indexing the data
> > (btw, configuration is looking good)
> >
> > otoh, this is strange, the search shouldn't get impacted before restart,
> > what version of the server are you using?
> >
> > >
> > > #!SEARCH REQUEST (146) OK
> > >
> > > #!CONNECTION ldap://localhost:10389
> > >
> > > #!DATE 2014-04-23T06:43:15.774
> > >
> > > # LDAP URL :
> > > ldap://localhost:10389/o=groupmind,c=au?objectClass?sub?(cn=*)
> > >
> > > # command line : ldapsearch -H ldap://localhost:10389 -x -D
> > > "uid=admin,ou=system" -W -b "o=groupmind,c=au" -s sub -a always -z 1000
> > > "(cn=*)" "objectClass"
> > >
> > > # baseObject : o=groupmind,c=au
> > >
> > > # scope : wholeSubtree (2)
> > >
> > > # derefAliases : derefAlways (3)
> > >
> > > # sizeLimit : 1000
> > >
> > > # timeLimit : 0
> > >
> > > # typesOnly : False
> > >
> > > # filter : (cn=*)
> > >
> > > # attributes : objectClass
> > >
> > >
> > > #!SEARCH RESULT DONE (146) OK
> > >
> > > #!CONNECTION ldap://localhost:10389
> > >
> > > #!DATE 2014-04-23T06:43:15.776
> > >
> > > # numEntries : 0
> > >
> > >
> > > ... hope this helps; however since the AWS version is working fine it's
> > no
> > > longer super-important to solve - I'll try blowing away the partition
> and
> > > recreating it along with the indexes at the same time, and see if that
> > > solves the problem :-).
> > >
> > sure, but see if restarting the server fixes this problem and let me
> know.
> >
> > > cheers,
> > >
> > > - Chris
> > >
> > >
> > > -----
> > > *Dr Christopher Betts*
> > > Australian Cloud Identity
> > > http://cloudidentity.com.au
> > > m: 0408 533 456
> > >
> > >
> > > On 23 April 2014 15:13, Kiran Ayyagari <ka...@apache.org> wrote:
> > >
> > >> can you share the partition configuration and the schema definition of
> > >> the attribute 'groupmindId'?
> > >>
> > >>
> > >> On Wed, Apr 23, 2014 at 5:57 AM, Chris Betts <pe...@gmail.com>
> wrote:
> > >>
> > >>> Hi Folks,
> > >>>
> > >>> I'm trying to migrate a project from openldap to apacheds, and
> I've
> > >>> hit
> > >>> a weird problem with indexing... I suspect I'm doing something wrong,
> > and
> > >>> I'm hoping someone can help :-).
> > >>>
> > >>> Basically, I have a DIT with a subtree of entries containing a
> > custom
> > >>> schema attribute 'groupmindID', which is a case sensitive ID code.
> The
> > >>> application uses this groupmindID for searching entries.
> > >>>
> > >>> After importing the schema and starting things up everything
> worked,
> > >>> but
> > >>> it was pretty slow, so I thought I'd index various attributes, which
> I
> > >>> did
> > >>> using the 'Partitions->Configuration' pane, at which point apacheds
> > >>> stopped
> > >>> finding them at all!
> > >>>
> > >>> I've replicated the problem in apacheds studio; here's some
> output.
> > >>>
> > >>> #!SEARCH REQUEST (212) OK
> > >>> #!CONNECTION ldap://localhost:10389
> > >>> #!DATE 2014-04-23T00:04:42.092
> > >>> # LDAP URL :
> > >>>
> > >>>
> >
> ldap://localhost:10389/cn=topics,o=groupmind,c=au?objectClass?sub?(groupmindID=*)
> > >>> # command line : ldapsearch -H ldap://localhost:10389 -x -D
> > >>> "uid=admin,ou=system" -W -b "cn=topics,o=groupmind,c=au" -s sub -a
> > always
> > >>> -z 1000 "(groupmindID=*)" "objectClass"
> > >>> # baseObject : cn=topics,o=groupmind,c=au
> > >>> # scope : wholeSubtree (2)
> > >>> # derefAliases : derefAlways (3)
> > >>> # sizeLimit : 1000
> > >>> # timeLimit : 0
> > >>> # typesOnly : False
> > >>> # filter : (groupmindID=*)
> > >>> # attributes : objectClass
> > >>>
> > >>> #!SEARCH RESULT DONE (212) OK
> > >>> #!CONNECTION ldap://localhost:10389
> > >>> #!DATE 2014-04-23T00:04:42.102
> > >>> # numEntries : 0
> > >>>
> > >>> ***
> > >>> *** Delete index on 'groupmindID' and restart server, rerun test:
> > >>> ***
> > >>>
> > >>> #!SEARCH REQUEST (253) OK
> > >>> #!CONNECTION ldap://localhost:10389
> > >>> #!DATE 2014-04-23T00:09:42.999
> > >>> # LDAP URL :
> > >>>
> > >>>
> >
> ldap://localhost:10389/cn=topics,o=groupmind,c=au?objectClass?sub?(groupmindID=*)
> > >>> # command line : ldapsearch -H ldap://localhost:10389 -x -D
> > >>> "uid=admin,ou=system" -W -b "cn=topics,o=groupmind,c=au" -s sub -a
> > always
> > >>> -z 1000 "(groupmindID=*)" "objectClass"
> > >>> # baseObject : cn=topics,o=groupmind,c=au
> > >>> # scope : wholeSubtree (2)
> > >>> # derefAliases : derefAlways (3)
> > >>> # sizeLimit : 1000
> > >>> # timeLimit : 0
> > >>> # typesOnly : False
> > >>> # filter : (groupmindID=*)
> > >>> # attributes : objectClass
> > >>>
> > >>> #!SEARCH RESULT DONE (253) OK
> > >>> #!CONNECTION ldap://localhost:10389
> > >>> #!DATE 2014-04-23T00:09:46.856
> > >>> # numEntries : 1000
> > >>>
> > >>>
> > >>>
> > >>> ... I've had a play around, and I can get the same behaviour even
> from
> > >>> 'normal' attributes like 'cn', so I suspect I'm doing something bone
> > >>> headed
> > >>> with the way I've set the indexes up - apologies if it's obvious, I'm
> > >>> very
> > >>> new to apacheds and apacheds studio!
> > >>>
> > >>> cheers,
> > >>>
> > >>> - Chris
> > >>>
> > >>> -----
> > >>> *Dr Christopher Betts*
> > >>>
> > >>> Australian Cloud Identity
> > >>> http://cloudidentity.com.au
> > >>> m: 0408 533 456
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Kiran Ayyagari
> > >> http://keydap.com
> > >>
> > >
> > >
> >
> >
> > --
> > Kiran Ayyagari
> > http://keydap.com
> >
>
--
Kiran Ayyagari
http://keydap.com
Re: Problem with indexes...
Posted by mark debusschere <ma...@debusschere.com>.
Was this issue related to M16 or has it been outstanding? I have had countless
problems with indexes, a few self induced, but recently indexes just stopped working.
This was after server failed to start due to system schema already present, then
second attempt server started but app failed to function as all searches were failing.
----- Original Message -----
From: "Kiran Ayyagari" <ka...@apache.org>
To: users@directory.apache.org
Sent: Tuesday, May 6, 2014 8:43:45 AM
Subject: Re: Problem with indexes...
I have filed a bug report[1] and working on the fix
[1] https://issues.apache.org/jira/browse/DIRSERVER-1976
On Tue, May 6, 2014 at 1:36 AM, Gabriel Albano <ga...@pilixo.com>wrote:
> I am having a similar problem. Search before adding the index:
>
> #!SEARCH REQUEST (161) OK
> #!CONNECTION ldap://stage-ds.pilixo.com:10389
> #!DATE 2014-05-05T19:59:01.677
> # LDAP URL : ldap://
>
> stage-ds.pilixo.com:10389/ou=organizations,dc=pilixo?*?sub?(displayName=gabriel*)
> # command line : ldapsearch -H ldap://stage-ds.pilixo.com:10389 -x -D
> "uid=admin,ou=system" -W -b "ou=organizations,dc=pilixo" -s sub -a always
> -z 100000 "(displayName=gabriel*)" "*"
> # baseObject : ou=organizations,dc=pilixo
> # scope : wholeSubtree (2)
> # derefAliases : derefAlways (3)
> # sizeLimit : 100000
> # timeLimit : 0
> # typesOnly : False
> # filter : (displayName=gabriel*)
> # attributes : *
>
> #!SEARCH RESULT DONE (161) OK
> #!CONNECTION ldap://stage-ds.pilixo.com:10389
> #!DATE 2014-05-05T19:59:01.825
> # numEntries : 2
>
>
> Added index:
>
> dn:
> ads-indexAttributeId=displayName,ou=indexes,ads-partitionId=pilixo,ou=pa
> rtitions,ads-directoryServiceId=default,ou=config
> objectclass: top
> objectclass: ads-jdbmIndex
> objectclass: ads-base
> objectclass: ads-index
> ads-indexattributeid: displayName
> ads-indexhasreverse: FALSE
> ads-enabled: TRUE
>
>
> Restarted Server and searched again:
>
> #!SEARCH REQUEST (173) OK
> #!CONNECTION ldap://stage-ds.pilixo.com:10389
> #!DATE 2014-05-05T20:02:16.850
> # LDAP URL : ldap://
>
> stage-ds.pilixo.com:10389/ou=organizations,dc=pilixo?*?sub?(displayName=gabriel*)
> # command line : ldapsearch -H ldap://stage-ds.pilixo.com:10389 -x -D
> "uid=admin,ou=system" -W -b "ou=organizations,dc=pilixo" -s sub -a always
> -z 100000 "(displayName=gabriel*)" "*"
> # baseObject : ou=organizations,dc=pilixo
> # scope : wholeSubtree (2)
> # derefAliases : derefAlways (3)
> # sizeLimit : 100000
> # timeLimit : 0
> # typesOnly : False
> # filter : (displayName=gabriel*)
> # attributes : *
>
> #!SEARCH RESULT DONE (173) OK
> #!CONNECTION ldap://stage-ds.pilixo.com:10389
> #!DATE 2014-05-05T20:02:16.985
> # numEntries : 0
>
> Running 2.0.0-M16, no errors in log as far as I can tell. Furthermore, if
> I modify existing or add new entries, they get picked up by the new index,
> but existing entries are not.
>
> Thanks,
> Gabriel
>
>
>
> On Wed, Apr 23, 2014 at 3:09 AM, Kiran Ayyagari <kayyagari@apache.org
> >wrote:
>
> > On Wed, Apr 23, 2014 at 12:29 PM, Chris Betts <pe...@gmail.com> wrote:
> >
> > > Hi Kirin,
> > >
> > > I get exactly the same behaviour with the 'cn' attribute, so I
> > > suspect the problem is something to do with creating the indexes after
> > the
> > > partition has been created rather than the attribute.
> > >
> > > Curiously, using the same data I successfully created a directory
> > > with working indices on an AWS linux image, but in that case I created
> > the
> > > indices at the same time I created the partition.
> > >
> > > Here's my current 'not working' config from my laptop (OSX / java
> 6 /
> > > apacheds 2) which returns 0 entries when searching for (cn=*):
> > >
> > you need to restart the server after adding the new index/indices in
> > configuration.
> > Can you try the search after restarting? note that server may take a
> while
> > to start due to indexing the data
> > (btw, configuration is looking good)
> >
> > otoh, this is strange, the search shouldn't get impacted before restart,
> > what version of the server are you using?
> >
> > >
> > > #!SEARCH REQUEST (146) OK
> > >
> > > #!CONNECTION ldap://localhost:10389
> > >
> > > #!DATE 2014-04-23T06:43:15.774
> > >
> > > # LDAP URL :
> > > ldap://localhost:10389/o=groupmind,c=au?objectClass?sub?(cn=*)
> > >
> > > # command line : ldapsearch -H ldap://localhost:10389 -x -D
> > > "uid=admin,ou=system" -W -b "o=groupmind,c=au" -s sub -a always -z 1000
> > > "(cn=*)" "objectClass"
> > >
> > > # baseObject : o=groupmind,c=au
> > >
> > > # scope : wholeSubtree (2)
> > >
> > > # derefAliases : derefAlways (3)
> > >
> > > # sizeLimit : 1000
> > >
> > > # timeLimit : 0
> > >
> > > # typesOnly : False
> > >
> > > # filter : (cn=*)
> > >
> > > # attributes : objectClass
> > >
> > >
> > > #!SEARCH RESULT DONE (146) OK
> > >
> > > #!CONNECTION ldap://localhost:10389
> > >
> > > #!DATE 2014-04-23T06:43:15.776
> > >
> > > # numEntries : 0
> > >
> > >
> > > ... hope this helps; however since the AWS version is working fine it's
> > no
> > > longer super-important to solve - I'll try blowing away the partition
> and
> > > recreating it along with the indexes at the same time, and see if that
> > > solves the problem :-).
> > >
> > sure, but see if restarting the server fixes this problem and let me
> know.
> >
> > > cheers,
> > >
> > > - Chris
> > >
> > >
> > > -----
> > > *Dr Christopher Betts*
> > > Australian Cloud Identity
> > > http://cloudidentity.com.au
> > > m: 0408 533 456
> > >
> > >
> > > On 23 April 2014 15:13, Kiran Ayyagari <ka...@apache.org> wrote:
> > >
> > >> can you share the partition configuration and the schema definition of
> > >> the attribute 'groupmindId'?
> > >>
> > >>
> > >> On Wed, Apr 23, 2014 at 5:57 AM, Chris Betts <pe...@gmail.com>
> wrote:
> > >>
> > >>> Hi Folks,
> > >>>
> > >>> I'm trying to migrate a project from openldap to apacheds, and
> I've
> > >>> hit
> > >>> a weird problem with indexing... I suspect I'm doing something wrong,
> > and
> > >>> I'm hoping someone can help :-).
> > >>>
> > >>> Basically, I have a DIT with a subtree of entries containing a
> > custom
> > >>> schema attribute 'groupmindID', which is a case sensitive ID code.
> The
> > >>> application uses this groupmindID for searching entries.
> > >>>
> > >>> After importing the schema and starting things up everything
> worked,
> > >>> but
> > >>> it was pretty slow, so I thought I'd index various attributes, which
> I
> > >>> did
> > >>> using the 'Partitions->Configuration' pane, at which point apacheds
> > >>> stopped
> > >>> finding them at all!
> > >>>
> > >>> I've replicated the problem in apacheds studio; here's some
> output.
> > >>>
> > >>> #!SEARCH REQUEST (212) OK
> > >>> #!CONNECTION ldap://localhost:10389
> > >>> #!DATE 2014-04-23T00:04:42.092
> > >>> # LDAP URL :
> > >>>
> > >>>
> >
> ldap://localhost:10389/cn=topics,o=groupmind,c=au?objectClass?sub?(groupmindID=*)
> > >>> # command line : ldapsearch -H ldap://localhost:10389 -x -D
> > >>> "uid=admin,ou=system" -W -b "cn=topics,o=groupmind,c=au" -s sub -a
> > always
> > >>> -z 1000 "(groupmindID=*)" "objectClass"
> > >>> # baseObject : cn=topics,o=groupmind,c=au
> > >>> # scope : wholeSubtree (2)
> > >>> # derefAliases : derefAlways (3)
> > >>> # sizeLimit : 1000
> > >>> # timeLimit : 0
> > >>> # typesOnly : False
> > >>> # filter : (groupmindID=*)
> > >>> # attributes : objectClass
> > >>>
> > >>> #!SEARCH RESULT DONE (212) OK
> > >>> #!CONNECTION ldap://localhost:10389
> > >>> #!DATE 2014-04-23T00:04:42.102
> > >>> # numEntries : 0
> > >>>
> > >>> ***
> > >>> *** Delete index on 'groupmindID' and restart server, rerun test:
> > >>> ***
> > >>>
> > >>> #!SEARCH REQUEST (253) OK
> > >>> #!CONNECTION ldap://localhost:10389
> > >>> #!DATE 2014-04-23T00:09:42.999
> > >>> # LDAP URL :
> > >>>
> > >>>
> >
> ldap://localhost:10389/cn=topics,o=groupmind,c=au?objectClass?sub?(groupmindID=*)
> > >>> # command line : ldapsearch -H ldap://localhost:10389 -x -D
> > >>> "uid=admin,ou=system" -W -b "cn=topics,o=groupmind,c=au" -s sub -a
> > always
> > >>> -z 1000 "(groupmindID=*)" "objectClass"
> > >>> # baseObject : cn=topics,o=groupmind,c=au
> > >>> # scope : wholeSubtree (2)
> > >>> # derefAliases : derefAlways (3)
> > >>> # sizeLimit : 1000
> > >>> # timeLimit : 0
> > >>> # typesOnly : False
> > >>> # filter : (groupmindID=*)
> > >>> # attributes : objectClass
> > >>>
> > >>> #!SEARCH RESULT DONE (253) OK
> > >>> #!CONNECTION ldap://localhost:10389
> > >>> #!DATE 2014-04-23T00:09:46.856
> > >>> # numEntries : 1000
> > >>>
> > >>>
> > >>>
> > >>> ... I've had a play around, and I can get the same behaviour even
> from
> > >>> 'normal' attributes like 'cn', so I suspect I'm doing something bone
> > >>> headed
> > >>> with the way I've set the indexes up - apologies if it's obvious, I'm
> > >>> very
> > >>> new to apacheds and apacheds studio!
> > >>>
> > >>> cheers,
> > >>>
> > >>> - Chris
> > >>>
> > >>> -----
> > >>> *Dr Christopher Betts*
> > >>>
> > >>> Australian Cloud Identity
> > >>> http://cloudidentity.com.au
> > >>> m: 0408 533 456
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Kiran Ayyagari
> > >> http://keydap.com
> > >>
> > >
> > >
> >
> >
> > --
> > Kiran Ayyagari
> > http://keydap.com
> >
>
--
Kiran Ayyagari
http://keydap.com
Re: Problem with indexes...
Posted by Kiran Ayyagari <ka...@apache.org>.
I have filed a bug report[1] and working on the fix
[1] https://issues.apache.org/jira/browse/DIRSERVER-1976
On Tue, May 6, 2014 at 1:36 AM, Gabriel Albano <ga...@pilixo.com>wrote:
> I am having a similar problem. Search before adding the index:
>
> #!SEARCH REQUEST (161) OK
> #!CONNECTION ldap://stage-ds.pilixo.com:10389
> #!DATE 2014-05-05T19:59:01.677
> # LDAP URL : ldap://
>
> stage-ds.pilixo.com:10389/ou=organizations,dc=pilixo?*?sub?(displayName=gabriel*)
> # command line : ldapsearch -H ldap://stage-ds.pilixo.com:10389 -x -D
> "uid=admin,ou=system" -W -b "ou=organizations,dc=pilixo" -s sub -a always
> -z 100000 "(displayName=gabriel*)" "*"
> # baseObject : ou=organizations,dc=pilixo
> # scope : wholeSubtree (2)
> # derefAliases : derefAlways (3)
> # sizeLimit : 100000
> # timeLimit : 0
> # typesOnly : False
> # filter : (displayName=gabriel*)
> # attributes : *
>
> #!SEARCH RESULT DONE (161) OK
> #!CONNECTION ldap://stage-ds.pilixo.com:10389
> #!DATE 2014-05-05T19:59:01.825
> # numEntries : 2
>
>
> Added index:
>
> dn:
> ads-indexAttributeId=displayName,ou=indexes,ads-partitionId=pilixo,ou=pa
> rtitions,ads-directoryServiceId=default,ou=config
> objectclass: top
> objectclass: ads-jdbmIndex
> objectclass: ads-base
> objectclass: ads-index
> ads-indexattributeid: displayName
> ads-indexhasreverse: FALSE
> ads-enabled: TRUE
>
>
> Restarted Server and searched again:
>
> #!SEARCH REQUEST (173) OK
> #!CONNECTION ldap://stage-ds.pilixo.com:10389
> #!DATE 2014-05-05T20:02:16.850
> # LDAP URL : ldap://
>
> stage-ds.pilixo.com:10389/ou=organizations,dc=pilixo?*?sub?(displayName=gabriel*)
> # command line : ldapsearch -H ldap://stage-ds.pilixo.com:10389 -x -D
> "uid=admin,ou=system" -W -b "ou=organizations,dc=pilixo" -s sub -a always
> -z 100000 "(displayName=gabriel*)" "*"
> # baseObject : ou=organizations,dc=pilixo
> # scope : wholeSubtree (2)
> # derefAliases : derefAlways (3)
> # sizeLimit : 100000
> # timeLimit : 0
> # typesOnly : False
> # filter : (displayName=gabriel*)
> # attributes : *
>
> #!SEARCH RESULT DONE (173) OK
> #!CONNECTION ldap://stage-ds.pilixo.com:10389
> #!DATE 2014-05-05T20:02:16.985
> # numEntries : 0
>
> Running 2.0.0-M16, no errors in log as far as I can tell. Furthermore, if
> I modify existing or add new entries, they get picked up by the new index,
> but existing entries are not.
>
> Thanks,
> Gabriel
>
>
>
> On Wed, Apr 23, 2014 at 3:09 AM, Kiran Ayyagari <kayyagari@apache.org
> >wrote:
>
> > On Wed, Apr 23, 2014 at 12:29 PM, Chris Betts <pe...@gmail.com> wrote:
> >
> > > Hi Kirin,
> > >
> > > I get exactly the same behaviour with the 'cn' attribute, so I
> > > suspect the problem is something to do with creating the indexes after
> > the
> > > partition has been created rather than the attribute.
> > >
> > > Curiously, using the same data I successfully created a directory
> > > with working indices on an AWS linux image, but in that case I created
> > the
> > > indices at the same time I created the partition.
> > >
> > > Here's my current 'not working' config from my laptop (OSX / java
> 6 /
> > > apacheds 2) which returns 0 entries when searching for (cn=*):
> > >
> > you need to restart the server after adding the new index/indices in
> > configuration.
> > Can you try the search after restarting? note that server may take a
> while
> > to start due to indexing the data
> > (btw, configuration is looking good)
> >
> > otoh, this is strange, the search shouldn't get impacted before restart,
> > what version of the server are you using?
> >
> > >
> > > #!SEARCH REQUEST (146) OK
> > >
> > > #!CONNECTION ldap://localhost:10389
> > >
> > > #!DATE 2014-04-23T06:43:15.774
> > >
> > > # LDAP URL :
> > > ldap://localhost:10389/o=groupmind,c=au?objectClass?sub?(cn=*)
> > >
> > > # command line : ldapsearch -H ldap://localhost:10389 -x -D
> > > "uid=admin,ou=system" -W -b "o=groupmind,c=au" -s sub -a always -z 1000
> > > "(cn=*)" "objectClass"
> > >
> > > # baseObject : o=groupmind,c=au
> > >
> > > # scope : wholeSubtree (2)
> > >
> > > # derefAliases : derefAlways (3)
> > >
> > > # sizeLimit : 1000
> > >
> > > # timeLimit : 0
> > >
> > > # typesOnly : False
> > >
> > > # filter : (cn=*)
> > >
> > > # attributes : objectClass
> > >
> > >
> > > #!SEARCH RESULT DONE (146) OK
> > >
> > > #!CONNECTION ldap://localhost:10389
> > >
> > > #!DATE 2014-04-23T06:43:15.776
> > >
> > > # numEntries : 0
> > >
> > >
> > > ... hope this helps; however since the AWS version is working fine it's
> > no
> > > longer super-important to solve - I'll try blowing away the partition
> and
> > > recreating it along with the indexes at the same time, and see if that
> > > solves the problem :-).
> > >
> > sure, but see if restarting the server fixes this problem and let me
> know.
> >
> > > cheers,
> > >
> > > - Chris
> > >
> > >
> > > -----
> > > *Dr Christopher Betts*
> > > Australian Cloud Identity
> > > http://cloudidentity.com.au
> > > m: 0408 533 456
> > >
> > >
> > > On 23 April 2014 15:13, Kiran Ayyagari <ka...@apache.org> wrote:
> > >
> > >> can you share the partition configuration and the schema definition of
> > >> the attribute 'groupmindId'?
> > >>
> > >>
> > >> On Wed, Apr 23, 2014 at 5:57 AM, Chris Betts <pe...@gmail.com>
> wrote:
> > >>
> > >>> Hi Folks,
> > >>>
> > >>> I'm trying to migrate a project from openldap to apacheds, and
> I've
> > >>> hit
> > >>> a weird problem with indexing... I suspect I'm doing something wrong,
> > and
> > >>> I'm hoping someone can help :-).
> > >>>
> > >>> Basically, I have a DIT with a subtree of entries containing a
> > custom
> > >>> schema attribute 'groupmindID', which is a case sensitive ID code.
> The
> > >>> application uses this groupmindID for searching entries.
> > >>>
> > >>> After importing the schema and starting things up everything
> worked,
> > >>> but
> > >>> it was pretty slow, so I thought I'd index various attributes, which
> I
> > >>> did
> > >>> using the 'Partitions->Configuration' pane, at which point apacheds
> > >>> stopped
> > >>> finding them at all!
> > >>>
> > >>> I've replicated the problem in apacheds studio; here's some
> output.
> > >>>
> > >>> #!SEARCH REQUEST (212) OK
> > >>> #!CONNECTION ldap://localhost:10389
> > >>> #!DATE 2014-04-23T00:04:42.092
> > >>> # LDAP URL :
> > >>>
> > >>>
> >
> ldap://localhost:10389/cn=topics,o=groupmind,c=au?objectClass?sub?(groupmindID=*)
> > >>> # command line : ldapsearch -H ldap://localhost:10389 -x -D
> > >>> "uid=admin,ou=system" -W -b "cn=topics,o=groupmind,c=au" -s sub -a
> > always
> > >>> -z 1000 "(groupmindID=*)" "objectClass"
> > >>> # baseObject : cn=topics,o=groupmind,c=au
> > >>> # scope : wholeSubtree (2)
> > >>> # derefAliases : derefAlways (3)
> > >>> # sizeLimit : 1000
> > >>> # timeLimit : 0
> > >>> # typesOnly : False
> > >>> # filter : (groupmindID=*)
> > >>> # attributes : objectClass
> > >>>
> > >>> #!SEARCH RESULT DONE (212) OK
> > >>> #!CONNECTION ldap://localhost:10389
> > >>> #!DATE 2014-04-23T00:04:42.102
> > >>> # numEntries : 0
> > >>>
> > >>> ***
> > >>> *** Delete index on 'groupmindID' and restart server, rerun test:
> > >>> ***
> > >>>
> > >>> #!SEARCH REQUEST (253) OK
> > >>> #!CONNECTION ldap://localhost:10389
> > >>> #!DATE 2014-04-23T00:09:42.999
> > >>> # LDAP URL :
> > >>>
> > >>>
> >
> ldap://localhost:10389/cn=topics,o=groupmind,c=au?objectClass?sub?(groupmindID=*)
> > >>> # command line : ldapsearch -H ldap://localhost:10389 -x -D
> > >>> "uid=admin,ou=system" -W -b "cn=topics,o=groupmind,c=au" -s sub -a
> > always
> > >>> -z 1000 "(groupmindID=*)" "objectClass"
> > >>> # baseObject : cn=topics,o=groupmind,c=au
> > >>> # scope : wholeSubtree (2)
> > >>> # derefAliases : derefAlways (3)
> > >>> # sizeLimit : 1000
> > >>> # timeLimit : 0
> > >>> # typesOnly : False
> > >>> # filter : (groupmindID=*)
> > >>> # attributes : objectClass
> > >>>
> > >>> #!SEARCH RESULT DONE (253) OK
> > >>> #!CONNECTION ldap://localhost:10389
> > >>> #!DATE 2014-04-23T00:09:46.856
> > >>> # numEntries : 1000
> > >>>
> > >>>
> > >>>
> > >>> ... I've had a play around, and I can get the same behaviour even
> from
> > >>> 'normal' attributes like 'cn', so I suspect I'm doing something bone
> > >>> headed
> > >>> with the way I've set the indexes up - apologies if it's obvious, I'm
> > >>> very
> > >>> new to apacheds and apacheds studio!
> > >>>
> > >>> cheers,
> > >>>
> > >>> - Chris
> > >>>
> > >>> -----
> > >>> *Dr Christopher Betts*
> > >>>
> > >>> Australian Cloud Identity
> > >>> http://cloudidentity.com.au
> > >>> m: 0408 533 456
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Kiran Ayyagari
> > >> http://keydap.com
> > >>
> > >
> > >
> >
> >
> > --
> > Kiran Ayyagari
> > http://keydap.com
> >
>
--
Kiran Ayyagari
http://keydap.com