[ApacheDS] Custom schema attributes used in search filter

[B G <fi...@gmail.com>]

Hi...

I came across a problem today using an embedded apacheds 1.5. When my
application starts I programatically add my custom schema to the directory
schema partition. Prior to 1.5 I did this using bootstrap schema, but was
unable to get ldif imports to work consistently on a windows machine with
1.5, could no longer use the 1.0 bootstrap mechanism and so I resorted to
doing it programatically. This seems to work fine as my schema does display
in an ldap browser and when I add entires schema validation against my
custom schema works as I would expect.

The problem comes when I restart my application and do a search using an
attribute from my custom schema in a search filter. This results in a null
pointer being thrown in the BTreePartition.lookup method due to not being
able to find an entry which is assumed to always be non-null at this point
in the code. I did some debugging and was able to determine that I could
avoid the problem if I did not allow the shutdown code for the
DefaultDirectoryService to be run. I then tried to determine what the
difference was between running the shutdown code and not running the
shutdown code and found the following:

1) If the shutdown code is run then when I restart the server the custom
schema that I added programatically on the first start was persisted and is
already in the schema partition when I restart. If I do not allow the
shutdown code to run the schema has not been persisted and it gets added
upon each restart (I do check to see if a schema element is present before I
add it).
2) When the custom schema has been persisted (via the shutdown code somehow)
an entry which uses attributes from my custom schema in its DN gets
normalized to using OID for the attribute name and the failure described
above occurs, but when it has not been persisted and the schema was added
during this start the entry's DN gets normalized to using the attribute name
and not the OID. In this case things work correctly and no null pointer.

I realize this may be a bit confusing, but there is definitely as issue here
and I will not pretend to understand all that is going on. I have worked
around the problem for the moment by not allowing the directory shutdown
code to be run, but I think this is a hack work around that will probably
cause some other problems. I suppose another work around could be to remove
the scheme entry if it is present and then readd it upon each restart, but
have not tried this to see if that works and seems like another hack. Is
there something obvious I am doing wrong? Is custom schema supposed to be
persisted? Any help to get this issue resolved correctly would be greatly
appreciated and if I can provide more or better information please let me
know.

Thanks a lot

Re: [ApacheDS] Custom schema attributes used in search filter

[Emmanuel Lecharny <el...@apache.org>]

B G a écrit :

> Hi...


Hi !

>
> I came across a problem today using an embedded apacheds 1.5. When my
> application starts I programatically add my custom schema to the 
> directory
> schema partition. Prior to 1.5 I did this using bootstrap schema, but was
> unable to get ldif imports to work consistently on a windows machine with
> 1.5, could no longer use the 1.0 bootstrap mechanism and so I resorted to
> doing it programatically. This seems to work fine as my schema does 
> display
> in an ldap browser and when I add entires schema validation against my
> custom schema works as I would expect.
>
> The problem comes when I restart my application and do a search using an
> attribute from my custom schema in a search filter. This results in a 
> null
> pointer being thrown in the BTreePartition.lookup method due to not being
> able to find an entry which is assumed to always be non-null at this 
> point
> in the code. I did some debugging and was able to determine that I could
> avoid the problem if I did not allow the shutdown code for the
> DefaultDirectoryService to be run. I then tried to determine what the
> difference was between running the shutdown code and not running the
> shutdown code and found the following:
>
> 1) If the shutdown code is run then when I restart the server the custom
> schema that I added programatically on the first start was persisted 
> and is
> already in the schema partition when I restart. If I do not allow the
> shutdown code to run the schema has not been persisted and it gets added
> upon each restart (I do check to see if a schema element is present 
> before I
> add it).
> 2) When the custom schema has been persisted (via the shutdown code 
> somehow)
> an entry which uses attributes from my custom schema in its DN gets
> normalized to using OID for the attribute name and the failure described
> above occurs, but when it has not been persisted and the schema was added
> during this start the entry's DN gets normalized to using the 
> attribute name
> and not the OID. In this case things work correctly and no null pointer.
>
> I realize this may be a bit confusing, but there is definitely as 
> issue here
> and I will not pretend to understand all that is going on. I have worked
> around the problem for the moment by not allowing the directory shutdown
> code to be run, but I think this is a hack work around that will probably
> cause some other problems. I suppose another work around could be to 
> remove
> the scheme entry if it is present and then readd it upon each restart, 
> but
> have not tried this to see if that works and seems like another hack. Is
> there something obvious I am doing wrong? Is custom schema supposed to be
> persisted? Any help to get this issue resolved correctly would be greatly
> appreciated and if I can provide more or better information please let me
> know.
>
> Thanks a lot

Whoahh... A pretty impressive description :) Ok, I have to read it 
again. What could help is if you could provide some code which expose 
the buggy behavior, so that we can debug it. Is it possible ?

Otherwise, we would really appreciate if you can create a JIRA for that, 
because it might take us some time to fix it, and I don't really want 
this potential bug to be lost in ML ...

Thanks !
Emmanuel

Re: [ApacheDS] Custom schema attributes used in search filter

[B G <fi...@gmail.com>]

Thanks for the responses...

When I sat down to write a simple app that demonstrates the bug, of course I
was unable to reproduce the problem...well not the original problem I
mentioned anyways. What I did find was a new bug in which I am able to
create duplicate entries without getting a NameAlreadyBoundException nor any
NamingException...they both live happily in the directory. This puzzled me
why I did not see the original bug I found in my app and was finally able to
get that one reproduced as well in this simple app.

For the dup entry bug the app:
- starts the directory
- adds schema context with one attribute and one objectclass definied
- creates a subcontext using the objectclass defined above and a rdn using
the attribute defined above
- does a search with the expected results
- restarts the directory allowing the shutdown code to run (if you were to
retsart it without the shutdown code the bug does not happen)
- adds the same entry as above (here is a problem...should get an exception
but it gets added twice)
- does the search and 2 entries are returned.

For the original null pointer on search the app: (needs to be run with an
empty work dir)
- starts the directory
- adds schema context with one attribute and one objectclass definied
- creates a subcontext using the objectclass defined above and a rdn using
the attribute defined above
- restarts the directory allowing the shutdown code to run (if you were to
retsart it without the shutdown code the bug does not happen)
- does a search with the search filter using the attribute defined in the
schema above which is also the rdn.

I can tell you that in both cases not allowing the shutdown code to run
before the restart causes the bugs not to happen.

The jira is created with the test app source code attached (DIRSERVER-938).

Let me know if this app demonstrates I am doing something illegal or if I
can provide more information.

Thanks

Thanks a lot for looking into this...

On 5/18/07, Alex Karasulu <ak...@apache.org> wrote:
>
> Hi,
>
> On 5/17/07, B G <fi...@gmail.com> wrote:
> >
> > Hi...
> >
> > I came across a problem today using an embedded apacheds 1.5. When my
> > application starts I programatically add my custom schema to the
> directory
> > schema partition. Prior to 1.5 I did this using bootstrap schema, but
> was
> > unable to get ldif imports to work consistently on a windows machine
> with
> > 1.5, could no longer use the 1.0 bootstrap mechanism and so I resorted
> to
> > doing it programatically. This seems to work fine as my schema does
> > display
> > in an ldap browser and when I add entires schema validation against my
> > custom schema works as I would expect.
> >
> > The problem comes when I restart my application and do a search using an
> > attribute from my custom schema in a search filter. This results in a
> null
> > pointer being thrown in the BTreePartition.lookup method due to not
> being
> > able to find an entry which is assumed to always be non-null at this
> point
> > in the code. I did some debugging and was able to determine that I could
> > avoid the problem if I did not allow the shutdown code for the
> > DefaultDirectoryService to be run. I then tried to determine what the
> > difference was between running the shutdown code and not running the
> > shutdown code and found the following:
> >
> > 1) If the shutdown code is run then when I restart the server the custom
> > schema that I added programatically on the first start was persisted and
> > is
> > already in the schema partition when I restart. If I do not allow the
> > shutdown code to run the schema has not been persisted and it gets added
> > upon each restart (I do check to see if a schema element is present
> before
> > I
> > add it).
>
>
> Yes the shutdown code installs a JVM shutdown hook that syncs up the
> partitions
> in the server.  To avoid having to use this code then you must sync up the
> server
> manually before a shutdown or toggle the configuration parameter that
> controls
> the syncOnWrite functionality of a partition.  There is a synchOnWrite
> boolean
> property on the PartitionConfiguration bean to allow you to do this.  This
> way writes
> are sync'd to disk as they occur instead of being cached.
>
> 2) When the custom schema has been persisted (via the shutdown code
> somehow)
> > an entry which uses attributes from my custom schema in its DN gets
> > normalized to using OID for the attribute name and the failure described
> > above occurs, but when it has not been persisted and the schema was
> added
> > during this start the entry's DN gets normalized to using the attribute
> > name
> > and not the OID. In this case things work correctly and no null pointer.
>
>
> Hmmm this is odd.  Please check to see if the schema is enabled.  To do so
> just
> check to see the schema you created does not have an m-disabled attribute
> in
> it's entry within the ou=schema partition.  If it does simply delete this
> attribute
> or set it to FALSE.  See if this corrects the problem.
>
> I realize this may be a bit confusing, but there is definitely as issue
> here
> > and I will not pretend to understand all that is going on.
>
>
> NP there probably is something going on.  The goal now is to isolate the
> issue
> then fix it.  Could you look into writing a simple reduced test case that
> reproduces
> this bug.  No need to use your complete schema or all the data you have.
> One
> schema attribute and a single entry is sufficient in the test
> case.  Perhaps
> you
> can file a JIRA about this and attach the test case to the JIRA.
>
> I have worked
> > around the problem for the moment by not allowing the directory shutdown
> > code to be run, but I think this is a hack work around that will
> probably
> > cause some other problems.
>
>
> Yes this could result in loosing your data if the cache does not sync and
> you
> Control-C the server process.  The shutdown code is there to make sure the
> disks are sync'd even on an abrupt shutdown.  Again though you can prevent
> this by toggling the synchOnWrite prop on the partition configuration
> beans.
>
> I suppose another work around could be to remove
> > the scheme entry if it is present and then readd it upon each restart,
> but
> > have not tried this to see if that works and seems like another hack. Is
> > there something obvious I am doing wrong?
>
>
> I cannot figure anything out at this point.  The best thing we can do at
> this
> point is to  get more info from you as well as a test case that isolates
> this
> bug.
>
> Is custom schema supposed to be
> > persisted?
>
>
> Yes it should persist in the ou=schema partition.
>
> Alex
>

Re: [ApacheDS] Custom schema attributes used in search filter

[Alex Karasulu <ak...@apache.org>]

Hi,

On 5/17/07, B G <fi...@gmail.com> wrote:
>
> Hi...
>
> I came across a problem today using an embedded apacheds 1.5. When my
> application starts I programatically add my custom schema to the directory
> schema partition. Prior to 1.5 I did this using bootstrap schema, but was
> unable to get ldif imports to work consistently on a windows machine with
> 1.5, could no longer use the 1.0 bootstrap mechanism and so I resorted to
> doing it programatically. This seems to work fine as my schema does
> display
> in an ldap browser and when I add entires schema validation against my
> custom schema works as I would expect.
>
> The problem comes when I restart my application and do a search using an
> attribute from my custom schema in a search filter. This results in a null
> pointer being thrown in the BTreePartition.lookup method due to not being
> able to find an entry which is assumed to always be non-null at this point
> in the code. I did some debugging and was able to determine that I could
> avoid the problem if I did not allow the shutdown code for the
> DefaultDirectoryService to be run. I then tried to determine what the
> difference was between running the shutdown code and not running the
> shutdown code and found the following:
>
> 1) If the shutdown code is run then when I restart the server the custom
> schema that I added programatically on the first start was persisted and
> is
> already in the schema partition when I restart. If I do not allow the
> shutdown code to run the schema has not been persisted and it gets added
> upon each restart (I do check to see if a schema element is present before
> I
> add it).


Yes the shutdown code installs a JVM shutdown hook that syncs up the
partitions
in the server.  To avoid having to use this code then you must sync up the
server
manually before a shutdown or toggle the configuration parameter that
controls
the syncOnWrite functionality of a partition.  There is a synchOnWrite
boolean
property on the PartitionConfiguration bean to allow you to do this.  This
way writes
are sync'd to disk as they occur instead of being cached.

2) When the custom schema has been persisted (via the shutdown code somehow)
> an entry which uses attributes from my custom schema in its DN gets
> normalized to using OID for the attribute name and the failure described
> above occurs, but when it has not been persisted and the schema was added
> during this start the entry's DN gets normalized to using the attribute
> name
> and not the OID. In this case things work correctly and no null pointer.


Hmmm this is odd.  Please check to see if the schema is enabled.  To do so
just
check to see the schema you created does not have an m-disabled attribute in
it's entry within the ou=schema partition.  If it does simply delete this
attribute
or set it to FALSE.  See if this corrects the problem.

I realize this may be a bit confusing, but there is definitely as issue here
> and I will not pretend to understand all that is going on.


NP there probably is something going on.  The goal now is to isolate the
issue
then fix it.  Could you look into writing a simple reduced test case that
reproduces
this bug.  No need to use your complete schema or all the data you have.
One
schema attribute and a single entry is sufficient in the test case.  Perhaps
you
can file a JIRA about this and attach the test case to the JIRA.

I have worked
> around the problem for the moment by not allowing the directory shutdown
> code to be run, but I think this is a hack work around that will probably
> cause some other problems.


Yes this could result in loosing your data if the cache does not sync and
you
Control-C the server process.  The shutdown code is there to make sure the
disks are sync'd even on an abrupt shutdown.  Again though you can prevent
this by toggling the synchOnWrite prop on the partition configuration beans.

I suppose another work around could be to remove
> the scheme entry if it is present and then readd it upon each restart, but
> have not tried this to see if that works and seems like another hack. Is
> there something obvious I am doing wrong?


I cannot figure anything out at this point.  The best thing we can do at
this
point is to  get more info from you as well as a test case that isolates
this
bug.

Is custom schema supposed to be
> persisted?


Yes it should persist in the ou=schema partition.

Alex