dynamic schema support

[Norbet Reilly <nr...@gmail.com>]

Hi DS-ers,
 I have been using ApacheDS for a couple of weeks and I'm enjoying the ride 
so far. It seems the initial focus has been flexibility at the front-end, 
and pluggability at the back-end will be eased when OSGi support is added in 
a future release.
 My immediate concern is supporting a custom back-end LDAP partition which 
has its own schema, prior to the availability of RFC 3672 support (which 
sounds like it will smooth the way in the future). I have plugged in my 
custom LDAP AbstractContextPartition successfully (using the IETF LDAP 
classes to convert JNDI back to LDAP) and have written some support classes 
which fetch the remote org.ietf.ldap.LDAPSchema (during doInit() using some 
credentials known from the partition's configuration properties) and 
internalise it into the
org.apache.ldap.server.schema.GlobalRegistriesinstance (noting in the
future AbstractBootstrapProducer should ideally
inherit from an abstract intermediate class to ease writing of classes like 
mine which create non-bootstrap instances of classes extending 
AbstractAttributeType/AbstractSchemaObject etc) . ...what a sentence...
 My question is: what is the best way to get the ApacheDS server's 
org.apache.ldap.server.schema.SchemaService to notice that additions have 
been made to the GlobalRegistries. I notice the addition of 
AddContextPartitionConfiguration in ApacheDS 0.9.2, but I'm not sure hwo to 
configure its use via an ApacheDS configuration file (or even if it is 
indeed the answer to my problems). Failing that the only other approach I 
can think of is to get the startup infrastructure to create a back-end 
InitialContext very early so that my custom partition is queried for its 
schema just after the bootstrap schemas are processed, which sounds a very 
experimental path to follow.
 Note that attempting to refresh the schema via JXplorer never seems to have 
any effect (I imagine that somewhere in the server it is assumed that there 
are only bootstrap schemas).
 Any guidance would be very much appreciated!
 Thanks

custom partition support (was "Re: dynamic schema support")

[Norbet Reilly <nr...@gmail.com>]

> On 9/9/05, Alex Karasulu <ao...@bellsouth.net> wrote:

>Glad to see your interest. Hope to see you contribute one day.

>Cheers,
>Alex

Thanks for your help, and I hope to contribute once I'm less of a DS noob 
and have more to offer!

This is just to report on my progress and brain dump...

I managed to get my custom partition contributing its schema to the global 
ApacheDS one. My problem was originally that I was peforming the LDAP bind 
to the remote directory proxied by my custom partition lazily (as the proxy 
had to insert itself into the middle of the start up sequence of an external 
server), and only retrieving the schema when the lazy bind was eventually 
performed. I now perform the LDAP bind in the custom partition's doInit() 
method and query the schema and add entries from it to the GlobalRegistries 
(I no longer use the IETF LDAP classes and now use vanilla JNDI, which is a 
bit of a pain for schema stuff but at least removes a dependency), resulting 
in an uber-schema which JXplorer happily displays.

I read RFC 3672 and have followed the recent additions to DS to support it. 
My gut feeling is that the subtree specification syntax is the icing on the 
cake, and that when custom partitions are implemented at the back-end the 
default behaviour should be to delegate all operations (schema/bind etc) to 
them based on DN suffix as happens for the other JNDI operations like 
search/lookup etc. So perhaps the lion's share of the work is going to be 
achieving this delegation more then driving the delegation using the subtree 
syntax.

Which brings me to the one remaining short-term hurdle I am facing: 
delegation of binds to my custom partition based on DN suffix. At the moment 
the BindHandler passes information from JXplorer through to the 
SimpleAuthenticator, which always does its checks against the static 
environment specified in my apacheds.xml server configuration. Hence the 
only credentials which will be accepted are the global DS admin credentials: 
"uid=admin,ou=system".

What I would like to achieve is delegating binds to my custom partition 
based on the DN suffix of the loginDN presented. I can see two ways of doing 
this:
1. Add an "environment" properties object to a class extending 
ContextPartitionConfiguration and change BindHandler to query all partition 
configs for instances of it. On finding one BindHandler would check for a 
suffix match, and if one is found will use its "environment" instead of the 
global when attempting its ctx = new InitialLdapContext().
2. Similar approach to 1., but instead of simply substituting an alternative 
environment add a "bind()" call on the partition and have the bind handler 
call in on a suffix match. I see no obvious way for the BindHandler to get 
hold of the relevant ContextPartitionNexus instance (and don't really 
understand the architecture well enough to know if this would be a "bad 
thing") so I suspect approach 1. is the better option.

Does anyone who is not so much a DS noob as myself have any opinions on the 
notion of bind delegation and/or the two approaches above?

I suspect that ultimately schema retrieval/normalisation semantics may 
follow a similar path for custom partitions: by default they will perform 
their own schema checking/normalisation for operations which deal with their 
subtree, and their schema contents won't appear in the global schema unless 
there is some explicit configuration stating that they should.

Any comments/feedback much appreciated...

Thanks


>

Re: dynamic schema support

[Alex Karasulu <ao...@bellsouth.net>]

Norbet Reilly wrote:

> Thanks very much for taking the time to address my query! Some 
> comments inline...
>
NP

> On 9/7/05, *Alex Karasulu* <aok123@bellsouth.net 
> <ma...@bellsouth.net>> wrote:
>
>     Norbet Reilly wrote:
>
>
<snip/>

>     > I notice the addition of AddContextPartitionConfiguration in
>     ApacheDS
>     > 0.9.2, but I'm not sure hwo to configure its use via an ApacheDS
>     > configuration file (or even if it is indeed the answer to my
>     problems).
>
>     Nah this is something else totally.  It's for dynamically adding and
>     removing partitions after system startup.
>
>
> Ultimately there will be some linkage as the dynamically added 
> partition will need to inform the DS core of some of its features, but 
> of course that isn't relevant until the pathway for this information 
> is known ;-)
>
Yes good point I did not think of that.

>     > Failing that the only other approach I can think of is to get the
>     > startup infrastructure to create a back-end InitialContext very
>     early
>     > so that my custom partition is queried for its schema just after the
>     > bootstrap schemas are processed, which sounds a very
>     experimental path
>     > to follow.
>
>     Yeah you're not in a very good situation actually.  There really
>     is no
>     clean way to solve your problem.  We just need to implement the Schema
>     subsystem properly.  You interested in giving that a try?
>
>
> I'm working as a permanent employee currently, so I might need to come 
> up with a short term solution initially (assuming that is possible). 
> Once I've read the RFC more thoroughly and formed of picture in my 
> mind of the work required, then I may consider asking my employer for 
> permission to work on the schema subsystem. While there is mutual 
> benefit, and assuming the amount of work required is not enormous, 
> I've got a fairly good chance of a favourable response.
>
> I should also mention that my interested in DS is to act as an 
> intermediate proxy between an existing C++ client and a directory, 
> where there is a strong focus on avoiding (or at least minimising) 
> changes to client. Thinking about it, supporting the RFC 3672 stuff 
> would most probably mean zero change to the client.
>
> Do you have any rough idea of a) the design and b) the amount of time 
> you guess it would take you to come up with a solution (I'd double it 
> at least, given my very limited exposure to the DS codebase/architecture).
>
Well I have not fully thought of it yet.  I am busy right now 
implementing the subentry RFC (3672).  Next I will move on to collective 
attributes (3671) to test the functionality in the subentry code.  Then 
I will work on adding X.501 defined ACIs with some other folks here.  
After this I think it's time to attack the schema issues.  However 
someone can work the schema code in parallel with us after this week 
when the subentry code is complete and tested.

>     > Note that attempting to refresh the schema via JXplorer never
>     seems to
>     > have any effect (I imagine that somewhere in the server it is
>     assumed
>     > that there are only bootstrap schemas).
>
>     You guessed right ... schema is static on startup.
>
>     >
>     > Any guidance would be very much appreciated!
>
>     Sorry can't really give you a better way here other than to build out
>     the schema subsystem so it can support your needs.  This means just
>     implementing it properly.  There is no other trick we can use at
>     the moment.
>
>     Alex
>
>
> Thanks again for taking time to answer my query.

Glad to see your interest.  Hope to see you contribute one day.

Cheers,
Alex

Re: dynamic schema support

[Norbet Reilly <nr...@gmail.com>]

Thanks very much for taking the time to address my query! Some comments 
inline...

On 9/7/05, Alex Karasulu <ao...@bellsouth.net> wrote:
> 
> Norbet Reilly wrote:
> 
> > Hi DS-ers,
> 
> Hello!
> 
> > My immediate concern is supporting a custom back-end LDAP partition
> > which has its own schema, prior to the availability of RFC 3672
> > support (which sounds like it will smooth the way in the future).
> 
> It sure will. You'll be able to define an autonomous administrative
> point for the context of your partition to manage all the aspects of the
> DIT therein.


I'm going to do a bit more reading so I understand this RFC fully.

> I have plugged in my custom LDAP AbstractContextPartition successfully
> > (using the IETF LDAP classes to convert JNDI back to LDAP) and have
> > written some support classes which fetch the remote
> > org.ietf.ldap.LDAPSchema (during doInit() using some credentials known
> > from the partition's configuration properties) and internalise it into
> > the org.apache.ldap.server.schema.GlobalRegistries instance (noting in
> > the future AbstractBootstrapProducer should ideally inherit from an
> > abstract intermediate class to ease writing of classes like mine which
> > create non-bootstrap instances of classes extending
> > AbstractAttributeType/AbstractSchemaObject etc) . ...what a sentence...
> 
> Made sense to me.
> 
> > My question is: what is the best way to get the ApacheDS server's
> > org.apache.ldap.server.schema.SchemaService to notice that additions
> > have been made to the GlobalRegistries.
> 
> Nothing like this so far. Really the schema subsystem is extremely
> primitive. In fact we never intended to use the boostrap mechanism for
> anything other than supporting some critical schema elements for the
> system partition. Now we're using the bootstrap mechanism for the whole
> shebang.
> 
> Part of this is my fault. I waited until 3672 came since this would
> give me the opportunity to really think about doing schema the right way.


Cool, I was just wanting to check that I hadn't missed something in the 
existing codebase.

> I notice the addition of AddContextPartitionConfiguration in ApacheDS
> > 0.9.2, but I'm not sure hwo to configure its use via an ApacheDS
> > configuration file (or even if it is indeed the answer to my problems).
> 
> Nah this is something else totally. It's for dynamically adding and
> removing partitions after system startup.


Ultimately there will be some linkage as the dynamically added partition 
will need to inform the DS core of some of its features, but of course that 
isn't relevant until the pathway for this information is known ;-)

> Failing that the only other approach I can think of is to get the
> > startup infrastructure to create a back-end InitialContext very early
> > so that my custom partition is queried for its schema just after the
> > bootstrap schemas are processed, which sounds a very experimental path
> > to follow.
> 
> Yeah you're not in a very good situation actually. There really is no
> clean way to solve your problem. We just need to implement the Schema
> subsystem properly. You interested in giving that a try?


I'm working as a permanent employee currently, so I might need to come up 
with a short term solution initially (assuming that is possible). Once I've 
read the RFC more thoroughly and formed of picture in my mind of the work 
required, then I may consider asking my employer for permission to work on 
the schema subsystem. While there is mutual benefit, and assuming the amount 
of work required is not enormous, I've got a fairly good chance of a 
favourable response.

I should also mention that my interested in DS is to act as an intermediate 
proxy between an existing C++ client and a directory, where there is a 
strong focus on avoiding (or at least minimising) changes to client. 
Thinking about it, supporting the RFC 3672 stuff would most probably mean 
zero change to the client.

Do you have any rough idea of a) the design and b) the amount of time you 
guess it would take you to come up with a solution (I'd double it at least, 
given my very limited exposure to the DS codebase/architecture).

> Note that attempting to refresh the schema via JXplorer never seems to
> > have any effect (I imagine that somewhere in the server it is assumed
> > that there are only bootstrap schemas).
> 
> You guessed right ... schema is static on startup.
> 
> >
> > Any guidance would be very much appreciated!
> 
> Sorry can't really give you a better way here other than to build out
> the schema subsystem so it can support your needs. This means just
> implementing it properly. There is no other trick we can use at the 
> moment.
> 
> Alex


Thanks again for taking time to answer my query.

I'm off to read the RFC and scout the DS codebase and see if I can come up 
with a guesstimate for the schema subsystem support!

Re: dynamic schema support

[Alex Karasulu <ao...@bellsouth.net>]

Norbet Reilly wrote:

> Hi DS-ers,

Hello!

> My immediate concern is supporting a custom back-end LDAP partition 
> which has its own schema, prior to the availability of RFC 3672 
> support (which sounds like it will smooth the way in the future).

It sure will.  You'll be able to define an autonomous administrative 
point for the context of your partition to manage all the aspects of the 
DIT therein.

> I have plugged in my custom LDAP AbstractContextPartition successfully 
> (using the IETF LDAP classes to convert JNDI back to LDAP) and have 
> written some support classes which fetch the remote 
> org.ietf.ldap.LDAPSchema (during doInit() using some credentials known 
> from the partition's configuration properties) and internalise it into 
> the org.apache.ldap.server.schema.GlobalRegistries instance (noting in 
> the future AbstractBootstrapProducer should ideally inherit from an 
> abstract intermediate class to ease writing of classes like mine which 
> create non-bootstrap instances of classes extending 
> AbstractAttributeType/AbstractSchemaObject etc) .   ...what a sentence...

Made sense to me.

> My question is: what is the best way to get the ApacheDS server's 
> org.apache.ldap.server.schema.SchemaService to notice that additions 
> have been made to the GlobalRegistries.

Nothing like this so far.  Really the schema subsystem is extremely 
primitive.  In fact we never intended to use the boostrap mechanism for 
anything other than supporting some critical schema elements for the 
system partition.  Now we're using the bootstrap mechanism for the whole 
shebang.

Part of this is my fault.  I waited until 3672 came since this would 
give me the opportunity to really think about doing schema the right way.

> I notice the addition of AddContextPartitionConfiguration in ApacheDS 
> 0.9.2, but I'm not sure hwo to configure its use via an ApacheDS 
> configuration file (or even if it is indeed the answer to my problems).

Nah this is something else totally.  It's for dynamically adding and 
removing partitions after system startup.

> Failing that the only other approach I can think of is to get the 
> startup infrastructure to create a back-end InitialContext very early 
> so that my custom partition is queried for its schema just after the 
> bootstrap schemas are processed, which sounds a very experimental path 
> to follow.

Yeah you're not in a very good situation actually.  There really is no 
clean way to solve your problem.  We just need to implement the Schema 
subsystem properly.  You interested in giving that a try?

> Note that attempting to refresh the schema via JXplorer never seems to 
> have any effect (I imagine that somewhere in the server it is assumed 
> that there are only bootstrap schemas).

You guessed right ... schema is static on startup.

>  
> Any guidance would be very much appreciated!

Sorry can't really give you a better way here other than to build out 
the schema subsystem so it can support your needs.  This means just 
implementing it properly.  There is no other trick we can use at the moment.

Alex