Schema updates

[Emmanuel Lécharny <el...@gmail.com>]

Hi guys,

I'm trying to do something that is not that specific : on a client tool,
load a SchemaManager with all the default schema presents (enabled and
disabled), then load a new external schema. And I'm hitting a wall here
: there is no simple way to get a schema which is not present in the
list of existing schemas being added afterward.

This is quite a pain, and I'd like to extend the SchemaManager interface
to allow something like : SchemaManager.add( String schemaFile[,
SchemaLoader] ).

This is not exactly simple, considering the format this schema file will
use. We have three kind of supported formats :
- ldif : one ldif file per schema element
- single ldif : one single ldif file containing all the elements
- JAR : a jar file containing a ldif format schema (1 format)

When you create a schemaManager, you inject a SchemaLoader into it, and
this is used for the lifetime of this SchemaManager. That makes it
impossible to use another SchemaLoader than the one that has been used
initially. Sadly, we still need the previous SchemaLoader because we
don't process all the schemas when we initialize the SchemaManager : the
disabled schemas aren't read at all. So we may need it later, if the
newly added schema requires a dependency from the old SchemaLoader.

At this point, I'm a bit stuck with the current API, and even though
there are some ways to deal with that, none of the solution is
convenient from a User POV, and we are talking about an API here.


So the solution I will come with is to associate each Schema with its
schema loader, instead of associate the SchemaLoader to the
SchemaManager. This way, we will be able to have more than one
SchemaLoader used by the SchemaManager.


Another thing : we do need a OpenLDAPSchemaLoader, able to process
OpenLDAP schema formatted files (currently all our schemaLoaders just
swallow LDIF files, which is far form being convenient). We have all the
logic available to parse OpenLDAP schemas, it's just a matter of
wrapping that into a dedicated SchemaLoader.

Thoughts ?

Re: Schema updates

[Emmanuel Lécharny <el...@gmail.com>]

Le 06/02/16 16:54, Kiran Ayyagari a écrit :
> On Sat, Feb 6, 2016 at 2:26 PM, Emmanuel Lécharny <el...@gmail.com>
> wrote:
>
>> Hi guys,
>>
>> I'm trying to do something that is not that specific : on a client tool,
>> load a SchemaManager with all the default schema presents (enabled and
>> disabled), then load a new external schema. And I'm hitting a wall here
>> : there is no simple way to get a schema which is not present in the
>> list of existing schemas being added afterward.
>>
>> This is quite a pain, and I'd like to extend the SchemaManager interface
>> to allow something like : SchemaManager.add( String schemaFile[,
>> SchemaLoader] ).
>>
>> This is not exactly simple, considering the format this schema file will
>> use. We have three kind of supported formats :
>> - ldif : one ldif file per schema element
>> - single ldif : one single ldif file containing all the elements
>> - JAR : a jar file containing a ldif format schema (1 format)
>>
>> the above mentioned differ only in the way of finding the schema file
> after that
> they all parse the LDIF file in the same way

Not exactly true. When you want to load a schema, it will try to check
the dependencis, in order to load the one that are not already loaded :

        for ( String depName : deps )
        {
            if ( registries.isSchemaLoaded( schemaName ) )
            {
                // The schema is already loaded. Loop on the next schema
                continue;
            }
            else
            {
                // Call recursively this method
                Schema schemaDep = schemaLoader.getSchema( depName );
                loadDepsFirst( registries, schemaDep );
            }

and here, we use the SchemaLoader the SchemaManager has been initialized
with. This is where we have a problem.

>
>> When you create a schemaManager, you inject a SchemaLoader into it, and
>> this is used for the lifetime of this SchemaManager. That makes it
>> impossible to use another SchemaLoader than the one that has been used
>> initially. Sadly, we still need the previous SchemaLoader because we
>> don't process all the schemas when we initialize the SchemaManager : the
>> disabled schemas aren't read at all. So we may need it later, if the
>> newly added schema requires a dependency from the old SchemaLoader.
>>
> IMO what we actually need is a way to add new Schemas (as in an instance of
> Schema class)
> to an already instantiated SchemaManager (using whichever loader during
> initialization)

That forces us to use a schema written using teh exact same format that
the initial SchemaLoader is supporting. In my case, the SchemManager is
using the LdifSchemaLoader, and I want to use the SingleLdifSchemaLoader
format. We can imagine other configuration (for instance, loading a
OpenLDAP formatted schema...)

>
>> At this point, I'm a bit stuck with the current API, and even though
>> there are some ways to deal with that, none of the solution is
>> convenient from a User POV, and we are talking about an API here.
>>
>> I think we have all the necessary pieces to get this done with a few
> changes.
>
> One solution is to modify the SingleLdifSchemaLoader(SLSL),
> DefaultSchemaManager and Schema classes.
>
> 1. add a new method, addSchemaObject(SchemaObject so) to DefaultSchema class
>   which when called populates the 'content' field
>
> 2. add a 'schema' field of type DefaultSchema in SLSL
>
> 3. add a new method, getSchema(), in SLSL (I am assuming this for a single
> schema defined in file,
> to support multiple schemas this can return a list)
>
>   a. make getSchema() call various loadXXX() methods
>
>   b. update the loadXXX() methods to create the appropriate SchemaObject
> instances
>     and populate the 'content' field of 'schema'
>
> 4. change the visibility of addSchemaObject() of DefaultSchemaManager to
> public
>
> 5. now, call getSchema() of SLSL and iterate over the 'content' and call
> addSchemaObject()
>
> Now, this is all theory ;) and I think it will work

The SchemaManager does that when it comes to load a Schema :

loadWithDeps( Schema... )
  loadDepsFirst( Registries, Schemas )
    ... (here we load the dependencies, which at the bottom, do the same
thing as what we will see on the next line)
    load( Registries, Schema )
      addSchemaObjects( Schema, Registries )
        addComparators( Schema, Registries ) // Same think for AT, OC, etc
          for ( Entry entry : schemaLoader.loadComparators( schema ) )
          {
            LdapComparator<?> comparator = factory.getLdapComparator(
this, entry, registries, schema.getSchemaName() );

            addSchemaObject( registries, comparator, schema );
          }

As you can see, we use the SchemaManager's declared SchemaLoader, which
is not what we want.

Having each Schema associated with its SchemaLoader would make it
possible to use a different SchemaLoader :

        addComparators( Schema schema, Registries registries ) // Same
think for AT, OC, etc
          for ( Entry entry : schema.getSchemaLoader().loadComparators(
schema ) )
          {
            LdapComparator<?> comparator = factory.getLdapComparator(
this, entry, registries, schema.getSchemaName() );

            addSchemaObject( registries, comparator, schema );
          }

Way simpler than any other change, IMO.
>
>
>> So the solution I will come with is to associate each Schema with its
>> schema loader, instead of associate the SchemaLoader to the
>> SchemaManager. This way, we will be able to have more than one
>> SchemaLoader used by the SchemaManager.
>>
>> IMO a loader should give us a schema rather than a Schema containing a
> loader.

But the SchemaLoader does not know about Schemas. Technically, it's the
other way out.

Thanks Kiran !

Re: Schema updates

[Kiran Ayyagari <ka...@apache.org>]

On Sat, Feb 6, 2016 at 2:26 PM, Emmanuel Lécharny <el...@gmail.com>
wrote:

> Hi guys,
>
> I'm trying to do something that is not that specific : on a client tool,
> load a SchemaManager with all the default schema presents (enabled and
> disabled), then load a new external schema. And I'm hitting a wall here
> : there is no simple way to get a schema which is not present in the
> list of existing schemas being added afterward.
>
> This is quite a pain, and I'd like to extend the SchemaManager interface
> to allow something like : SchemaManager.add( String schemaFile[,
> SchemaLoader] ).
>
> This is not exactly simple, considering the format this schema file will
> use. We have three kind of supported formats :
> - ldif : one ldif file per schema element
> - single ldif : one single ldif file containing all the elements
> - JAR : a jar file containing a ldif format schema (1 format)
>
> the above mentioned differ only in the way of finding the schema file
after that
they all parse the LDIF file in the same way

> When you create a schemaManager, you inject a SchemaLoader into it, and
> this is used for the lifetime of this SchemaManager. That makes it
> impossible to use another SchemaLoader than the one that has been used
> initially. Sadly, we still need the previous SchemaLoader because we
> don't process all the schemas when we initialize the SchemaManager : the
> disabled schemas aren't read at all. So we may need it later, if the
> newly added schema requires a dependency from the old SchemaLoader.
>
IMO what we actually need is a way to add new Schemas (as in an instance of
Schema class)
to an already instantiated SchemaManager (using whichever loader during
initialization)

>
> At this point, I'm a bit stuck with the current API, and even though
> there are some ways to deal with that, none of the solution is
> convenient from a User POV, and we are talking about an API here.
>
> I think we have all the necessary pieces to get this done with a few
changes.

One solution is to modify the SingleLdifSchemaLoader(SLSL),
DefaultSchemaManager and Schema classes.

1. add a new method, addSchemaObject(SchemaObject so) to DefaultSchema class
  which when called populates the 'content' field

2. add a 'schema' field of type DefaultSchema in SLSL

3. add a new method, getSchema(), in SLSL (I am assuming this for a single
schema defined in file,
to support multiple schemas this can return a list)

  a. make getSchema() call various loadXXX() methods

  b. update the loadXXX() methods to create the appropriate SchemaObject
instances
    and populate the 'content' field of 'schema'

4. change the visibility of addSchemaObject() of DefaultSchemaManager to
public

5. now, call getSchema() of SLSL and iterate over the 'content' and call
addSchemaObject()

Now, this is all theory ;) and I think it will work


> So the solution I will come with is to associate each Schema with its
> schema loader, instead of associate the SchemaLoader to the
> SchemaManager. This way, we will be able to have more than one
> SchemaLoader used by the SchemaManager.
>
> IMO a loader should give us a schema rather than a Schema containing a
loader.

>
> Another thing : we do need a OpenLDAPSchemaLoader, able to process
> OpenLDAP schema formatted files (currently all our schemaLoaders just
> swallow LDIF files, which is far form being convenient). We have all the
> logic available to parse OpenLDAP schemas, it's just a matter of
> wrapping that into a dedicated SchemaLoader.
>
> sure, and this loader can implement the above said schema extraction (if
that algorithm works)

> Thoughts ?
>
>
Kiran Ayyagari
http://keydap.com