Some thoughts I had under my shower this morning...

[Emmanuel Lecharny <el...@gmail.com>]

  The best place when it comes to think about the future...

So we will have some potential issues with the LDAP API : as we now have 
a schema Aware, we have to load the schema into a SchemaManager. Right 
now, all the schema is stored into a subproject (shared/ldap-schema) as 
LDIF files. When we try to initialize the schemaManager, we first 
extract the LDIF to the disk.

This is *very* bad ! When one will try to embed the API (in tomcat for 
instance), one will face serious issues as the file system might not be 
writable.

We must fix that so that we can just read the LDIF directly from the 
resources without having to extract the LDIF files to disk. That also 
means we must not try to update the LDIF on disk when modifying the schema.

On solution would be to use the AvlPartition to back the LdifPartition.

I'm not 100% sure that we don't currently have a workaround for this 
problem, but if so, it must be clearly documented. Otherwise, we have to 
provided a way to fix it.

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Some thoughts I had under my shower this morning...

[Stefan Seelmann <se...@apache.org>]

>> However, we should have a mode where the SchemaManager is not even loaded.
>> Be it a flag would be good enough.
> by default the SchemaManager won't be initialized, user has to explicitly call
> connection.loadSchema(), till then the API will not be schema-aware.

Ah, cool. Thanks Kiran!

Re: Some thoughts I had under my shower this morning...

[Alex Karasulu <ak...@apache.org>]

On Fri, Aug 6, 2010 at 12:20 PM, Kiran Ayyagari <ka...@apache.org>wrote:

> On Fri, Aug 6, 2010 at 2:49 PM, Emmanuel Lecharny <el...@gmail.com>
> wrote:
> >  On 8/6/10 10:45 AM, Kiran Ayyagari wrote:
> >>
> >> On Fri, Aug 6, 2010 at 2:02 PM, Stefan Seelmann<se...@apache.org>
> >>  wrote:
> >>>>
> >>>> So we will have some potential issues with the LDAP API : as we now
> have
> >>>> a
> >>>> schema Aware, we have to load the schema into a SchemaManager. Right
> >>>> now,
> >>>> all the schema is stored into a subproject (shared/ldap-schema) as
> LDIF
> >>>> files. When we try to initialize the schemaManager, we first extract
> the
> >>>> LDIF to the disk.
> >>>>
> >>>> This is *very* bad ! When one will try to embed the API (in tomcat for
> >>>> instance), one will face serious issues as the file system might not
> be
> >>>> writable.
> >>>
> >>> I think that is more worse when one wants to use the API as a client.
> >>> IMO the usage of the API must be lightweight. If I startup my program
> >>> to just do a search I probably don't want that a schema is loaded at
> >>> all, because it is just too heavy. Can we have a DummySchemaManager
> >>> implementation, that just does no validation and normalization?
> >>
> >> not really, we let schema manager use the in-memory LDIF loader
> >> (JarLdifSchemaLoader) so
> >> no files will be written to disk.
> >
> > Ok, so we are fine.
> >
> > However, we should have a mode where the SchemaManager is not even
> loaded.
> > Be it a flag would be good enough.
> by default the SchemaManager won't be initialized, user has to explicitly
> call
> connection.loadSchema(), till then the API will not be schema-aware.
>
>
This is where we can have an API flag to control whether or not the schema
is loaded. This works out just perfectly.

-- 
Alex Karasulu
My Blog :: http://www.jroller.com/akarasulu/
Apache Directory Server :: http://directory.apache.org
Apache MINA :: http://mina.apache.org
To set up a meeting with me: http://tungle.me/AlexKarasulu

Re: Some thoughts I had under my shower this morning...

[Kiran Ayyagari <ka...@apache.org>]

On Fri, Aug 6, 2010 at 2:49 PM, Emmanuel Lecharny <el...@gmail.com> wrote:
>  On 8/6/10 10:45 AM, Kiran Ayyagari wrote:
>>
>> On Fri, Aug 6, 2010 at 2:02 PM, Stefan Seelmann<se...@apache.org>
>>  wrote:
>>>>
>>>> So we will have some potential issues with the LDAP API : as we now have
>>>> a
>>>> schema Aware, we have to load the schema into a SchemaManager. Right
>>>> now,
>>>> all the schema is stored into a subproject (shared/ldap-schema) as LDIF
>>>> files. When we try to initialize the schemaManager, we first extract the
>>>> LDIF to the disk.
>>>>
>>>> This is *very* bad ! When one will try to embed the API (in tomcat for
>>>> instance), one will face serious issues as the file system might not be
>>>> writable.
>>>
>>> I think that is more worse when one wants to use the API as a client.
>>> IMO the usage of the API must be lightweight. If I startup my program
>>> to just do a search I probably don't want that a schema is loaded at
>>> all, because it is just too heavy. Can we have a DummySchemaManager
>>> implementation, that just does no validation and normalization?
>>
>> not really, we let schema manager use the in-memory LDIF loader
>> (JarLdifSchemaLoader) so
>> no files will be written to disk.
>
> Ok, so we are fine.
>
> However, we should have a mode where the SchemaManager is not even loaded.
> Be it a flag would be good enough.
by default the SchemaManager won't be initialized, user has to explicitly call
connection.loadSchema(), till then the API will not be schema-aware.

Kiran Ayyagari

Re: Some thoughts I had under my shower this morning...

[Emmanuel Lecharny <el...@gmail.com>]

  On 8/6/10 10:45 AM, Kiran Ayyagari wrote:
> On Fri, Aug 6, 2010 at 2:02 PM, Stefan Seelmann<se...@apache.org>  wrote:
>>> So we will have some potential issues with the LDAP API : as we now have a
>>> schema Aware, we have to load the schema into a SchemaManager. Right now,
>>> all the schema is stored into a subproject (shared/ldap-schema) as LDIF
>>> files. When we try to initialize the schemaManager, we first extract the
>>> LDIF to the disk.
>>>
>>> This is *very* bad ! When one will try to embed the API (in tomcat for
>>> instance), one will face serious issues as the file system might not be
>>> writable.
>> I think that is more worse when one wants to use the API as a client.
>> IMO the usage of the API must be lightweight. If I startup my program
>> to just do a search I probably don't want that a schema is loaded at
>> all, because it is just too heavy. Can we have a DummySchemaManager
>> implementation, that just does no validation and normalization?
> not really, we let schema manager use the in-memory LDIF loader
> (JarLdifSchemaLoader) so
> no files will be written to disk.

Ok, so we are fine.

However, we should have a mode where the SchemaManager is not even 
loaded. Be it a flag would be good enough.

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Some thoughts I had under my shower this morning...

[Kiran Ayyagari <ka...@apache.org>]

On Fri, Aug 6, 2010 at 2:02 PM, Stefan Seelmann <se...@apache.org> wrote:
>> So we will have some potential issues with the LDAP API : as we now have a
>> schema Aware, we have to load the schema into a SchemaManager. Right now,
>> all the schema is stored into a subproject (shared/ldap-schema) as LDIF
>> files. When we try to initialize the schemaManager, we first extract the
>> LDIF to the disk.
>>
>> This is *very* bad ! When one will try to embed the API (in tomcat for
>> instance), one will face serious issues as the file system might not be
>> writable.
>
> I think that is more worse when one wants to use the API as a client.
> IMO the usage of the API must be lightweight. If I startup my program
> to just do a search I probably don't want that a schema is loaded at
> all, because it is just too heavy. Can we have a DummySchemaManager
> implementation, that just does no validation and normalization?

not really, we let schema manager use the in-memory LDIF loader
(JarLdifSchemaLoader) so
no files will be written to disk.

Kiran Ayyagari

Re: Some thoughts I had under my shower this morning...

[Stefan Seelmann <se...@apache.org>]

> So we will have some potential issues with the LDAP API : as we now have a
> schema Aware, we have to load the schema into a SchemaManager. Right now,
> all the schema is stored into a subproject (shared/ldap-schema) as LDIF
> files. When we try to initialize the schemaManager, we first extract the
> LDIF to the disk.
>
> This is *very* bad ! When one will try to embed the API (in tomcat for
> instance), one will face serious issues as the file system might not be
> writable.

I think that is more worse when one wants to use the API as a client.
IMO the usage of the API must be lightweight. If I startup my program
to just do a search I probably don't want that a schema is loaded at
all, because it is just too heavy. Can we have a DummySchemaManager
implementation, that just does no validation and normalization?

Kind Regards,
Stefan

Re: Some thoughts I had under my shower this morning...

[Kiran Ayyagari <ka...@apache.org>]

On Fri, Aug 6, 2010 at 1:57 PM, Emmanuel Lecharny <el...@gmail.com> wrote:
>  On 8/6/10 10:15 AM, Kiran Ayyagari wrote:
>>
>>> IMO, this is barely acceptable for API 0.2, but has to be fixed for 0.3.
>>> There is no reason to extract to disk when all the ldifs are available in
>>> a
>>> jar, as it's just a matter of loading them in memory.
>>>
>>> It should be a 2 hours work to get the code read the jar and load the
>>> schema
>>> in memory...
>>
>> it is already possible to do this using JarLdifSchemaLoader , but note
>> that this will always
>> contain the bundled default schemas only. A user can only add custom
>> schema if the
>> schema directory is present on disk.
>
> Ah, so we are golden.
>
> Is there a place where this is documented (even roughly) ?
hmm, no,
>
> I think we have some users facing this issue.
>
> Btw, I don't really care if users can't extend the schema when they can't
> write on disk. But we can discuss about allowing this too.
let me be a bit more clear here,
from the API pov all that it needs is just a schema manager, so we can
simply keep all the
data in memory(assuming we don't want to update the schema)
(Note: currently, API operates in this in-memory mode, so there is no
extraction of LDIF files)

but for a embedded server the case is different, it requires a LDIF
partition for which we need to
extract the files on to the disk, ( cause LdifPartition currently
works this way).
To make this work in a container environment we have a fix.

Kiran Ayyagari

Re: Some thoughts I had under my shower this morning...

[Emmanuel Lecharny <el...@gmail.com>]

  On 8/6/10 10:15 AM, Kiran Ayyagari wrote:
>
>> IMO, this is barely acceptable for API 0.2, but has to be fixed for 0.3.
>> There is no reason to extract to disk when all the ldifs are available in a
>> jar, as it's just a matter of loading them in memory.
>>
>> It should be a 2 hours work to get the code read the jar and load the schema
>> in memory...
> it is already possible to do this using JarLdifSchemaLoader , but note
> that this will always
> contain the bundled default schemas only. A user can only add custom
> schema if the
> schema directory is present on disk.

Ah, so we are golden.

Is there a place where this is documented (even roughly) ?

I think we have some users facing this issue.

Btw, I don't really care if users can't extend the schema when they 
can't write on disk. But we can discuss about allowing this too.

Thanks Kiran.

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Some thoughts I had under my shower this morning...

[Kiran Ayyagari <ka...@apache.org>]

On Fri, Aug 6, 2010 at 4:09 AM, Emmanuel Lecharny <el...@gmail.com> wrote:
>  On 8/5/10 8:31 PM, Alex Karasulu wrote:
>>
>> we do have a workaround/fix for this (by setting a special VM option
>>>
>>> specifying the location of the
>>> schema directory or the jar file in which schema files are present)
>>>
>>>
>> This workaround IMHO is sufficient for embedding into containers. I would
>> not alter the whole server schema loading behavior just for this very
>> special case.
>>
>> Alex
>>
> IMO, this is barely acceptable for API 0.2, but has to be fixed for 0.3.
> There is no reason to extract to disk when all the ldifs are available in a
> jar, as it's just a matter of loading them in memory.
>
> It should be a 2 hours work to get the code read the jar and load the schema
> in memory...

it is already possible to do this using JarLdifSchemaLoader , but note
that this will always
contain the bundled default schemas only. A user can only add custom
schema if the
schema directory is present on disk.

Kiran Ayyagari

Re: Some thoughts I had under my shower this morning...

[Emmanuel Lecharny <el...@gmail.com>]

  On 8/5/10 8:31 PM, Alex Karasulu wrote:
> we do have a workaround/fix for this (by setting a special VM option
>> specifying the location of the
>> schema directory or the jar file in which schema files are present)
>>
>>
> This workaround IMHO is sufficient for embedding into containers. I would
> not alter the whole server schema loading behavior just for this very
> special case.
>
> Alex
>
IMO, this is barely acceptable for API 0.2, but has to be fixed for 0.3. 
There is no reason to extract to disk when all the ldifs are available 
in a jar, as it's just a matter of loading them in memory.

It should be a 2 hours work to get the code read the jar and load the 
schema in memory...


-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Some thoughts I had under my shower this morning...

[Alex Karasulu <ak...@apache.org>]

On Thu, Aug 5, 2010 at 8:28 PM, Kiran Ayyagari <ka...@apache.org> wrote:

> On Thu, Aug 5, 2010 at 8:55 PM, Emmanuel Lecharny <el...@gmail.com>
> wrote:
> >  The best place when it comes to think about the future...
> >
> > So we will have some potential issues with the LDAP API : as we now have
> a
> > schema Aware, we have to load the schema into a SchemaManager. Right now,
> > all the schema is stored into a subproject (shared/ldap-schema) as LDIF
> > files. When we try to initialize the schemaManager, we first extract the
> > LDIF to the disk.
> >
> > This is *very* bad ! When one will try to embed the API (in tomcat for
> > instance), one will face serious issues as the file system might not be
> > writable.
> hmm, no, at least we can have access to the container's or system's
> temporary area
> >
> > We must fix that so that we can just read the LDIF directly from the
> > resources without having to extract the LDIF files to disk. That also
> means
> > we must not try to update the LDIF on disk when modifying the schema.
> currently the client cannot update the schema but IMO we should add
> such a feature
> sooner or later. For this to work efficiently it is good to have the
> schema stored on disk
> (just like the way Studio does it)
> >
> > On solution would be to use the AvlPartition to back the LdifPartition.
> yeah but currently LdifPartition can only work if the files are stored on
> disk
> >
> > I'm not 100% sure that we don't currently have a workaround for this
> > problem, but if so, it must be clearly documented. Otherwise, we have to
> > provided a way to fix it.
> we do have a workaround/fix for this (by setting a special VM option
> specifying the location of the
> schema directory or the jar file in which schema files are present)
>
>
This workaround IMHO is sufficient for embedding into containers. I would
not alter the whole server schema loading behavior just for this very
special case.

Alex

Re: Some thoughts I had under my shower this morning...

[Kiran Ayyagari <ka...@apache.org>]

On Thu, Aug 5, 2010 at 8:55 PM, Emmanuel Lecharny <el...@gmail.com> wrote:
>  The best place when it comes to think about the future...
>
> So we will have some potential issues with the LDAP API : as we now have a
> schema Aware, we have to load the schema into a SchemaManager. Right now,
> all the schema is stored into a subproject (shared/ldap-schema) as LDIF
> files. When we try to initialize the schemaManager, we first extract the
> LDIF to the disk.
>
> This is *very* bad ! When one will try to embed the API (in tomcat for
> instance), one will face serious issues as the file system might not be
> writable.
hmm, no, at least we can have access to the container's or system's
temporary area
>
> We must fix that so that we can just read the LDIF directly from the
> resources without having to extract the LDIF files to disk. That also means
> we must not try to update the LDIF on disk when modifying the schema.
currently the client cannot update the schema but IMO we should add
such a feature
sooner or later. For this to work efficiently it is good to have the
schema stored on disk
(just like the way Studio does it)
>
> On solution would be to use the AvlPartition to back the LdifPartition.
yeah but currently LdifPartition can only work if the files are stored on disk
>
> I'm not 100% sure that we don't currently have a workaround for this
> problem, but if so, it must be clearly documented. Otherwise, we have to
> provided a way to fix it.
we do have a workaround/fix for this (by setting a special VM option
specifying the location of the
schema directory or the jar file in which schema files are present)

Kiran Ayyagari