[LDIF partition] File names

[Emmanuel Lecharny <el...@apache.org>]

Hi,

after having thought about the OS filesystem name limitations (forbidden 
chars, case sensitivity, length), here is what I propose for the file 
names on a ldif partition :

- a file/directory name will be composed by the AT short name (or the 
oid if the AT does not have a short name) in lowercase, followed by the 
'=' sign, and the normalized value, ad ".ldif" as a suffix
- if we have special characters in the normalized value (like '/', '<', 
'>', '?', ':', '|', '&', '"', ' ', '*', they will be escaped using '\'
- the normalized value will be lowercased
- we don't handle file names longer than 255 characters
- we don't handle case sensitive normalized values (ou=test.ldif and 
ou=TEST.ldif won't exist on the same directory)
- if an entry has a child at least, a directory will be created, without 
the .ldif extension

thoughts ?

-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org

Re: [LDIF partition] File names

[Alex Karasulu <ak...@gmail.com>]

OK then let's carry on if you can whip this together that fast let's do it.

Alex

On Sun, Sep 6, 2009 at 12:45 AM, Emmanuel Lecharny <el...@apache.org>wrote:

> Alex Karasulu wrote:
>
>> I think it's a waste of time trying to force fit the FS namespace to the
>> LDAP namespace.  I think we should just load LDIF files if they exist in
>> or
>> below some partition directory.  They can contain any LDIF and have any
>> number of LDIF entries in them.  It's up to the server to track files and
>> contents.
>>
>> This way there is no constraint on users to manage any kind of layout on
>> the
>> FS to have a partition that reads and uses LDIFs.
>>
>>
>
> Atm, what the LdifPartition should do is really to be able to support the
> SchemaPartition.
>
> That mean : manipulating Ldif files with one entry = one file.
>
> Also the current LdifPartiton, as it has been written, does not support
> anything but a FS image of the Dit structure.
>
> We can think about dropping a big Ldif file into a directory, and expect
> the LdifPartition to deal with it, even if it means updating or adding an
> entry in the middle of a 10 Mb file, with all the consequences it has like
> dealing with concurrent access, and performances issues. But atm, this is
> *not* what the LdifPartition code does.
>
> And doing that does not solve the problem of having to pick the right file
> in which we have to update an entry, assuming a user dump 10 ldif in the
> working directory, as we have no link between an entry in memory and the
> Ldif file it is stored in. Nor we have any mechanism to generate a file or
> to pick a file to store a new entry in.
>
> In other words, right now, with the existing code, which is an exact
> replication of the DIT structure, I see no other ways to go but to do :
> 1 entry = 1 ldif file
> and
> entry RDN => file name.
>
> with a best effort to get a name which covers 99% of the limitations.
>
> Like it or not, but a minimal handling of the few cases I mentioned in my
> mail will be a matter of two hours, tests included, when any other solution
> will cost a couple of days, at least, as we already have an existing code
> which does 90% of what we need.
>
> Being a good or bad idea to mimic the DIT structure is irrelevant atm,
> IMHO. We can discuss the pros and cons of it later.
>
>


-- 
Alex Karasulu
My Blog :: http://www.jroller.com/akarasulu/
Apache Directory Server :: http://directory.apache.org
Apache MINA :: http://mina.apache.org

Re: [LDIF partition] File names

[Emmanuel Lecharny <el...@apache.org>]

Alex Karasulu wrote:
> I think it's a waste of time trying to force fit the FS namespace to the
> LDAP namespace.  I think we should just load LDIF files if they exist in or
> below some partition directory.  They can contain any LDIF and have any
> number of LDIF entries in them.  It's up to the server to track files and
> contents.
>
> This way there is no constraint on users to manage any kind of layout on the
> FS to have a partition that reads and uses LDIFs.
>   

Atm, what the LdifPartition should do is really to be able to support 
the SchemaPartition.

That mean : manipulating Ldif files with one entry = one file.

Also the current LdifPartiton, as it has been written, does not support 
anything but a FS image of the Dit structure.

We can think about dropping a big Ldif file into a directory, and expect 
the LdifPartition to deal with it, even if it means updating or adding 
an entry in the middle of a 10 Mb file, with all the consequences it has 
like dealing with concurrent access, and performances issues. But atm, 
this is *not* what the LdifPartition code does.

And doing that does not solve the problem of having to pick the right 
file in which we have to update an entry, assuming a user dump 10 ldif 
in the working directory, as we have no link between an entry in memory 
and the Ldif file it is stored in. Nor we have any mechanism to generate 
a file or to pick a file to store a new entry in.

In other words, right now, with the existing code, which is an exact 
replication of the DIT structure, I see no other ways to go but to do :
1 entry = 1 ldif file
and
entry RDN => file name.

with a best effort to get a name which covers 99% of the limitations.

Like it or not, but a minimal handling of the few cases I mentioned in 
my mail will be a matter of two hours, tests included, when any other 
solution will cost a couple of days, at least, as we already have an 
existing code which does 90% of what we need.

Being a good or bad idea to mimic the DIT structure is irrelevant atm, 
IMHO. We can discuss the pros and cons of it later.

Re: [LDIF partition] File names

[Alex Karasulu <ak...@gmail.com>]

I think it's a waste of time trying to force fit the FS namespace to the
LDAP namespace.  I think we should just load LDIF files if they exist in or
below some partition directory.  They can contain any LDIF and have any
number of LDIF entries in them.  It's up to the server to track files and
contents.

This way there is no constraint on users to manage any kind of layout on the
FS to have a partition that reads and uses LDIFs.

On Sat, Sep 5, 2009 at 8:06 PM, Emmanuel Lecharny <el...@apache.org>wrote:

> Hi,
>
> after having thought about the OS filesystem name limitations (forbidden
> chars, case sensitivity, length), here is what I propose for the file names
> on a ldif partition :
>
> - a file/directory name will be composed by the AT short name (or the oid
> if the AT does not have a short name) in lowercase, followed by the '='
> sign, and the normalized value, ad ".ldif" as a suffix
> - if we have special characters in the normalized value (like '/', '<',
> '>', '?', ':', '|', '&', '"', ' ', '*', they will be escaped using '\'
> - the normalized value will be lowercased
> - we don't handle file names longer than 255 characters
> - we don't handle case sensitive normalized values (ou=test.ldif and
> ou=TEST.ldif won't exist on the same directory)
> - if an entry has a child at least, a directory will be created, without
> the .ldif extension
>
> thoughts ?
>
> --
> --
> cordialement, regards,
> Emmanuel Lécharny
> www.iktek.com
> directory.apache.org
>
>
>


-- 
Alex Karasulu
My Blog :: http://www.jroller.com/akarasulu/
Apache Directory Server :: http://directory.apache.org
Apache MINA :: http://mina.apache.org