About LdifPartition and AvlPartition

[Emmanuel Lecharny <el...@apache.org>]

Hi,

atm, the LdifPartition is using an internal AvlPartition to manage the 
entries and index in memory. That means we have to load all the entries 
from disk, and store them in memory. Of course, when some modifications 
are done, they are stored on disk too.

The problem with this approach is that it limits the number of entries 
we are likely to handle in such a partition to a few thousands. It's 
fine for the schema partition and the configuration partition, which 
must be in memory anyway for performance reasons. But in the near 
future, it's a real limitation.

There is also a technical issue with the LdifPartition API : we can't 
switch the current AvlPartition to use another one. It would be really 
cool if we could inject a wrapped partition into the LdifPartition, 
replacing the default AvlPartition, but atm, it's not possible, as we 
don't have an interface for that. We can't use Partition, it's too 
generic, and we can't use BTreePartition, as it does not expose the store.

What would be cool is to improve this part of the Partition API. It's 
not urgent, but I think it has to be done before 2.0. It's not a big 
change too.

thoughts ?

-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org

Re: About LdifPartition and AvlPartition

[Emmanuel Lecharny <el...@apache.org>]

Jeff MAURY wrote:
> I don't know the internal of a Partition but can't you implement some kind
> of lazy loading, ie an entry is loaded when requested by an LDAP request
> into the AvlPartition ? Btw, this should be used with some cache mecanism.
>   
The real problem here is not the entry (we can use a cache for that), 
but the indexes. The internal AvlPartition not only holds the entries, 
but many indexes on these entries (ObjectClass, DN, EntryCSN, entryUUID, 
etc). If we have hundred of thousands of entries, the index size will be 
quite big and can suck memory quite fast too.

A good implementation would be to have a mixed system where entries are 
stored on disk, and index managed with JDBM.

But for that, we must be able to replace the embedded AvlPartition but 
such a system...

And, yes, definitively, we must have a cache in the picture !

-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org

Re: About LdifPartition and AvlPartition

[Jeff MAURY <je...@gmail.com>]

I don't know the internal of a Partition but can't you implement some kind
of lazy loading, ie an entry is loaded when requested by an LDAP request
into the AvlPartition ? Btw, this should be used with some cache mecanism.

Regards
Jeff

On Tue, Sep 8, 2009 at 11:02 PM, Emmanuel Lecharny <el...@apache.org>wrote:

> Hi,
>
> atm, the LdifPartition is using an internal AvlPartition to manage the
> entries and index in memory. That means we have to load all the entries from
> disk, and store them in memory. Of course, when some modifications are done,
> they are stored on disk too.
>
> The problem with this approach is that it limits the number of entries we
> are likely to handle in such a partition to a few thousands. It's fine for
> the schema partition and the configuration partition, which must be in
> memory anyway for performance reasons. But in the near future, it's a real
> limitation.
>
> There is also a technical issue with the LdifPartition API : we can't
> switch the current AvlPartition to use another one. It would be really cool
> if we could inject a wrapped partition into the LdifPartition, replacing the
> default AvlPartition, but atm, it's not possible, as we don't have an
> interface for that. We can't use Partition, it's too generic, and we can't
> use BTreePartition, as it does not expose the store.
>
> What would be cool is to improve this part of the Partition API. It's not
> urgent, but I think it has to be done before 2.0. It's not a big change too.
>
> thoughts ?
>
> --
> --
> cordialement, regards,
> Emmanuel Lécharny
> www.iktek.com
> directory.apache.org
>
>
>


-- 
La mélancolie c’est communiste
Tout le monde y a droit de temps en temps
La mélancolie n’est pas capitaliste
C’est même gratuit pour les perdants
La mélancolie c’est pacifiste
On ne lui rentre jamais dedans
La mélancolie oh tu sais ça existe
Elle se prend même avec des gants
La mélancolie c’est pour les syndicalistes
Il faut juste sa carte de permanent

Miossec (2006)

http://www.jeffmaury.com
http://riadiscuss.jeffmaury.com
http://www.lastfm.fr/listen/user/jeffmaury/personal