You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Emmanuel Lecharny <el...@gmail.com> on 2011/07/20 19:12:49 UTC
Partitions Heads up : resent with correct formating for the hierarchies
Hi guys,
sorry for having been silent for a few days, I was playing locally with
Partitions. Let me give you some feedback on what I'm currently doing...
The idea is to refactor the way we work with Partitions. It doe snot
make too much sense to have Partitions on one side and Store on another
one, so I have merged those two guys (it was separated last year, but
Alex told me that was a mistake).
The current Partition and Store hierarchies are ([] = Abstract classes,
() = Interfaces, <> = concrete classes ) :
(Partition)
[AbstractPartition]
[AbstractBTreePartition]
[AbstractLdifPartition]
LdifPartition
ReadOnlyConfigurationPartition
SingleFileLdifPartition
[AbstractXdbmPartition]
AvlPartition
JdbmPartition
DefaultNexusPartition
SchemaPartition
(PartitionNexus)
DefaultNexusPartition
(Store)
[AbstractStore]
AvlStore
JdbmStore
Once we realize that the (Avl/jdbm)Partition are backed by an
(Avl/jdbm)Store, there is no reason not to have the classes merged. For
convenience, we can keep the Store interface (as it exposes some methods
specific to the backend.
Now, we can go a bit further : the Ldif partitions are all encapsulating
an AvlPartition, because we load them in memory and it allows the system
to create a cursor on top of them. There is no real reason not to have
the Ldif partitions class inherit directly from the AvlPartition. That
would give the following hierarchy :
(Partition)
[AbstractPartition]
[AbstractBTreePartition]
[AbstractXdbmPartition]
AvlPartition
[AbstractLdifPartition]
LdifPartition
ReadOnlyConfigurationPartition>
SingleFileLdifPartition>
JdbmPartition
DefaultNexusPartition
SchemaPartition
(PartitionNexus)
DefaultNexusPartition
As we keep the Store interface, we will make the
AbstractBTreePartitionimplement it :
(Store)
[AbstractBTreePartition]
The SchemaPartition and the DefaultNexusPartition are two different beasts :
- the SchemaPartition have a wrapped partition, which can be either a
Ldif partition, or a Jdbm partition, or whatever partition we want.
- the DefaultNexusPartition is just a proxy which reroute all the
requests to the associated partition, depending on the operation DN.
Right now, this is what I'm having, and after a lot of cleanup, I have
something which is quite consistent. It allows us to easily add a new
Partition implementation §for instance, the HBase partition)
Now, there are some slight issues that needed to be addressed :
- first, all those Partitions now take a SchemaManager as a constructor
parameter. This is not a problem, as the SchemaManager instance is
always created *before* we create any partition.
- second, partitions initialization was a nightmare. We had a
Store.init(SchemaManager) method, a final AbstractPartition.initialize()
method and an abstract AbstractPartition.doInit() method. I removed the
Store.init(SchemaManager), and refactored the doInti() methods in ordr
to have a proper initialization sequence (it's not perfect though).
I do think we should merge the initialization() and doInit() method at
some point, I don't exactly know what is the best options atm.
What remains to be done is to cleanup the DirectoryService
initialization and the DirectoryServiceFactory initialization.
Basically, partitions initialization is depending on a complex process,
requiring many steps to be done :
- setting the Id
- setting the SuffixDn
- creating the underlying storage if needed
- associating an optimizer, a cursorBuilder, an evaluatorBuilder and a
SearchEngine to the partition (for BTreePartitions)
- defining and initializing the indexes,creating the master table (for
the BTreePartitions)
- creating the contextEntry (for the LdifPartitions)
Most of those steps are done in the doInit() methods, but it requires
the presence of configuration elements (Id, Indexes, SuffixDn, etc)
which *have* to be injected *before* we initialize the partition.
The way we initialize the DirectoryService requires a lot of cleanup
too. For instance, the DefaultNexusPartition initialize the System
partition, and all the other partitions, which does not make a lot of
sense to me. Partitions should be associated with the DirectoryService,
not with the DefaultNexusPartition. For the same reason, the optimizer
and searchEngine should not be associated with a specific partition, as
it makes thing way more complex when we try to do search across partitions.
One more thing would be to use a unique identifier across all the
partitions. Currently, we use a Long, starting at 1L for each
partition's contextEntry. Alex suggested that we use the entryUUID
instead, something we can do easily, as the classes are all generic (it
uses ID).
That's pretty much it. I'm working on my computer, because I was offline
last week-end, but I will create a branch and commit all my changes into
it so that you can land an eye.
Feel free to provide any feedback !
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Partitions Heads up : resent with correct formating for the hierarchies
Posted by Emmanuel Lecharny <el...@gmail.com>.
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 19:01.362s
Got it working with the refactored partitions !
Time for some other cleanup now.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Partitions Heads up : resent with correct formating for the hierarchies
Posted by Emmanuel Lécharny <el...@apache.org>.
On 7/20/11 8:50 PM, Stefan Seelmann wrote:
> Sounds all reasonable. The only thing that bugs me is that the
> hierarchy gets bigger, I think of "Composition over Inheritance". But
> I have no real argument against your proposal.
I have still to think about the PartitionNexus interface, I'm not 100%
convinced it should extend the Partition interface.
>
> One open question: in your final class hierarchy there are two
> abstract classes in a row: AbstractBTreePartition and
> AbstractXdbmPartition: I think it should be possbile to merge them
> too, isn't it?
yes, but atm, I'm trying to get the tests passing before going any further.
There are some few things I haven't talked about :
- SchemaService class should disappear. It's useless
- I was thinking about merging the initialize() and doInit() methods
- all of what I'm currently doing has one purpose : being able to
simplify the init phase in order to be ready for OSGi
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Partitions Heads up : resent with correct formating for the hierarchies
Posted by Emmanuel Lécharny <el...@apache.org>.
On 7/20/11 8:50 PM, Stefan Seelmann wrote:
> Sounds all reasonable. The only thing that bugs me is that the
> hierarchy gets bigger, I think of "Composition over Inheritance". But
> I have no real argument against your proposal.
>
> One open question: in your final class hierarchy there are two
> abstract classes in a row: AbstractBTreePartition and
> AbstractXdbmPartition: I think it should be possbile to merge them
> too, isn't it?
I have now merged those two abstract classes. One less level in he
hierarchy !
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Partitions Heads up : resent with correct formating for the hierarchies
Posted by Stefan Seelmann <se...@apache.org>.
Sounds all reasonable. The only thing that bugs me is that the
hierarchy gets bigger, I think of "Composition over Inheritance". But
I have no real argument against your proposal.
One open question: in your final class hierarchy there are two
abstract classes in a row: AbstractBTreePartition and
AbstractXdbmPartition: I think it should be possbile to merge them
too, isn't it?
Kind Regards,
Stefan
On Wed, Jul 20, 2011 at 7:12 PM, Emmanuel Lecharny <el...@gmail.com> wrote:
> Hi guys,
>
> sorry for having been silent for a few days, I was playing locally with
> Partitions. Let me give you some feedback on what I'm currently doing...
>
> The idea is to refactor the way we work with Partitions. It doe snot make
> too much sense to have Partitions on one side and Store on another one, so I
> have merged those two guys (it was separated last year, but Alex told me
> that was a mistake).
>
> The current Partition and Store hierarchies are ([] = Abstract classes, () =
> Interfaces, <> = concrete classes ) :
>
> (Partition)
> [AbstractPartition]
> [AbstractBTreePartition]
> [AbstractLdifPartition]
> LdifPartition
> ReadOnlyConfigurationPartition
> SingleFileLdifPartition
> [AbstractXdbmPartition]
> AvlPartition
> JdbmPartition
> DefaultNexusPartition
> SchemaPartition
>
> (PartitionNexus)
> DefaultNexusPartition
>
> (Store)
> [AbstractStore]
> AvlStore
> JdbmStore
>
> Once we realize that the (Avl/jdbm)Partition are backed by an
> (Avl/jdbm)Store, there is no reason not to have the classes merged. For
> convenience, we can keep the Store interface (as it exposes some methods
> specific to the backend.
>
> Now, we can go a bit further : the Ldif partitions are all encapsulating an
> AvlPartition, because we load them in memory and it allows the system to
> create a cursor on top of them. There is no real reason not to have the Ldif
> partitions class inherit directly from the AvlPartition. That would give the
> following hierarchy :
>
> (Partition)
> [AbstractPartition]
> [AbstractBTreePartition]
> [AbstractXdbmPartition]
> AvlPartition
> [AbstractLdifPartition]
> LdifPartition
> ReadOnlyConfigurationPartition>
> SingleFileLdifPartition>
> JdbmPartition
> DefaultNexusPartition
> SchemaPartition
>
> (PartitionNexus)
> DefaultNexusPartition
>
> As we keep the Store interface, we will make the
> AbstractBTreePartitionimplement it :
>
> (Store)
> [AbstractBTreePartition]
>
> The SchemaPartition and the DefaultNexusPartition are two different beasts :
> - the SchemaPartition have a wrapped partition, which can be either a Ldif
> partition, or a Jdbm partition, or whatever partition we want.
> - the DefaultNexusPartition is just a proxy which reroute all the requests
> to the associated partition, depending on the operation DN.
>
> Right now, this is what I'm having, and after a lot of cleanup, I have
> something which is quite consistent. It allows us to easily add a new
> Partition implementation §for instance, the HBase partition)
>
> Now, there are some slight issues that needed to be addressed :
> - first, all those Partitions now take a SchemaManager as a constructor
> parameter. This is not a problem, as the SchemaManager instance is always
> created *before* we create any partition.
> - second, partitions initialization was a nightmare. We had a
> Store.init(SchemaManager) method, a final AbstractPartition.initialize()
> method and an abstract AbstractPartition.doInit() method. I removed the
> Store.init(SchemaManager), and refactored the doInti() methods in ordr to
> have a proper initialization sequence (it's not perfect though).
>
> I do think we should merge the initialization() and doInit() method at some
> point, I don't exactly know what is the best options atm.
>
> What remains to be done is to cleanup the DirectoryService initialization
> and the DirectoryServiceFactory initialization. Basically, partitions
> initialization is depending on a complex process, requiring many steps to be
> done :
> - setting the Id
> - setting the SuffixDn
> - creating the underlying storage if needed
> - associating an optimizer, a cursorBuilder, an evaluatorBuilder and a
> SearchEngine to the partition (for BTreePartitions)
> - defining and initializing the indexes,creating the master table (for the
> BTreePartitions)
> - creating the contextEntry (for the LdifPartitions)
>
> Most of those steps are done in the doInit() methods, but it requires the
> presence of configuration elements (Id, Indexes, SuffixDn, etc) which *have*
> to be injected *before* we initialize the partition.
>
> The way we initialize the DirectoryService requires a lot of cleanup too.
> For instance, the DefaultNexusPartition initialize the System partition, and
> all the other partitions, which does not make a lot of sense to me.
> Partitions should be associated with the DirectoryService, not with the
> DefaultNexusPartition. For the same reason, the optimizer and searchEngine
> should not be associated with a specific partition, as it makes thing way
> more complex when we try to do search across partitions.
>
> One more thing would be to use a unique identifier across all the
> partitions. Currently, we use a Long, starting at 1L for each partition's
> contextEntry. Alex suggested that we use the entryUUID instead, something we
> can do easily, as the classes are all generic (it uses ID).
>
> That's pretty much it. I'm working on my computer, because I was offline
> last week-end, but I will create a branch and commit all my changes into it
> so that you can land an eye.
>
> Feel free to provide any feedback !
>
>
>
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>