[apacheds] Idea on refactoring Database, ContextPartition, and RootNexus

[Trustin Lee <tr...@gmail.com>]

Hi folks,

I thought over simplifying Database and ContextPartition, and here's my idea:

1) Do we need that much modification operations (add, delete, move
...) in Database?

No we don't need that much, all we need is:

* Put (used to add and replace entries)
* Remove (used to delete entries)

We could be able to combine put and remove operation to move or rename
entries, and therefore rename and move operation is splitted into many
small operations, so we need transaction here:

Database db = ...;
Transaction tx = db.beginTransaction();
tx.delete( entryWithOldName );
tx.put( entryWithNewName );
tx.commit();

2) Let's remove ContextPartition

If once the behavior of Database is defined in detail, we will be able
to create concrete implementation of
frontend to Database, and users won't need to reimplement it and just
use it.  So we can hide it easily:

ContextPartitionConfiguration cfg = ...;
cfg.setDatabase( new InMemoryDatabase() );

3) DatabaseNexus and RootNexus

The role of DatabaseNexus is similar to that of RootNexus.  We could
forward any operations to appropriate Database implementation.  Using
DatabaseNexus, we will be able to make RootNexus just a frontend to
DatabaseNexus that translates complex JNDI operations into small
Database operations.

Plus we'll need to expose DatabaseNexus to Interceptors for
interceptors that need a fine-grained control over database.

I attached class diagram and sequence diagram for this change to help
you understand this easily.

Any feedbacks are welcome!

Trustin
-- 
what we call human nature is actually human habit
--
http://gleamynode.net/

Re: [apacheds] Idea on refactoring Database, ContextPartition, and RootNexus

[Trustin Lee <tr...@gmail.com>]

2005/6/19, Alex Karasulu <ao...@bellsouth.net>:
> Niclas Hedhman wrote:
> >I am not sure that I can the total impact of what you are suggesting, but I
> >would like to hear your thoughts on the following scenario;
> Great points you make below Niclas.

+1

> >I can create a Nexus that exposes an entire application as a directory tree
> >through JavaBean patterns and reflection. I can also create a wrapper on
> >legacy system and expose it as a nexus. IMHO, this has tremendous advantages
> >for system integration work, especially between systems that are
> >traditionally 'far apart'.
>
> Basically an LDAP proxy would have a hard time with this since the
> operations are split apart.  Users would have to implement a Database to
> have it talk to an LDAP server so ... they would have to collect the
> tiny operations with put and gets to interpret what they are on a
> commit.  This is impossible without transactions since the operations
> don't have a distinct cut off.

I see.. so we need to retain ContextPartition interface.

Then what about creating DatabaseContextPartition class and make it
translate operations into database operations (put and remove?)  This
will both simplify database interface and help users to implement
partitions like LDAP proxy.

> >I truly liked the concept of a routing RootNexus, and Alex convinced me of the
> >value for setups explained above.
> Aye me too. But I don't want to dispel this idea it has some serious
> potential.  Let's explore it thoroughly as you suggested.

Me, too. :)

Trustin
-- 
what we call human nature is actually human habit
--
http://gleamynode.net/

Re: [apacheds] Idea on refactoring Database, ContextPartition, and RootNexus

[Alex Karasulu <ao...@bellsouth.net>]

Niclas Hedhman wrote:

>On Sunday 19 June 2005 12:15, Trustin Lee wrote:
>  
>
>>3) DatabaseNexus and RootNexus
>>
>>The role of DatabaseNexus is similar to that of RootNexus.  We could
>>forward any operations to appropriate Database implementation.  Using
>>DatabaseNexus, we will be able to make RootNexus just a frontend to
>>DatabaseNexus that translates complex JNDI operations into small
>>Database operations.
>>    
>>
>
>I am not sure that I can the total impact of what you are suggesting, but I 
>would like to hear your thoughts on the following scenario;
>  
>
Great points you make below Niclas.

>I can create a Nexus that exposes an entire application as a directory tree 
>through JavaBean patterns and reflection. I can also create a wrapper on  
>legacy system and expose it as a nexus. IMHO, this has tremendous advantages 
>for system integration work, especially between systems that are 
>traditionally 'far apart'.
>  
>
Basically an LDAP proxy would have a hard time with this since the 
operations are split apart.  Users would have to implement a Database to 
have it talk to an LDAP server so ... they would have to collect the 
tiny operations with put and gets to interpret what they are on a 
commit.  This is impossible without transactions since the operations 
don't have a distinct cut off.

>I truly liked the concept of a routing RootNexus, and Alex convinced me of the 
>value for setups explained above.
>  
>
Aye me too. But I don't want to dispel this idea it has some serious 
potential.  Let's explore it thoroughly as you suggested.

>IMHO, it doesn't look as easy as before to do the above with your proposal.
>  
>
Perhaps but you don't know until someone whips it together in a branch 
and we can all look at it and experiment with it's potential or its 
drawbacks.

Thanks Nic,
Alex

Re: [apacheds] Idea on refactoring Database, ContextPartition, and RootNexus

[Niclas Hedhman <ni...@hedhman.org>]

On Sunday 19 June 2005 12:15, Trustin Lee wrote:
> 3) DatabaseNexus and RootNexus
>
> The role of DatabaseNexus is similar to that of RootNexus.  We could
> forward any operations to appropriate Database implementation.  Using
> DatabaseNexus, we will be able to make RootNexus just a frontend to
> DatabaseNexus that translates complex JNDI operations into small
> Database operations.

I am not sure that I can the total impact of what you are suggesting, but I 
would like to hear your thoughts on the following scenario;

I can create a Nexus that exposes an entire application as a directory tree 
through JavaBean patterns and reflection. I can also create a wrapper on  
legacy system and expose it as a nexus. IMHO, this has tremendous advantages 
for system integration work, especially between systems that are 
traditionally 'far apart'.

I truly liked the concept of a routing RootNexus, and Alex convinced me of the 
value for setups explained above.
IMHO, it doesn't look as easy as before to do the above with your proposal.


Cheers
Niclas

Re: [apacheds] Idea on refactoring Database, ContextPartition, and RootNexus

[Trustin Lee <tr...@gmail.com>]

2005/6/19, Alex Karasulu <ao...@bellsouth.net>:
> >* Put (used to add and replace entries)
> >* Remove (used to delete entries)
> Right but we may need to carry some information specific to the type of
> operation being performed.  A few people commented on needing this
> feature but I guess we can treat it separately since its not there now
> :-).

What kine of extra information?

> >2) Let's remove ContextPartition
> You're removing context partition totally?  The following JIRA issue
> here re: nested nexus' will be done using DatabaseNexus objects instead?
> 
> http://issues.apache.org/jira/browse/DIREVE-23

Yes, DatabaseNexus will have mount and unmount (or
register/deregister) methods so that we can mount Database
implementations to specific DN.

> Also these changes will not break alias dereferencing I hope as its
> documented here:
> http://ldapd.sourceforge.net/dist/Aliases%20and%20Search.pdf

Is this rule programmed somewhere in ApacheDS?  Which class?

> I like what you've done here but how about calling Database a context
> partition.  In LDAP a context maps to an entry in the DIT and your
> Database will do that too.  Leave the RootNexus as you planned below
> with its new functionality.  Create a new ContextPartitionNexus and make
> it concrete to replace the DatabaseNexus.  So basically we are creating
> another layer of indirection as you need just keeping existing terms.

I see.  Then we're removing Database interface and simplifying
ContextPartition. :)

> >If once the behavior of Database is defined in detail, we will be able
> >to create concrete implementation of
> >frontend to Database, and users won't need to reimplement it and just
> >use it.  So we can hide it easily:
> This front end is what the ContextPartitionConfiguration?  Or is it what
> the ContextPartition used to be?  Getting a little confused here.

The 'comcrete implementation' here is RootNexus that translates
complex operations into smaller pieces.  But I'm not sure its the duty
of RootNexus.  We could do that somewhere else like
Invocation.execute().

> >ContextPartitionConfiguration cfg = ...;
> >cfg.setDatabase( new InMemoryDatabase() );
> But people will implement their own databases and there will be more
> than one.  Above you say the that a concrete implementation of the
> Database will not have to be reimplemented.  Users want to do this.

Database should be reimplemented by users of course if they are not
using JDBM backend for example.  The concrete implementation is
frontend part that translates JNDI operations into Database (or
ContextPartition if we use the new term) operations.

> >The role of DatabaseNexus is similar to that of RootNexus.  We could
> >forward any operations to appropriate Database implementation.  Using
> >DatabaseNexus, we will be able to make RootNexus just a frontend to
> >DatabaseNexus that translates complex JNDI operations into small
> >Database operations.
> Would the RootNexus still serve out the RootDSE? Like the translation
> aspect btw and it makes sense to have this be a ContextPartitionNexus.
> Then we can implement the functionality of the JIRA issue I listed above.

Right, it will have to move to ContextPartitionNexus.

> #3 is a great idea if the names are adjusted as I noted above.  This way
> the terminology people are used to associating with pluggable stores
> remains the same as a ContextPartition .. it just has interface changes
> and sits one level down after introducing another ContextPartitionNexus.



> >Plus we'll need to expose DatabaseNexus to Interceptors for
> >interceptors that need a fine-grained control over database.
> Yep I understand this and ContextPartitionConfiguration is the best way
> to give them what they need or am I mistaken?

What about just adding RootNexus.getContextPartitions() method that
returns a ContextPartitionNexus as ContextPartition type?

Trustin
-- 
what we call human nature is actually human habit
--
http://gleamynode.net/

Re: [apacheds] Idea on refactoring Database, ContextPartition, and RootNexus

[Alex Karasulu <ao...@bellsouth.net>]

Sorry took me a while since I had some investigation to do before 
responding.

Trustin Lee wrote:

>No we don't need that much, all we need is:
>
>* Put (used to add and replace entries)
>* Remove (used to delete entries)
>  
>
Right but we may need to carry some information specific to the type of 
operation being performed.  A few people commented on needing this 
feature but I guess we can treat it separately since its not there now 
:-). 

>We could be able to combine put and remove operation to move or rename
>entries, and therefore rename and move operation is splitted into many
>small operations, so we need transaction here:
>
>Database db = ...;
>Transaction tx = db.beginTransaction();
>tx.delete( entryWithOldName );
>tx.put( entryWithNewName );
>tx.commit();
>  
>
Yep I'm following you.

>2) Let's remove ContextPartition
>
>  
>
You're removing context partition totally?  The following JIRA issue 
here re: nested nexus' will be done using DatabaseNexus objects instead?

http://issues.apache.org/jira/browse/DIREVE-23

Also these changes will not break alias dereferencing I hope as its 
documented here:

http://ldapd.sourceforge.net/dist/Aliases%20and%20Search.pdf

(BTW I need to move this into our repo)

Also I have some terminology issues as you might expect.  First off I 
would like the data containing areas to be referred to as partitions.  
Reason being is they are subsets of the information served by the 
server.   The entire subsystem is the backend subsystem.  I don't want 
that mixed up with backend or database. 

I like what you've done here but how about calling Database a context 
partition.  In LDAP a context maps to an entry in the DIT and your 
Database will do that too.  Leave the RootNexus as you planned below 
with its new functionality.  Create a new ContextPartitionNexus and make 
it concrete to replace the DatabaseNexus.  So basically we are creating 
another layer of indirection as you need just keeping existing terms. 

>If once the behavior of Database is defined in detail, we will be able
>to create concrete implementation of
>frontend to Database, and users won't need to reimplement it and just
>use it.  So we can hide it easily:
>  
>
This front end is what the ContextPartitionConfiguration?  Or is it what 
the ContextPartition used to be?  Getting a little confused here.

>ContextPartitionConfiguration cfg = ...;
>cfg.setDatabase( new InMemoryDatabase() );
>  
>
But people will implement their own databases and there will be more 
than one.  Above you say the that a concrete implementation of the 
Database will not have to be reimplemented.  Users want to do this.

>3) DatabaseNexus and RootNexus
>
>The role of DatabaseNexus is similar to that of RootNexus.  We could
>forward any operations to appropriate Database implementation.  Using
>DatabaseNexus, we will be able to make RootNexus just a frontend to
>DatabaseNexus that translates complex JNDI operations into small
>Database operations.
>  
>
Would the RootNexus still serve out the RootDSE? Like the translation 
aspect btw and it makes sense to have this be a ContextPartitionNexus.  
Then we can implement the functionality of the JIRA issue I listed above.

#3 is a great idea if the names are adjusted as I noted above.  This way 
the terminology people are used to associating with pluggable stores 
remains the same as a ContextPartition .. it just has interface changes 
and sits one level down after introducing another ContextPartitionNexus.

>Plus we'll need to expose DatabaseNexus to Interceptors for
>interceptors that need a fine-grained control over database.
>  
>
Yep I understand this and ContextPartitionConfiguration is the best way 
to give them what they need or am I mistaken?

>I attached class diagram and sequence diagram for this change to help
>you understand this easily.
>  
>
Thanks for going out of your way to really make it easy to understand.  
Forgive me for having so many questions.

Alex

Re: [apacheds] Idea on refactoring Database, ContextPartition, and RootNexus

[Trustin Lee <tr...@gmail.com>]

Hi Emmanuel,

2005/6/19, Emmanuel Lecharny <el...@apache.org>:
> Not so sure. You have two basic choices here :
> - use a specific indexed Database (BDB, or whatever fit your needs)
> - use a classic relationnal Database (Or*cle(TM), Posgresql,
> MySql, ...)
> 
> I bet that if you are to use ApacheDS in production, you will favor
> reliability above performance. This will be the choice of many clients,
> I assure you ! Not that performance is not important, but clients hate
> to loose data, strange enough ;)
> 
> Or*cle(TM) has implemented a new hierarchical engine in V10, which
> improve greatly performances for data which has this kind of structure.
> 
> Whatever, I really buy your idea to have the simplest interface as
> possible, because it will help to implements any kind of backend
> Databse.

Thanks for your information, and now I moved most BTree-specific
classes to btree package, so it will be more clear to implement those
context partitions.

> There is also a point that I want to rise : a Modify request does not
> give you the attributes of an entry, as you perfectly know. So a Put
> will be quite complicated to implement, as you won't be able to deal
> with deleted attributes without fetching the full entry before. So the
> sequence will be :
> - fetch the entry from the Database
> - modify the entry in memory
> - update the entry into the database
> 
> It will be very costly.
> 
> I also bet that we won't have ten thousands of Database to deal with, so
> even if the proposed interface is a little bit more complicated than
> just a Put and a Delete, it's not a big deal.

Right, so I retained current API.

> So your NotDeletingDatabase is still seen as a Database, am I wrong? If
> so, this is quite a smart idea. The end user should not know what's
> going on in the kitchen, he just want to have good food in his plate !

Yes exactly, but I found it is more powerful to make interceptor look
more like ContextPartitions to reduce impedence mismatch between
Interceptor and ContextPartition.  I'll show how can I make this
happen today or tomorrow.

> Much more important : just add transaction because it's a must. This is
> something I always had problem with OpenLdap, because updates are
> dangerous on a running environment. If you change an information, you
> can't guaranty that somebody is already using it. The problem is that
> you have to extend the transaction mechanism to reads, because you want
> a reader to have a fresh entry, not something that is currently being
> modified. (Updating en entry can be something costly, if you store
> images, so this is a scenario which could occur)

Right, I'm trying to find out the best way to add transaction support now.

> > 5) add AbstractDatabase class that helps users to implement databases
> > easily.  (e.g. two modify operations are delegated to one modify
> > method after some normalization)
> 
> +1, but can you develop a little bit what you mean by 'two modify
> operations are delegated to one modify method after some
> normalization' ?

I thought wrong by mistake just ignore it please. :)

> > 6) provide standardized initialization method like 'open' instead of
> > constructors like we did for ContextPartition so that users can
> > instantiate Database and DatabaseContextPartition in configuration
> > phase.
> +1

I've done this in the branch. :)

> really cool, and we need it actually, we will have 30° today here in
> Paris ;)

It's also hot here in South Korea.  I'd like drink a cup of Chi Chi cocktail! :)

Trustin
-- 
what we call human nature is actually human habit
--
http://gleamynode.net/

Re: [apacheds] Idea on refactoring Database, ContextPartition, and RootNexus

[Emmanuel Lecharny <el...@apache.org>]

Hi Trustin,

> If you're going to change an entry, put operation will be sufficient. 
> But if underlying database implementaion stores attributes sparsely,
> it can be insuffucient of course.  I guess that kind of implementation
> will be rare and can be improved by using cache IMHO, so the problem
> here is possibly moving or renaming.

Not so sure. You have two basic choices here :
 - use a specific indexed Database (BDB, or whatever fit your needs)
 - use a classic relationnal Database (Or*cle(TM), Posgresql,
MySql, ...)

I bet that if you are to use ApacheDS in production, you will favor
reliability above performance. This will be the choice of many clients,
I assure you ! Not that performance is not important, but clients hate
to loose data, strange enough ;)

Or*cle(TM) has implemented a new hierarchical engine in V10, which
improve greatly performances for data which has this kind of structure. 

Whatever, I really buy your idea to have the simplest interface as
possible, because it will help to implements any kind of backend
Databse. 

There is also a point that I want to rise : a Modify request does not
give you the attributes of an entry, as you perfectly know. So a Put
will be quite complicated to implement, as you won't be able to deal
with deleted attributes without fetching the full entry before. So the
sequence will be :
 - fetch the entry from the Database
 - modify the entry in memory
 - update the entry into the database

It will be very costly. 

I also bet that we won't have ten thousands of Database to deal with, so
even if the proposed interface is a little bit more complicated than
just a Put and a Delete, it's not a big deal.

> 
> The reason why I proposed these simple operations were actually to
> create an interceptor that supresses delete operation and marks the
> target entry as 'deleted' instead of deleting it really.  

I guess that you'll use it for Mitosis?

> I was able
> to simplify the implementation of that filter using this ways.  But
> now I found that we can implement NotDeletingDatabase which wraps
> existing Database to intercept delete, move or modifyRN operations and
> fulfill my aim, and of course, retaining current Database interface.

So your NotDeletingDatabase is still seen as a Database, am I wrong? If
so, this is quite a smart idea. The end user should not know what's
going on in the kitchen, he just want to have good food in his plate !


> So.. here's my more refined idea:
> 
> 1) create a new package 'partition' and place all partition related
> stuffs there for better package layout.
> 2) merge AbstractContextPartition and ApplicationContextPartition to
> DatabaseContextPartition.
> 3) move db package into 'partition' package. (i.e. partition.database)
> and put DatabaseContextPartition to 'database' package for better
> package layout.
> 4) retain the interface of current Database class, and add transaction
> support to make all operations atomic.

Much more important : just add transaction because it's a must. This is
something I always had problem with OpenLdap, because updates are
dangerous on a running environment. If you change an information, you
can't guaranty that somebody is already using it. The problem is that
you have to extend the transaction mechanism to reads, because you want
a reader to have a fresh entry, not something that is currently being
modified. (Updating en entry can be something costly, if you store
images, so this is a scenario which could occur)

> 5) add AbstractDatabase class that helps users to implement databases
> easily.  (e.g. two modify operations are delegated to one modify
> method after some normalization)

+1, but can you develop a little bit what you mean by 'two modify
operations are delegated to one modify method after some
normalization' ?

> 6) provide standardized initialization method like 'open' instead of
> constructors like we did for ContextPartition so that users can
> instantiate Database and DatabaseContextPartition in configuration
> phase.
+1

> 
> I must admit that my first idea was a kind of expression of
> stupidness.  

-1. You are not stupid at all !

> > Put aside the previous comment, I really like the Tx stuff. This is
> > something that many LDAP implementations don't have.
> 
> Yeah we'll be able to put quite useful metadata to Database. Cool? :)

really cool, and we need it actually, we will have 30° today here in
Paris ;)

Emmanuel Lécharny

Re: [apacheds] Idea on refactoring Database, ContextPartition, and RootNexus

[Trustin Lee <tr...@gmail.com>]

Hi Emmanuel,

2005/6/19, Emmanuel Lecharny <el...@apache.org>:
> > * Put (used to add and replace entries)
> > * Remove (used to delete entries)
> 
> I suggest that you keep an operation to change entries. It may be much
> faster to do a Change than to do a Remove/Put. Even a Put instead of a
> Change can be costly, as all the elements of an entry are to be updated,
> which could cost a lot. Obviously, it depends on the underlying
> database !

If you're going to change an entry, put operation will be sufficient. 
But if underlying database implementaion stores attributes sparsely,
it can be insuffucient of course.  I guess that kind of implementation
will be rare and can be improved by using cache IMHO, so the problem
here is possibly moving or renaming.

The reason why I proposed these simple operations were actually to
create an interceptor that supresses delete operation and marks the
target entry as 'deleted' instead of deleting it really.  I was able
to simplify the implementation of that filter using this ways.  But
now I found that we can implement NotDeletingDatabase which wraps
existing Database to intercept delete, move or modifyRN operations and
fulfill my aim, and of course, retaining current Database interface.

But my idea on DatabaseContextPartition is still valid.  Users who
want to implement partition like LDAP proxy will have to implement
ContextPartition directly, but users who wants to use ApacheDS's
Database support will have to extend DatabaseContextPartition.   For
now, AbstractContextPartition and ApplicationContextPartition seems to
do what DatabaseContextPartition should do.

So.. here's my more refined idea:

1) create a new package 'partition' and place all partition related
stuffs there for better package layout.
2) merge AbstractContextPartition and ApplicationContextPartition to
DatabaseContextPartition.
3) move db package into 'partition' package. (i.e. partition.database)
and put DatabaseContextPartition to 'database' package for better
package layout.
4) retain the interface of current Database class, and add transaction
support to make all operations atomic.
5) add AbstractDatabase class that helps users to implement databases
easily.  (e.g. two modify operations are delegated to one modify
method after some normalization)
6) provide standardized initialization method like 'open' instead of
constructors like we did for ContextPartition so that users can
instantiate Database and DatabaseContextPartition in configuration
phase.

I must admit that my first idea was a kind of expression of
stupidness.  But this discussion led me to find out more better
solutions.  I think I'll satisfy with these 6 improbements.

> Put aside the previous comment, I really like the Tx stuff. This is
> something that many LDAP implementations don't have.

Yeah we'll be able to put quite useful metadata to Database. Cool? :)

Thanks for everyone's feedback!

Trustin
-- 
what we call human nature is actually human habit
--
http://gleamynode.net/

Re: [apacheds] Idea on refactoring Database, ContextPartition, and RootNexus

[Emmanuel Lecharny <el...@apache.org>]

Hi Trustin,

> I thought over simplifying Database and ContextPartition, and here's my idea:
> 
> 1) Do we need that much modification operations (add, delete, move
> ...) in Database?
> 
> No we don't need that much, all we need is:
> 
> * Put (used to add and replace entries)
> * Remove (used to delete entries)

I suggest that you keep an operation to change entries. It may be much
faster to do a Change than to do a Remove/Put. Even a Put instead of a
Change can be costly, as all the elements of an entry are to be updated,
which could cost a lot. Obviously, it depends on the underlying
database !


> We could be able to combine put and remove operation to move or rename
> entries, and therefore rename and move operation is splitted into many
> small operations, so we need transaction here:
> 
> Database db = ...;
> Transaction tx = db.beginTransaction();
> tx.delete( entryWithOldName );
> tx.put( entryWithNewName );
> tx.commit();

Put aside the previous comment, I really like the Tx stuff. This is
something that many LDAP implementations don't have.

Emmanuel Lécharny