[ApacheDS] [BigBang] Status Report

[Alex Karasulu <ak...@apache.org>]

Hi all,


Overview
======

In the bigbang branches (BBB) [0] we're conducting a massive code
refactoring of ApacheDS 1.5 in preparation
for the 2.0 road map [1].  There are three major phases to the big bang
effort:

Phase I:
   Remove configuration bean hierarchy to directly configure service objects
with dependency injection.  At
   the same time we decouple configuration via JNDI but not the use of JNDI
for in-vm based ldap operations
   against the directory core.

Phase II:
   Decouple JNDI completely from the core and from protocol providers for
solid state operations: i.e. add, del
   modify, search etc.  JNDI classes and interfaces will no longer be used
within the server.  Consolidate all
   JNDI initial context factories into a single factory implementation and
move this wrapper provider outside of
   the core into it's own module.

Phase III:
   Cleanup after refactoring and implement additional high impact changes
such as:
       - virtual attributes
       - instrumentation
       - a better interceptor mechanism.
       - finalize facades for schema and other subsystems
       - code security to sandbox stored procedures

After this last phase the BBBs will replace their respective trunks for
shared, daemon, apacheds,
and installers.


Status
=====

I just finished a major part of phase I in the BBBs.  Specifically I have
removed the top level object for the
configuration as well as everything in the configuration bean hierarchy
except the protocol providers (PP).
Those still remain but they should be trivial to remove and configure the
PP's themselves.  After this change
I had to deal with several broken tests which I fixed just tonight.  They
still use JNDI for solid state operations
which is fine for now.  David if you're ready you can now start working
those protocol provider configuration
beans.  I'll start on the LDAP one for now which will perhaps take the
longest time to cleanup.

There are still some residual things remaining however I'll clean these
scars up with some more refactoring
as we work on the protocol providers.  Just to see the impact of these
changes up until this point you might
want to look at some test cases and the server.xml file.  It's really
looking good and a slew (not even a 3rd)
of the JNDI garbage has been removed.  There's much more clarity now and I'm
certain this is the right path
to take.  Here are links to those files:

http://svn.apache.org/repos/asf/directory/apacheds/branches/bigbang/server-xml/src/main/resources/server.xml

As you can see the server.xml has become much more concise with greater
clarity.  This is thanks to David's
recommendation to use XBean.  Furthermore notice how the components
themselves are being assembled
instead of the configuration classes that were used before.

One important thing to note is that there is no longer an environment
section used before for JNDI environment
properties.  This used to store among other things the username/password of
the administrative user to start
up the server.  This will no longer be needed.  All embedded code is
trusted, including the protocol providers,
until we build a sandbox for code added to the server as stored procedures
to run with the permissions of the
owner in phase III.

Finally here are some test cases whose setUp() methods show how to construct
and start the core and the
server with all protocols respectively:

http://iedia.notlong.com

Notice that JNDI and the configuration beans are not used any longer to
start up the core.

http://agace.notlong.com

Again notice the same thing with the ApacheDS object. Also note that by
default everything is configured to
work properly via smart defaults however note that LDAP is the only protocol
active by default.  All others
including LDAPS needs to be configured.


Plan for the Near Future
=================

The next step I guess is still up to David and I.  We'll be removing all the
protocol configuration beans to
directly configure these beans by themselves.  Perhaps other components like
MINA thread pools etc
will be configured and injected as well.  There's a lot of cleanup that will
still take place once these objects
have been removed.

After that we can begin working on phase II.  Some or all of the work for
this phase can happen in parallel.
Emmanuel and I discussed this to some degree. Also Ersin those changes to
properly enable virtual/computed
attributes can now start as well as some sandboxing infrastructure for
stored procedures.


Thanks,
Alex


-------------------------------
[0] -
http://svn.apache.org/repos/asf/directory/apacheds/branches/bigbang-with-dependencies
[1] - http://cwiki.apache.org/confluence/display/DIRxPMGT/2.0+Roadmap

Re: [ApacheDS] [BigBang] Status Report

[Alex Karasulu <ak...@apache.org>]

Hi Emmanuel,

On 10/8/07, Emmanuel Lecharny <el...@gmail.com> wrote:
>
> Thanks a lot Alex for this repport ! It gives us a lot of visibility,
> and will help to get in synch when you will be done with all those
> configuration cleanup.


NP this is a must as the team grows.  We're not in Kansas anymore toto!

In the same time, - beside breaking the trunk with stupid commits ;)
> -, I'm finishing to define a new version of JNDI classes used
> internally to the server, like :
> - ServerEntry, a replacement for Attributes
> - ServerAttribute, a replacement for Attribute
> - Value, a handler for attributes' values (JNDI uses Object)
> - ServerModification, a replacement for ModificationItem
>
> This is also done, tests included. Some more classes and interfaces
> have to be replaced, and I have listed all those classes and
> interfaces here :
> http://cwiki.apache.org/confluence/display/DIRxPMGT/JNDI+removing
>
> The biggest part will be to replace the NamingEnumeration which is
> used everywhere, and the SearchResult. The other classes are used from
> place to place, and can be removed quite easily.


Yes we need a ServerResult object as well as a Cursor interface which should
be more like a jdbc ResultSet.  I have some ideas on this from past
experiences.
Just quickly the Cursor should not be pre-fetching in the constructor as we
do
now with NamingEnumerations.  This is why JDBC has getObject() and next()
but next returns a boolean.  It does not prefetch but drives the initial
fetch in the
loop.  Anyways we can discuss this in more detail.  However this work can
start
happening soon in the JDBM implementation as long as we have tools to
butress
the changes by converting back to what higher levels of the server expect in
terms
of JNDI objects like NamingEnumeration and SearchResult.

Another task I'm working on, and I saw that Stefan Szörner has also
> working on it,  is to speed up integration tests. It some point, tests
> are getting soooo long that we are reluctant to launch them before
> committing (I did it a couple of time, and it was a disaster, but
> having to wait 20 minutes each time is just overkilling). I really
> think we can have tests running in less than 5 minutes, and I will
> work on that this week, waiting for Alex to finish his cleanup.


Actually this can start happening now.  You have most of what you need to do
this.  We might
step on each others toes on occasion but these tests are killing us.  Having
them out of the
way ASAP before Phase II might help accelerate our big bang effort.

Thanks,
Alex

Re: [ApacheDS] [BigBang] Status Report

[Emmanuel Lecharny <el...@gmail.com>]

Thanks a lot Alex for this repport ! It gives us a lot of visibility,
and will help to get in synch when you will be done with all those
configuration cleanup.

In the same time, - beside breaking the trunk with stupid commits ;)
-, I'm finishing to define a new version of JNDI classes used
internally to the server, like :
- ServerEntry, a replacement for Attributes
- ServerAttribute, a replacement for Attribute
- Value, a handler for attributes' values (JNDI uses Object)
- ServerModification, a replacement for ModificationItem

This is also done, tests included. Some more classes and interfaces
have to be replaced, and I have listed all those classes and
interfaces here :
http://cwiki.apache.org/confluence/display/DIRxPMGT/JNDI+removing

The biggest part will be to replace the NamingEnumeration which is
used everywhere, and the SearchResult. The other classes are used from
place to place, and can be removed quite easily.

Another task I'm working on, and I saw that Stefan Szörner has also
working on it,  is to speed up integration tests. It some point, tests
are getting soooo long that we are reluctant to launch them before
committing (I did it a couple of time, and it was a disaster, but
having to wait 20 minutes each time is just overkilling). I really
think we can have tests running in less than 5 minutes, and I will
work on that this week, waiting for Alex to finish his cleanup.

Thanks to all of you, and many thanks to Alex and David for this great cleanup !
-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com