Re: code organization

[Daniel John Debrunner <dj...@debrunners.com>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

shahbaz chaudhary wrote:

> I'm curious to hear some thoughts on code organization from the experts
> here.  While looking at the code, the following issues seemd interesting:
>
> 1. Some of the terms used are not immediately obvious (perhaps just to
me?):
> DRDA (network code?), IAPI (interface api?), (vtit...?), noputresultset
> (NoPut ?)

A glossary sounds like a good idea.

drda is a open standard, Distributed Relational Database Architecture,
the network server and jcc client are based upon that protocol for
communication
iapi is internal api, as opposed to external or public api which is for
application use.
vti is Virtual Table Interface

>
> 2. Is it necessary to keep implementation and interfaces in completely
> seperate packages (impl | iapi)...since implementation packages contain
> abstract classes?  What about simply refactoring interface files and
> adding an 'I' in front to distinguish...many implementation classes
> already have *Impl* in their name.  Basically this will reduce the
> number of directories and packages one has to traverse to find the right
> set of code files.

I'll explain the reasoning behind it. The idea was to enforce a strict
split between the internal api (iapi) and any implementations (impl).
The split at the high level allows easy visual feedback of if an
incorrect dependency exists on an implementation by looking for
org.apache.derby.impl imports. In addition an internal api package or
module may be implemented in multiple ways, some of these might be very
different to each other. Thus putting all the implementations and the
internal api in a single package might be more confusing than keeping
them separate, and potentially increasing the chance of independent
implementations accidentally becoming dependent on each other, or the
implementations bleeding into the api.
The abstract classes in the implementation side tend to be to support
implementation variants within a common base, not as part of any api. An
alternate implementation is not required to use those abstract classes.

Does the current directory structure cause problems with IDEs such as
Eclipse or Netbeans, don't their indexing and search mechanisms mean
that a developer doesn't have to traverse trees?

> 3. If I remember correctly, org.apache.derby.*.sql package has around
> 500 files, ...sql.execute has about 150 files.  Could this be broken out
> more?  For example, couldn't the various CursorResultSet derivitives
> have their own package...so even a quick look at just packages gives an
> overview of the architecture?

One of the issues with breaking out implementation packages is that it
tends to increase the number of public methods, this I think is bad for
a couple of reasons. One that the domain search for use of that method
increases to the complete code tree, the other is general security
concerns, public methods can be called from anywhere, so how does that
affect security.
An example is the impl.jdbc package, sometime before the product was
open-sourced it used to be three packages, jdbc12, jdbc20 and jdbc30
with various public methods that allowed jdbc20 and jdbc30 classes to
access key internal methods in the base classes. Since these public
methods on are classes that are directly handed back to applications (as
java.sql.* objects) there is a chance that some code will use reflection
to find the additional public methods and then call them, possibly
gaining access to objects that should not be accessed. Maybe this isn't
a concern because this is only true in embedded mode and the ability to
install malicious code is blocked elsewhere (or its your own machine!).



Instead of moving code around to an alternate convention (I_/Impl class
name tags iapi/impl packages) that as you say gains no functional value,
I would prefer to see activity around reducing footprint by removing the
iapi/impl split and merging classes in *some* situations. I think we
went a little too far on some of the splits, and so maybe could some of
them be merged back to a single concrete implementation that is the
internal api. It would be worth working out which abstract classes and
interfaces only have a single implementation, and then discuss on the
list if that code is suitable for compacting. An example is the type
system, will there every be a separate type system, most unlikely,
though some isolation is required. The lack of BigDecimal in J2ME and
the changes in BigDecimal in J2SE 5.0 so there needs to be some
separation of api and implementation. But maybe the root of the type
system (interface) DataValueDescriptor should be replaced by the
abstract class that all types are based upon (DataType with renaming as
it's not a type but a value).

Dan.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFCARMUIv0S4qsbfuQRAkrtAKC5Z3QkDuLlFnvRzhMPO1nvlryhhQCg3KmZ
3uwuu1vHtN41SB14fgm163o=
=tPRW
-----END PGP SIGNATURE-----

Re: code organization

[Dibyendu Majumdar <di...@mazumdar.demon.co.uk>]

Daniel John Debrunner wrote:
> Does the current directory structure cause problems with IDEs such as
> Eclipse or Netbeans, don't their indexing and search mechanisms mean
> that a developer doesn't have to traverse trees?

Dan,

IDEs do help, but I found that they cannot always tell you what the 
implementation of an interface is. For example, if I point the cursor at 
a method, it will show the comments associated with that method. But 
this is usually an interface, and if I want to see the implementation, I 
have to do a search to find out.

For this reason, I decided to generate Javadoc documentation so that I 
could use Javadoc's ability to list out implementations, etc.

I wish that the interfaces and implementations were packaged close 
together - like org.package.module for interface, and 
org.package.module.impl for implementations. This would have made it 
easier to find the relevant implementations. The problem with current 
structure is that one has to navigate down two different directories.

> Instead of moving code around to an alternate convention (I_/Impl class
> name tags iapi/impl packages) that as you say gains no functional value,
> I would prefer to see activity around reducing footprint by removing the
> iapi/impl split and merging classes in *some* situations. 

I would argue that we should leave things as they are for the time 
being, as changing things will make it harder for anyone trying to 
understand the code. Stability is important for anyone trying to follow 
the code.

Regards

Dibyendu

Re: code organization

[Dibyendu Majumdar <di...@mazumdar.demon.co.uk>]

RPost wrote:
> I believe that Dibyendu not only plans on including a glossary in the
> documentation work he is doing but that he has already begun work on it.

I do intend to produce a glossary - but if someone else wants to have a 
go ... then please do.

> I would suggest that the fact that the original developers already know the
> current directory/package structure is the strongest argument for leaving
> the structure the way it is.
> 
> IMHO that the knowledge and expertise that these original developers have is
> far too valuable at this stage of the project to risk diluting. As soon as
> any refactoring is done NONE of these experts will be able to provide
> assistance until they spend time getting up to speed on the changes that
> were done. Thus their ability to contribute is immediately diluted. I'd
> rather have the expert resources being used to fix critical problems, work
> on new functionality and explain how (and why) things are being done now (as
> Dan does so well).

That's a good point.

Regards

Re: code organization

[RPost <rp...@pacbell.net>]

> shahbaz chaudhary wrote:
>
>1. Some of the terms used are not immediately obvious (perhaps just to
me?):
>DRDA (network code?), IAPI (interface api?), (vtit...?), noputresultset
(NoPut ?)

I believe that Dibyendu not only plans on including a glossary in the
documentation work he is doing but that he has already begun work on it.

>2. Is it necessary to keep implementation and interfaces in completely
seperate packages (impl | iapi)...since >implementation packages contain
abstract classes?  What about simply refactoring interface files and adding
an 'I' in >front to distinguish...many implementation classes already have
*Impl* in their name.  Basically this will reduce the >number of directories
and packages one has to traverse to find the right set of code files.

I certainly share your pain in trying to find the right set of code files to
follow what is happening. As Dan suggests, you really need to use an IDE to
assist in this process. Following the raw code manually can only convey, at
best, what is possible. You need to follow the implementation to ascertain
what is actually being done. Obviously, this is because of the way that
implementation classes, abstract classes and interfaces all meld together.
Interfaces don't include variables and abstract methods don't show the
actual code.

I use JBuilder to do the implementation class/package traversal to
understand a specific code hierarchy and then use raw code inspection to get
the bigger architectural picture.

>. . . refactoring the code won't make any functional improvements but it
will be beneficial from a usability >perspective...since large number of
'users' of this code will actually be developers looking to get familiar
with the >code and contribute.

I would suggest that the fact that the original developers already know the
current directory/package structure is the strongest argument for leaving
the structure the way it is.

IMHO that the knowledge and expertise that these original developers have is
far too valuable at this stage of the project to risk diluting. As soon as
any refactoring is done NONE of these experts will be able to provide
assistance until they spend time getting up to speed on the changes that
were done. Thus their ability to contribute is immediately diluted. I'd
rather have the expert resources being used to fix critical problems, work
on new functionality and explain how (and why) things are being done now (as
Dan does so well).

Dibyendu has begun an ambitious project to document this expert knowledge
and his efforts will go a long way towards providing the type of information
you are looking for. So far he is getting excellent and, I dare say,
enthusiastic support from the developers.