You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Jeremy Boynes <jb...@apache.org> on 2005/09/09 21:14:45 UTC
Modular build, was: VOTE: Approach for sharing code
David W. Van Couvering wrote:
> I thought Kathey/Dan's idea of generating copies of the common code
> into two separate directories was interesting and solved a lot of problems,
> and I thought it would be worthwhile to walk through this in a bit more
> detail.
>
This seems like a lot of build time complexity for very little benefit.
We are doing all this pre-processing (in source or, please no, in
binary) just to avoid the need for users to include two jars on the
classpath instead of one.
This is also reinforcing Derby's not-invented-here attitude that
requires implementation of everything rather than using libraries freely
available from other open source projects.
I'd like to crack this discussion wide open and say that we should move
away from the one-jar-to-rule-them model and switch to a model where the
different components inside Derby can be built and versioned
independently and where we are comfortable depending on external
libraries to provide core functionality.
In that context, components that come to mind are engine, client, net,
tools and common and external dependencies for consideration include
logging, configuration and thread management.
I would like to discuss moving to a modular structure where these are
built separately and can be released separately. That gives us the
benfits of simplicity, shared code, and reduced maintenance without the
wierdness of code templates and preprocessing.
For users that want one single jar, we can merge them togther either by
combining jars or through the use of Class-Path references in the
manifests. But that should be a special distribution rather than the norm.
--
Jeremy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "M. Mortazavi" <Ma...@Sun.COM>.
Daniel John Debrunner wrote:
>Jeremy Boynes wrote:
>
>
>
>>Think of the carnage if it was java_14208_b13.sql_300.Connection
>>
>>
>
>It's actually instructive to look how Java solves this:
>
>- The interface is kept upwards compatible, by following rules such as
>only new methods/fields etc.
>
>Then to look at how a consumer, Derby, deals with the fact of multiple
>versions of the interface:
>
>- Derby only provides and uses the functionality to match the version of
>the interface (java.sql.Connection) loaded, determined at runtime.
>
>I'd thought this was a workable direction being proposed about six
>thousand message ago, upwards compatible apis and the ability for a
>consumer to handle a lower version. Not sure what derailed it.
>
>Dan.
>
>
I'm not sure if this will help or hinder with your considerations but . . .
Compatible type evolution is described in the Java Serialization Spec.
The reason it is described there is because the problem arises when
Class A is compiled against a particular version of Class B but then at
run-time an instance of A encounters (through class loading upon
de-serialization, for example) an instance of some other version of
Class B.
These type compatibility rules can be found here:
http://java.sun.com/j2se/1.4.2/docs/guide/serialization/spec/version.html#wp9419
M.
--
M. Mortazavi
+1 408 276 7205 (office)
+1 408 421 4093 (mobile)
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Jeremy Boynes <jb...@apache.org>.
David W. Van Couvering wrote:
>
> You say Java does this -- is this documented somewhere? I'd love to have
> that as a reference.
>
Do you mean the JVM impl or the general philosophy? I think the
versioning model is defined in the language spec but would have to go
digging.
> First of all, Kathey was concerned we wouldn't be able to follow this
> kind of discipline.
>
> http://mail-archives.apache.org/mod_mbox/db-derby-dev/200509.mbox/%3c43207ABC.8080000@sbcglobal.net%3e
>
It is on us and our users to police ourselves.
>
> But I'll also show where I saw this get tripped up:
>
> > * a consumer expecting X.Y will work with X.Y' when Y' < Y by version
> > detection and degraded functionality (Y' level). If Y level function
> > is required then it will gracefully die (able to detect rather
> > than AbstractMethodError)
>
>
> In my original proposal:
>
> * compatibility will be strongly encouraged but not guaranteed
> against previous minor versions (e.g. a 10.2 consumer works
> with 10.1 common classes, but a 10.3 consumer has a hard
> dependency on new methods, it can not work with 10.2
> common classes).
>
> Perhaps I remember incorrectly, but I remember us (or enough of us)
> generally agreeing that gracefully dying when Y level function was
> required was not acceptable, as this was a regression of existing
> behavior. This was the "nail in the coffin" for my original proposal.
>
> http://mail-archives.apache.org/mod_mbox/db-derby-dev/200509.mbox/%3c43289F58.6000707@mtrad.com%3e
>
> http://mail-archives.apache.org/mod_mbox/db-derby-dev/200509.mbox/%3c4328AC1F.6000504@sun.com%3e
>
>
> Unless you can address these issues I think we're back at square one...
>
I'd put this as part of the discipline. The Y consumer, as the newer
version, knows about earlier Y' versions and hence can be coded to
accept the downrev dependency - it can downgrade itself, it can offer
different behaviour, it can implement the functionality itself, it can
die gracefully, or we can just make it X+1.
Bear in mind there are no guarantees. For example, someone else could
have modified Y' themselves without any regard to compatibility and just
stuck it on the classpath.
Also, let's not forget that this is only in the case where we have
multiple applications with poor classpath discipline installed
concurrently. (And I haven't forgotten the request for a writeup on
stuff like ClassWorlds).
--
Jeremy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Well, that works for me!
David
Daniel John Debrunner wrote:
> David W. Van Couvering wrote:
>
>
>
>>In my original proposal:
>>
>> * compatibility will be strongly encouraged but not guaranteed
>> against previous minor versions (e.g. a 10.2 consumer works
>> with 10.1 common classes, but a 10.3 consumer has a hard
>> dependency on new methods, it can not work with 10.2
>> common classes).
>>
>>Perhaps I remember incorrectly, but I remember us (or enough of us)
>>generally agreeing that gracefully dying when Y level function was
>>required was not acceptable, as this was a regression of existing
>>behavior. This was the "nail in the coffin" for my original proposal.
>
>
> Maybe I'm an optimist, but I think that a consumer of common code can
> always be coded to keep running (in a reduced mode) when faced with an
> older version of the common code. Thus I think this approach can be made
> to work, just start with the mindset that dying is unacceptable, rather
> than inevitable.
>
> Dan.
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Daniel John Debrunner <dj...@debrunners.com>.
David W. Van Couvering wrote:
> In my original proposal:
>
> * compatibility will be strongly encouraged but not guaranteed
> against previous minor versions (e.g. a 10.2 consumer works
> with 10.1 common classes, but a 10.3 consumer has a hard
> dependency on new methods, it can not work with 10.2
> common classes).
>
> Perhaps I remember incorrectly, but I remember us (or enough of us)
> generally agreeing that gracefully dying when Y level function was
> required was not acceptable, as this was a regression of existing
> behavior. This was the "nail in the coffin" for my original proposal.
Maybe I'm an optimist, but I think that a consumer of common code can
always be coded to keep running (in a reduced mode) when faced with an
older version of the common code. Thus I think this approach can be made
to work, just start with the mindset that dying is unacceptable, rather
than inevitable.
Dan.
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Thanks for writing up a proposed solution, I was getting a little tired
of putting up targets for others to shoot down.
I agree with this wholeheartedly this was what I have wanted to do
(although perhaps not as well communicated and there are some details
here that are new and which seem really good/interesting).
You say Java does this -- is this documented somewhere? I'd love to have
that as a reference.
First of all, Kathey was concerned we wouldn't be able to follow this
kind of discipline.
http://mail-archives.apache.org/mod_mbox/db-derby-dev/200509.mbox/%3c43207ABC.8080000@sbcglobal.net%3e
But I'll also show where I saw this get tripped up:
> * a consumer expecting X.Y will work with X.Y' when Y' < Y by version
> detection and degraded functionality (Y' level). If Y level function
> is required then it will gracefully die (able to detect rather
> than AbstractMethodError)
In my original proposal:
* compatibility will be strongly encouraged but not guaranteed
against previous minor versions (e.g. a 10.2 consumer works
with 10.1 common classes, but a 10.3 consumer has a hard
dependency on new methods, it can not work with 10.2
common classes).
Perhaps I remember incorrectly, but I remember us (or enough of us)
generally agreeing that gracefully dying when Y level function was
required was not acceptable, as this was a regression of existing
behavior. This was the "nail in the coffin" for my original proposal.
http://mail-archives.apache.org/mod_mbox/db-derby-dev/200509.mbox/%3c43289F58.6000707@mtrad.com%3e
http://mail-archives.apache.org/mod_mbox/db-derby-dev/200509.mbox/%3c4328AC1F.6000504@sun.com%3e
Unless you can address these issues I think we're back at square one...
David
Further asides below:
> As a quick braindump:
> * there are rules that restrict changes to the interface going forward
> * Sun (et al) stick to those rules even when it is painful
> * no class/method is removed without being deprecated first
> * the JVM supports version skew a little by not requiring an
> implementation to implement all methods in a interface (at runtime)
That's nice to know, I didn't know that
> * the interfaces often provide a mechanism to determine what features of
> an API the implementation implements
> * frameworks can utilize multiple classloaders to load multiple versions
> of a class into the VM - tools have developed to simplify this
>
I don't know much about such frameworks and tools, but I don't think
this solves Dan's problem of "accidentally" loading multiple versions.
> I think we can support a scenario where:
> * a comsumer expecting version X.Y will work with any implementation
> X.Y' when Y' >= Y
> * X.Y defines the interface version, i.e. X.Y.Z works with X.Y.Z' for
> all Z'
> * X defines the compatibility level i.e. X and X' are not guaranteed
> to work together (although we will try to ensure they do)
>
> Basic rules:
> * at the Z level, no API change is allowed, just implementation changes
> * at the Y level, additive API changes are allowed but must be
> accompanied by support in the versioning mechanism so that they can be
> detected. Things can be deprecated
> * at the Z level, incompatible changes are allowed. Items deprecated in
> version X can be removed in version X+2 (implicitly (X+2).0.0).
>
> This all applies to operation in a single classloader. Where multiple
> classloaders are involved (inside one VM or in multiple VMs) a different
> set of versioning behaviours applies to the wire protocol.
>
> --
> Jeremy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Thanks, Kathey, I missed that, I'll add that to this page.
David
Kathey Marsden wrote:
> David W. Van Couvering wrote:
>
>
>>Hi, all. I haven't gotten any more comments on the proposal at
>>http://wiki.apache.org/db-derby/ModuleVersioningGuidelines, so I'm
>>going to update this page based on the comments I have received so far
>>and then we'll try for another vote.
>>
>
>
> David will there be another document that describes the packaging and
> any external impact of this change or will it be included in this one?
>
> Thanks
>
> Kathey
>
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Kathey Marsden <km...@sbcglobal.net>.
David W. Van Couvering wrote:
> Hi, all. I haven't gotten any more comments on the proposal at
> http://wiki.apache.org/db-derby/ModuleVersioningGuidelines, so I'm
> going to update this page based on the comments I have received so far
> and then we'll try for another vote.
>
David will there be another document that describes the packaging and
any external impact of this change or will it be included in this one?
Thanks
Kathey
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Sorry, pilot error, I needed to log in.
David W. Van Couvering wrote:
> Hm, the front page appears to be an "Immutable Page." Is this perhaps
> because someone started editing it and didn't commit/back out their
> changes?
>
> David
>
> Daniel John Debrunner wrote:
>
>> David W. Van Couvering wrote:
>>
>>
>>> Hi, all. I haven't gotten any more comments on the proposal at
>>> http://wiki.apache.org/db-derby/ModuleVersioningGuidelines, so I'm going
>>> to update this page based on the comments I have received so far and
>>> then we'll try for another vote.
>>
>>
>>
>> Is there any reason this (and other wiki pages added by folks) are not
>> linked into the front page of the wiki? Is this just standard wiki
>> practice or are we just missing making the pages more visible?
>>
>> It seems like unless you know of the page then it's hard to join in any
>> discussion about it, the automated wiki diffs sent to the committer list
>> are basically unreadable.
>>
>>
>>
>> Dan.
>>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Hm, the front page appears to be an "Immutable Page." Is this perhaps
because someone started editing it and didn't commit/back out their changes?
David
Daniel John Debrunner wrote:
> David W. Van Couvering wrote:
>
>
>>Hi, all. I haven't gotten any more comments on the proposal at
>>http://wiki.apache.org/db-derby/ModuleVersioningGuidelines, so I'm going
>>to update this page based on the comments I have received so far and
>>then we'll try for another vote.
>
>
> Is there any reason this (and other wiki pages added by folks) are not
> linked into the front page of the wiki? Is this just standard wiki
> practice or are we just missing making the pages more visible?
>
> It seems like unless you know of the page then it's hard to join in any
> discussion about it, the automated wiki diffs sent to the committer list
> are basically unreadable.
>
>
>
> Dan.
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Kathey Marsden <km...@sbcglobal.net>.
David W. Van Couvering wrote:
> I can gladly add this to the front page, and perhaps we can make this
> a general policy. I would like to suggest that we have a sub-page off
> the front page for proposals that are being discussed/reviewed on the
> Wiki site.
>
That sounds like a good idea.
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
I can gladly add this to the front page, and perhaps we can make this a
general policy. I would like to suggest that we have a sub-page off the
front page for proposals that are being discussed/reviewed on the Wiki site.
David
Daniel John Debrunner wrote:
> David W. Van Couvering wrote:
>
>
>>Hi, all. I haven't gotten any more comments on the proposal at
>>http://wiki.apache.org/db-derby/ModuleVersioningGuidelines, so I'm going
>>to update this page based on the comments I have received so far and
>>then we'll try for another vote.
>
>
> Is there any reason this (and other wiki pages added by folks) are not
> linked into the front page of the wiki? Is this just standard wiki
> practice or are we just missing making the pages more visible?
>
> It seems like unless you know of the page then it's hard to join in any
> discussion about it, the automated wiki diffs sent to the committer list
> are basically unreadable.
>
>
>
> Dan.
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Daniel John Debrunner <dj...@debrunners.com>.
David W. Van Couvering wrote:
> Hi, all. I haven't gotten any more comments on the proposal at
> http://wiki.apache.org/db-derby/ModuleVersioningGuidelines, so I'm going
> to update this page based on the comments I have received so far and
> then we'll try for another vote.
Is there any reason this (and other wiki pages added by folks) are not
linked into the front page of the wiki? Is this just standard wiki
practice or are we just missing making the pages more visible?
It seems like unless you know of the page then it's hard to join in any
discussion about it, the automated wiki diffs sent to the committer list
are basically unreadable.
Dan.
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Hi, all. I haven't gotten any more comments on the proposal at
http://wiki.apache.org/db-derby/ModuleVersioningGuidelines, so I'm going
to update this page based on the comments I have received so far and
then we'll try for another vote.
Thanks,
David
David W. Van Couvering wrote:
> So, I'll wait for another overnight (over-day) period, but it sounds
> like we may have a level of consensus here (fingers crossed).
>
> What I propose to do is take Jeremy's quick brain dump (thanks again for
> doing this Jeremy) and turn it into a more complete policy around this.
> Hopefully I'll capture the essence of this correctly, but I'm sure
> you'll correct me if I miss something :)
>
> I'll put this on the Derby Wiki, to make it more readable/reviewable.
>
> Thanks all,
>
> David
>
> Jeremy Boynes wrote:
>
>> Daniel John Debrunner wrote:
>>
>>> Jeremy Boynes wrote:
>>>
>>>
>>>> Think of the carnage if it was java_14208_b13.sql_300.Connection
>>>
>>>
>>>
>>>
>>> It's actually instructive to look how Java solves this:
>>>
>>> - The interface is kept upwards compatible, by following rules such as
>>> only new methods/fields etc.
>>>
>>> Then to look at how a consumer, Derby, deals with the fact of multiple
>>> versions of the interface:
>>>
>>> - Derby only provides and uses the functionality to match the version of
>>> the interface (java.sql.Connection) loaded, determined at runtime.
>>>
>>> I'd thought this was a workable direction being proposed about six
>>> thousand message ago, upwards compatible apis and the ability for a
>>> consumer to handle a lower version. Not sure what derailed it.
>>>
>>
>> Kathey asked if I would be willing to write something up and I started
>> thinking about it but haven't had the bandwidth to write anything.
>>
>> As a quick braindump:
>> * there are rules that restrict changes to the interface going forward
>> * Sun (et al) stick to those rules even when it is painful
>> * no class/method is removed without being deprecated first
>> * the JVM supports version skew a little by not requiring an
>> implementation to implement all methods in a interface (at runtime)
>> * the interfaces often provide a mechanism to determine what features of
>> an API the implementation implements
>> * frameworks can utilize multiple classloaders to load multiple versions
>> of a class into the VM - tools have developed to simplify this
>>
>> I think we can support a scenario where:
>> * a comsumer expecting version X.Y will work with any implementation
>> X.Y' when Y' >= Y
>> * a consumer expecting X.Y will work with X.Y' when Y' < Y by version
>> detection and degraded functionality (Y' level). If Y level function
>> is required then it will gracefully die (able to detect rather
>> than AbstractMethodError)
>> * X.Y defines the interface version, i.e. X.Y.Z works with X.Y.Z' for
>> all Z'
>> * X defines the compatibility level i.e. X and X' are not guaranteed
>> to work together (although we will try to ensure they do)
>>
>> Basic rules:
>> * at the Z level, no API change is allowed, just implementation changes
>> * at the Y level, additive API changes are allowed but must be
>> accompanied by support in the versioning mechanism so that they can be
>> detected. Things can be deprecated
>> * at the Z level, incompatible changes are allowed. Items deprecated in
>> version X can be removed in version X+2 (implicitly (X+2).0.0).
>>
>> This all applies to operation in a single classloader. Where multiple
>> classloaders are involved (inside one VM or in multiple VMs) a
>> different set of versioning behaviours applies to the wire protocol.
>>
>> --
>> Jeremy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
So, I'll wait for another overnight (over-day) period, but it sounds
like we may have a level of consensus here (fingers crossed).
What I propose to do is take Jeremy's quick brain dump (thanks again for
doing this Jeremy) and turn it into a more complete policy around this.
Hopefully I'll capture the essence of this correctly, but I'm sure
you'll correct me if I miss something :)
I'll put this on the Derby Wiki, to make it more readable/reviewable.
Thanks all,
David
Jeremy Boynes wrote:
> Daniel John Debrunner wrote:
>
>> Jeremy Boynes wrote:
>>
>>
>>> Think of the carnage if it was java_14208_b13.sql_300.Connection
>>
>>
>>
>> It's actually instructive to look how Java solves this:
>>
>> - The interface is kept upwards compatible, by following rules such as
>> only new methods/fields etc.
>>
>> Then to look at how a consumer, Derby, deals with the fact of multiple
>> versions of the interface:
>>
>> - Derby only provides and uses the functionality to match the version of
>> the interface (java.sql.Connection) loaded, determined at runtime.
>>
>> I'd thought this was a workable direction being proposed about six
>> thousand message ago, upwards compatible apis and the ability for a
>> consumer to handle a lower version. Not sure what derailed it.
>>
>
> Kathey asked if I would be willing to write something up and I started
> thinking about it but haven't had the bandwidth to write anything.
>
> As a quick braindump:
> * there are rules that restrict changes to the interface going forward
> * Sun (et al) stick to those rules even when it is painful
> * no class/method is removed without being deprecated first
> * the JVM supports version skew a little by not requiring an
> implementation to implement all methods in a interface (at runtime)
> * the interfaces often provide a mechanism to determine what features of
> an API the implementation implements
> * frameworks can utilize multiple classloaders to load multiple versions
> of a class into the VM - tools have developed to simplify this
>
> I think we can support a scenario where:
> * a comsumer expecting version X.Y will work with any implementation
> X.Y' when Y' >= Y
> * a consumer expecting X.Y will work with X.Y' when Y' < Y by version
> detection and degraded functionality (Y' level). If Y level function
> is required then it will gracefully die (able to detect rather
> than AbstractMethodError)
> * X.Y defines the interface version, i.e. X.Y.Z works with X.Y.Z' for
> all Z'
> * X defines the compatibility level i.e. X and X' are not guaranteed
> to work together (although we will try to ensure they do)
>
> Basic rules:
> * at the Z level, no API change is allowed, just implementation changes
> * at the Y level, additive API changes are allowed but must be
> accompanied by support in the versioning mechanism so that they can be
> detected. Things can be deprecated
> * at the Z level, incompatible changes are allowed. Items deprecated in
> version X can be removed in version X+2 (implicitly (X+2).0.0).
>
> This all applies to operation in a single classloader. Where multiple
> classloaders are involved (inside one VM or in multiple VMs) a different
> set of versioning behaviours applies to the wire protocol.
>
> --
> Jeremy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Jeremy Boynes <jb...@apache.org>.
Daniel John Debrunner wrote:
> Jeremy Boynes wrote:
>
>
>>Think of the carnage if it was java_14208_b13.sql_300.Connection
>
>
> It's actually instructive to look how Java solves this:
>
> - The interface is kept upwards compatible, by following rules such as
> only new methods/fields etc.
>
> Then to look at how a consumer, Derby, deals with the fact of multiple
> versions of the interface:
>
> - Derby only provides and uses the functionality to match the version of
> the interface (java.sql.Connection) loaded, determined at runtime.
>
> I'd thought this was a workable direction being proposed about six
> thousand message ago, upwards compatible apis and the ability for a
> consumer to handle a lower version. Not sure what derailed it.
>
Kathey asked if I would be willing to write something up and I started
thinking about it but haven't had the bandwidth to write anything.
As a quick braindump:
* there are rules that restrict changes to the interface going forward
* Sun (et al) stick to those rules even when it is painful
* no class/method is removed without being deprecated first
* the JVM supports version skew a little by not requiring an
implementation to implement all methods in a interface (at runtime)
* the interfaces often provide a mechanism to determine what features of
an API the implementation implements
* frameworks can utilize multiple classloaders to load multiple versions
of a class into the VM - tools have developed to simplify this
I think we can support a scenario where:
* a comsumer expecting version X.Y will work with any implementation
X.Y' when Y' >= Y
* a consumer expecting X.Y will work with X.Y' when Y' < Y by version
detection and degraded functionality (Y' level). If Y level function
is required then it will gracefully die (able to detect rather
than AbstractMethodError)
* X.Y defines the interface version, i.e. X.Y.Z works with X.Y.Z' for
all Z'
* X defines the compatibility level i.e. X and X' are not guaranteed
to work together (although we will try to ensure they do)
Basic rules:
* at the Z level, no API change is allowed, just implementation changes
* at the Y level, additive API changes are allowed but must be
accompanied by support in the versioning mechanism so that they can be
detected. Things can be deprecated
* at the Z level, incompatible changes are allowed. Items deprecated in
version X can be removed in version X+2 (implicitly (X+2).0.0).
This all applies to operation in a single classloader. Where multiple
classloaders are involved (inside one VM or in multiple VMs) a different
set of versioning behaviours applies to the wire protocol.
--
Jeremy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Hm, that has continued to be the proposal I liked best. I thought what
derailed it was the inability to guarantee forward compatibility, that
is when a new consumer depends upon a new API.
For example, let's say you have an interface in release 1:
public MyInterface
{
int method1(int val);
}
Now in release 2 the consumer wants to add a new method:
public MyInterface
{
int method1(int val);
String method2(String val);
}
This is a backward-compatible change but not a forward-compatible
change, because if the new consumer calls method2 on the original
MyInterface, he'll get an error.
You could argue this is solvable by saying any modifications to an
interface require a new interface:
public MyInterface2 extends MyInterface
{
int method2(String val);
}
This is both backward- and forward-compatible.
The problem with this is the jar sealing problem. If the old jar is
first in the classpath, then MyInterface will be loaded from the old jar
and MyInterface2 will be loaded from the new jar. If these interfaces
are in the same package (org.apache.derby.common) then you'll get a
SecurityException.
I don't think that Java solves this particular problem, as it generally
assumes you have only one version in your classpath at a given time.
That's what I think has continued to derail any solution we propose.
Thus my proposal to rename the package, to solve the jar sealing problem.
David
Daniel John Debrunner wrote:
> Jeremy Boynes wrote:
>
>
>>Think of the carnage if it was java_14208_b13.sql_300.Connection
>
>
> It's actually instructive to look how Java solves this:
>
> - The interface is kept upwards compatible, by following rules such as
> only new methods/fields etc.
>
> Then to look at how a consumer, Derby, deals with the fact of multiple
> versions of the interface:
>
> - Derby only provides and uses the functionality to match the version of
> the interface (java.sql.Connection) loaded, determined at runtime.
>
> I'd thought this was a workable direction being proposed about six
> thousand message ago, upwards compatible apis and the ability for a
> consumer to handle a lower version. Not sure what derailed it.
>
> Dan.
>
>
>
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Daniel John Debrunner <dj...@debrunners.com>.
Jeremy Boynes wrote:
> Think of the carnage if it was java_14208_b13.sql_300.Connection
It's actually instructive to look how Java solves this:
- The interface is kept upwards compatible, by following rules such as
only new methods/fields etc.
Then to look at how a consumer, Derby, deals with the fact of multiple
versions of the interface:
- Derby only provides and uses the functionality to match the version of
the interface (java.sql.Connection) loaded, determined at runtime.
I'd thought this was a workable direction being proposed about six
thousand message ago, upwards compatible apis and the ability for a
consumer to handle a lower version. Not sure what derailed it.
Dan.
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
David W. Van Couvering wrote:
> This proposal was based on the assumption that our common code would
> *never* be exposed to the outside world.
>
> I also never claimed that we should apply this solution to third-party
> components. That's a fish I don't think we need to fry right now.
>
Although I do want to remind us all that, based on Dan's email a while
back, the JDK itself encountered the mixed version problem and solved/is
solving it through package renaming...
http://mail-archives.apache.org/mod_mbox/db-derby-dev/200509.mbox/%3c43275363.6000503@debrunners.com%3e
"The future plan is to rename the org.apache.** packages to be something
like com.sun.org.apache.** to fix this problem."
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
This proposal was based on the assumption that our common code would
*never* be exposed to the outside world.
I also never claimed that we should apply this solution to third-party
components. That's a fish I don't think we need to fry right now.
I think you are probably unhappy with this solution because it goes
against your vision of a modular architecture for Derby.
If you don't assume that any "module" of Derby can go with any other
version of any other module, but instead assume that all of Derby goes
together as a single unit, then it's fine if the client and engine code
for a given version are tied to the common code for the same version.
It actually makes sense -- you have well-defined and predictable
behavior in an unpredictable environment where multiple versions live in
the same classpath.
If you aren't thinking modular, I personally don't think it's that
arcane and confusing, particularly compared to making multiple copies of
an original "common" class.
I also can't think of any other solution besides class cloning (which I
really dislike) that can solve this problem and still give us common code.
David
Jeremy Boynes wrote:
> I find this arcane and confusing; worse, I don't think it really works.
>
> Code that is using the common API becomes tied to a specific version. By
> renaming the package we force all users of the common code to modify
> their code. We can do that for things in our source tree but not in others.
>
> Similarly this forces us to modify any external libraries we use to
> exhibit the same behaviour - for example, renaming o.a.commons.logging
> to o.a.commons.logging_1208. This means we have custom versions of all
> external libraries.
>
> Think of the carnage if it was java_14208_b13.sql_300.Connection
>
> --
> Jeremy
>
> David W. Van Couvering wrote:
>
>> Hi, all. I've been letting this brew in my subconscious this weekend,
>> with the strong belief that *somehow* there is a solution that can meet
>> all of our requirements (simple architecture, ease-of-use,
>> compatibility, no regressions, easy to maintain, reduce code
>> duplication).
>>
>> I think I may have something that will do this, and your comments are
>> much appreciated. There is a bit of overhead for the release process,
>> but I think if we are going to have overhead, that's the place to do it.
>>
>> The principle is that for each release, the common package is unique.
>> This is achieved by appending the release number to the package name.
>> So for the 10.1.2 release the package name is org.apache.common_1012.
>> In this way a consumer of the common classes for a given version is
>> guaranteed to load the right classes and no conflicts or confusion occur.
>>
>> As part of the release process, when you make a branch for a release,
>> you run a script that renames the common package to match your release
>> name. This script does the following:
>>
>> - In the source, replace all instances of
>> org.apache.common_<oldversion> with org.apache.common_<newversion>
>>
>> - Do an svn move of java/common/org/apache/common_<oldversion> to
>> java/common/org/apache/common_<newversion>
>>
>> Because you are using svn move, although it's a bit confusing, all
>> history is maintained and you can still use svn to do merges and ports
>> of changes.
>>
>> The common package is put into both derby.jar and derby-client.jar,
>> and we do not create a new JAR file.
>>
>> Thanks,
>>
>> David
>
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Jeremy Boynes <jb...@apache.org>.
I find this arcane and confusing; worse, I don't think it really works.
Code that is using the common API becomes tied to a specific version. By
renaming the package we force all users of the common code to modify
their code. We can do that for things in our source tree but not in others.
Similarly this forces us to modify any external libraries we use to
exhibit the same behaviour - for example, renaming o.a.commons.logging
to o.a.commons.logging_1208. This means we have custom versions of all
external libraries.
Think of the carnage if it was java_14208_b13.sql_300.Connection
--
Jeremy
David W. Van Couvering wrote:
> Hi, all. I've been letting this brew in my subconscious this weekend,
> with the strong belief that *somehow* there is a solution that can meet
> all of our requirements (simple architecture, ease-of-use,
> compatibility, no regressions, easy to maintain, reduce code duplication).
>
> I think I may have something that will do this, and your comments are
> much appreciated. There is a bit of overhead for the release process,
> but I think if we are going to have overhead, that's the place to do it.
>
> The principle is that for each release, the common package is unique.
> This is achieved by appending the release number to the package name. So
> for the 10.1.2 release the package name is org.apache.common_1012. In
> this way a consumer of the common classes for a given version is
> guaranteed to load the right classes and no conflicts or confusion occur.
>
> As part of the release process, when you make a branch for a release,
> you run a script that renames the common package to match your release
> name. This script does the following:
>
> - In the source, replace all instances of org.apache.common_<oldversion>
> with org.apache.common_<newversion>
>
> - Do an svn move of java/common/org/apache/common_<oldversion> to
> java/common/org/apache/common_<newversion>
>
> Because you are using svn move, although it's a bit confusing, all
> history is maintained and you can still use svn to do merges and ports
> of changes.
>
> The common package is put into both derby.jar and derby-client.jar, and
> we do not create a new JAR file.
>
> Thanks,
>
> David
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Rick Hillegas <Ri...@Sun.COM>.
There are now three cloning proposals for handling shared code:
1) Hand cloning by the developer as is done today.
2) Build-time cloning.
3) Release-time cloning (David's latest proposal).
All of these cloning approaches address Dan's problem case: a client and
server at diffferent rev levels in the same classpath. It's worth
pointing out that none of these proposals addresses the other mixed
usage scenario: wiring into the classpath two applications each of which
embeds a different version of the server. In that scenario we can still
expect Heisenbugs. We've never tested this scenario and so we have no
idea what our current exposure is. I agree with Jeremy that this
scenario is a systemic defect in Java and that we are not responsible
for solving it. The best we can do is point our users at best-of-breed
techniques for containing the problem.
Here's what I think about the cloning proposals:
(1)
+ Requires no changes.
- Brittle, bug-prone.
- Can't cut clone bloat by refactoring jar files.
(2)
+ Simple for bug porting.
- Can't cut clone bloat by refactoring jar files.
- Awkward to code and debug.
(3)
+ Easy to code and debug.
+ Clone bloat not necessary if client and server at same rev level.
- Awkward for bug porting.
In and of themselves, none of these approaches presents the naive
customer any additional complexity out-of-the-box. Additional complexity
would vex the customer if, for option (3) we refactored the jar files to
cut clone bloat. I dislike (3) the least.
Cheers,
-Rick
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Francois Orsini wrote:
> I think you meant 'org.apache.derby.common' in your examples.
Yes, thanks.
>
> Now, if the common package versioning is well automated as part of
> generating a new release, then it is probably ok to have a new version
> being generated for every release.
>
> Are you thinking that we would always see 'org.apache.derby.common' in
> the source tree and that the common pkg version suffix would only be
> added when a (new) release gets generated?
Well, this is a good question, if you're asking about what should the
package name be in the trunk. Yes, I suppose it could just be
'org.apache.derby.common' and the rename happens only when a branch is
made for a release. I am trying to think if this would cause any issues
for people pulling the nightly builds -- I can't think of any, unless
people try to mix versions from old and new nightly builds, in which
case I think they're really on their own :)
An alternate approach is to do the rename on the trunk just prior to
making the branch, so the trunk package name always matches the most
recent release branch. But this doesn't seem as clean...
David
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Francois Orsini <fr...@gmail.com>.
I think you meant 'org.apache.derby.common' in your examples.
Now, if the common package versioning is well automated as part of
generating a new release, then it is probably ok to have a new version being
generated for every release.
Are you thinking that we would always see 'org.apache.derby.common' in the
source tree and that the common pkg version suffix would only be added when
a (new) release gets generated?
--francois
On 9/19/05, David W. Van Couvering <Da...@sun.com> wrote:
>
> Hi, all. I've been letting this brew in my subconscious this weekend,
> with the strong belief that *somehow* there is a solution that can meet
> all of our requirements (simple architecture, ease-of-use,
> compatibility, no regressions, easy to maintain, reduce code duplication).
>
> I think I may have something that will do this, and your comments are
> much appreciated. There is a bit of overhead for the release process,
> but I think if we are going to have overhead, that's the place to do it.
>
> The principle is that for each release, the common package is unique.
> This is achieved by appending the release number to the package name.
> So for the 10.1.2 release the package name is org.apache.common_1012.
> In this way a consumer of the common classes for a given version is
> guaranteed to load the right classes and no conflicts or confusion occur.
>
> As part of the release process, when you make a branch for a release,
> you run a script that renames the common package to match your release
> name. This script does the following:
>
> - In the source, replace all instances of org.apache.common_<oldversion>
> with org.apache.common_<newversion>
>
> - Do an svn move of java/common/org/apache/common_<oldversion> to
> java/common/org/apache/common_<newversion>
>
> Because you are using svn move, although it's a bit confusing, all
> history is maintained and you can still use svn to do merges and ports
> of changes.
>
> The common package is put into both derby.jar and derby-client.jar, and
> we do not create a new JAR file.
>
> Thanks,
>
> David
>
>
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Hi, all. I've been letting this brew in my subconscious this weekend,
with the strong belief that *somehow* there is a solution that can meet
all of our requirements (simple architecture, ease-of-use,
compatibility, no regressions, easy to maintain, reduce code duplication).
I think I may have something that will do this, and your comments are
much appreciated. There is a bit of overhead for the release process,
but I think if we are going to have overhead, that's the place to do it.
The principle is that for each release, the common package is unique.
This is achieved by appending the release number to the package name.
So for the 10.1.2 release the package name is org.apache.common_1012.
In this way a consumer of the common classes for a given version is
guaranteed to load the right classes and no conflicts or confusion occur.
As part of the release process, when you make a branch for a release,
you run a script that renames the common package to match your release
name. This script does the following:
- In the source, replace all instances of org.apache.common_<oldversion>
with org.apache.common_<newversion>
- Do an svn move of java/common/org/apache/common_<oldversion> to
java/common/org/apache/common_<newversion>
Because you are using svn move, although it's a bit confusing, all
history is maintained and you can still use svn to do merges and ports
of changes.
The common package is put into both derby.jar and derby-client.jar, and
we do not create a new JAR file.
Thanks,
David
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
It sounds to me then that it's OK to have the same packages in derby.jar
and derby-client.jar. The classes in the common package would get
loaded from one and only of the jar files. We just need them in both so
the client and engine jars can be independent.
David
Daniel John Debrunner wrote:
> Andrew McIntyre wrote:
>
>
>
>>When a jar is sealed, classes cannot be loaded from multiple if the
>>package they are in is sealed. You can unseal specific packages, so
>>that's less of an issue than for signed jars. I believe attempting to
>>load the same class from different jars if the jars are signed will
>>cause a SecurityException, but I'm not well versed in the ins and outs
>>of signed jars.
>
>
> I think signing and sealing are independent concepts.
>
> If a class has been loaded from a signed jar, then it won't be loaded
> again from any jar (within the same class loader), so I don't think
> there can be any exception there.
>
> A sealed package means that all classes within that package must be
> loaded from the same jar. I assume this is true for a single
> classloader, not across the vm.
>
>
> Dan.
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Daniel John Debrunner <dj...@debrunners.com>.
Andrew McIntyre wrote:
> When a jar is sealed, classes cannot be loaded from multiple if the
> package they are in is sealed. You can unseal specific packages, so
> that's less of an issue than for signed jars. I believe attempting to
> load the same class from different jars if the jars are signed will
> cause a SecurityException, but I'm not well versed in the ins and outs
> of signed jars.
I think signing and sealing are independent concepts.
If a class has been loaded from a signed jar, then it won't be loaded
again from any jar (within the same class loader), so I don't think
there can be any exception there.
A sealed package means that all classes within that package must be
loaded from the same jar. I assume this is true for a single
classloader, not across the vm.
Dan.
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Andrew McIntyre <mc...@gmail.com>.
On 9/16/05, David W. Van Couvering <Da...@sun.com> wrote:
> I appreciate your pragmatic approach, Andrew. The thing is I have seen
> a number of other pieces of functionality queued up for code sharing
> between client and server. These include DRDA network code, potentially
> some higher-level JDBC functionality, logging and tracing, and
> management via JMX. I also discovered that the versioning mechanism in
> org.apache.derby.iapi.services.info is actually reused across tools,
> client and engine.
This is true, I realize that there are a lot of opportunities for code
sharing and reuse. I just don't know if *this* is the one that is
worth it. I think the way these possible code-sharing opportunities
should all be handled is for someone to implement some new
functionality as a module and package it into a new *optional* jar
file. The new code can be used if it is present, and the old code can
be deprecated, but remain packaged for a certain number of releases
before it is removed, if it is removed. This gives users time to
prepare for a change and developers to work through all the issues
without causing any unnecessary pain along the way.
Do we really want to go to this trouble for the client localization
functionality? I don't know. To me, logging seems like a better first
project for that.
> We may still decide this is intractable and throw up our hands, but I
> hope that's not the case. It's been a healthy debate, and has brought
> to bear some good discussions about "who" we want to be, what our key
> principles are, and I think that's been good too.
I agree, it's been a good healthy debate, but I thought it might be
good to reflect upon the task at hand. :-)
> You talked about signing/sealing problems about packaging the classes in
> multiple jars. I saw this mentioned before, can you or someone else
> elaborate?
When a jar is sealed, classes cannot be loaded from multiple if the
package they are in is sealed. You can unseal specific packages, so
that's less of an issue than for signed jars. I believe attempting to
load the same class from different jars if the jars are signed will
cause a SecurityException, but I'm not well versed in the ins and outs
of signed jars.
andrew
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
I appreciate your pragmatic approach, Andrew. The thing is I have seen
a number of other pieces of functionality queued up for code sharing
between client and server. These include DRDA network code, potentially
some higher-level JDBC functionality, logging and tracing, and
management via JMX. I also discovered that the versioning mechanism in
org.apache.derby.iapi.services.info is actually reused across tools,
client and engine.
Given this, I thought that although it would be a tough debate it was
worthwhile having and one which, if we solved, would enable a lot of
other common services to be built across client and engine. So I took
the challenge and decided to try and solve this.
We may still decide this is intractable and throw up our hands, but I
hope that's not the case. It's been a healthy debate, and has brought
to bear some good discussions about "who" we want to be, what our key
principles are, and I think that's been good too.
You talked about signing/sealing problems about packaging the classes in
multiple jars. I saw this mentioned before, can you or someone else
elaborate? If it's really true that we can't package the same classes
in multiple jars, that's a very important data point...
Thanks,
David
Andrew McIntyre wrote:
> On 9/15/05, Kathey Marsden <km...@sbcglobal.net> wrote:
>
>>Jeremy Boynes wrote:
>>
>>
>>>This dooms us forever to reinvent any functionality that could
>>>be provided by other projects.
>>>
>>
>>We are not "doomed forever". Requiring a new jar file for new
>>functionality seems an entirely reasonable thing to me and at that time
>>we can impose whatever restrictions the community sees fit. Requiring a
>>new jar file to have the product continue work, is another matter all
>>together.
>
>
> +1!
>
> It looks like we've got a really intractable situation here. There are
> those against source/binary modification because of the inherent
> problems with maintenance and debugging, but you can't package the
> same classes in multiple jars because of signing/sealing and
> classloader issues, and if you split the classes out into a new jar
> then you've got usability and deployment issues. So what do you do?
>
> I'm wondering if this is really the functionality to be considering
> all of this for. Two or three small classes related to localization.
> Is it really worth the trouble? Are we really going to make a common
> jar file with two classes, without which you don't get error messages?
> Really?
>
> And remember, this isn't about reusing someone else's code or
> libraries. I don't think anyone is going to be against reusing other
> project's code or libraries in order to promote cross-project goodwill
> and general open source happiness. We're talking about copying two of
> our own classes from one part of the tree to the other.
>
> I think we can all tackle the code-sharing problem with some new
> functionality later. For which, by the way, I think a good candidate
> would be integrating Lucene for full text indexing as a good trial for
> both code reuse from other projects and code sharing among the
> different parts of Derby. And personally, I'd like to see David be
> able to finish localizing the error messages sometime this decade. :-)
>
> andrew
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Jeremy Boynes <jb...@apache.org>.
Evolution would start with a small change that was then built on and
expanded. From that perspective, starting with a small change like the
localization would be the beginning of an evolutionary change. Doing it
now, near the start of a feature release, and providing plenty of time
for feedback from users in the community is less disruptive than trying
to rush it in at the end.
This is one area where open source is different than closed source
development - all users get to see changes early and can provide
feedback into the process, not just a selected few during a beta cycle
when it is usually far to late.
As for the size of the common jar, once you go beyond one adding
additional libraries in becomes just more of the same. That leads to a
model where packaging is based on loose vs. close coupling
considerations. Some services may be small, some may be larger, but the
boundaries are determined by the use case and not an arbitrary packaging
scheme.
This was an issue originally with Jakarata Commons. In its early days,
although the libraries were separated out from the original monolith
they were still coupled together; it wasn't until later that the
decoupling of services we have now was attained (and to some extent that
is still a work in progress).
--
Jeremy
Satheesh Bandaram wrote:
> +1. I also vote to solve the common code issues when we have sufficient
> critical mass of it. Currently
> there doesn't seem to be enough to justify a new mechanism, especially a
> solution that is directly visible
> to end users and could cause some disruptions. While the discussion has
> been very useful to evolve a
> possible solution in the future, I don't see an urgent need to address
> this yet. Not sure how end-users
> would see a derbycommon.jar that is only a few kilobytes (tens?) in size
> and wonder why it is even there.
>
> We just released a 10.1 feature release... Not sure when our next
> feature release is going to be, but seems
> like there might be many months of time to address any packaging issues.
> Let us continue to enhance
> Derby in the current framework and as we get closer to another release,
> we could evaluate the packaging
> issues.
>
> Evolution, not revolution... Someone said this before, don't remember
> who ... :-)
>
> Satheesh
>
> Andrew McIntyre wrote:
>
>
>>+1!
>>
>>It looks like we've got a really intractable situation here. There are
>>those against source/binary modification because of the inherent
>>problems with maintenance and debugging, but you can't package the
>>same classes in multiple jars because of signing/sealing and
>>classloader issues, and if you split the classes out into a new jar
>>then you've got usability and deployment issues. So what do you do?
>>
>>I'm wondering if this is really the functionality to be considering
>>all of this for. Two or three small classes related to localization.
>>Is it really worth the trouble? Are we really going to make a common
>>jar file with two classes, without which you don't get error messages?
>>Really?
>>
>>And remember, this isn't about reusing someone else's code or
>>libraries. I don't think anyone is going to be against reusing other
>>project's code or libraries in order to promote cross-project goodwill
>>and general open source happiness. We're talking about copying two of
>>our own classes from one part of the tree to the other.
>>
>>I think we can all tackle the code-sharing problem with some new
>>functionality later. For which, by the way, I think a good candidate
>>would be integrating Lucene for full text indexing as a good trial for
>>both code reuse from other projects and code sharing among the
>>different parts of Derby. And personally, I'd like to see David be
>>able to finish localizing the error messages sometime this decade. :-)
>>
>>andrew
>>
>>
>>
>>
>>
>
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Satheesh Bandaram <sa...@Sourcery.Org>.
+1. I also vote to solve the common code issues when we have sufficient
critical mass of it. Currently
there doesn't seem to be enough to justify a new mechanism, especially a
solution that is directly visible
to end users and could cause some disruptions. While the discussion has
been very useful to evolve a
possible solution in the future, I don't see an urgent need to address
this yet. Not sure how end-users
would see a derbycommon.jar that is only a few kilobytes (tens?) in size
and wonder why it is even there.
We just released a 10.1 feature release... Not sure when our next
feature release is going to be, but seems
like there might be many months of time to address any packaging issues.
Let us continue to enhance
Derby in the current framework and as we get closer to another release,
we could evaluate the packaging
issues.
Evolution, not revolution... Someone said this before, don't remember
who ... :-)
Satheesh
Andrew McIntyre wrote:
>+1!
>
>It looks like we've got a really intractable situation here. There are
>those against source/binary modification because of the inherent
>problems with maintenance and debugging, but you can't package the
>same classes in multiple jars because of signing/sealing and
>classloader issues, and if you split the classes out into a new jar
>then you've got usability and deployment issues. So what do you do?
>
>I'm wondering if this is really the functionality to be considering
>all of this for. Two or three small classes related to localization.
>Is it really worth the trouble? Are we really going to make a common
>jar file with two classes, without which you don't get error messages?
>Really?
>
>And remember, this isn't about reusing someone else's code or
>libraries. I don't think anyone is going to be against reusing other
>project's code or libraries in order to promote cross-project goodwill
>and general open source happiness. We're talking about copying two of
>our own classes from one part of the tree to the other.
>
>I think we can all tackle the code-sharing problem with some new
>functionality later. For which, by the way, I think a good candidate
>would be integrating Lucene for full text indexing as a good trial for
>both code reuse from other projects and code sharing among the
>different parts of Derby. And personally, I'd like to see David be
>able to finish localizing the error messages sometime this decade. :-)
>
>andrew
>
>
>
>
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Andrew McIntyre <mc...@gmail.com>.
On 9/15/05, Kathey Marsden <km...@sbcglobal.net> wrote:
> Jeremy Boynes wrote:
>
> >
> > This dooms us forever to reinvent any functionality that could
> > be provided by other projects.
> >
> We are not "doomed forever". Requiring a new jar file for new
> functionality seems an entirely reasonable thing to me and at that time
> we can impose whatever restrictions the community sees fit. Requiring a
> new jar file to have the product continue work, is another matter all
> together.
+1!
It looks like we've got a really intractable situation here. There are
those against source/binary modification because of the inherent
problems with maintenance and debugging, but you can't package the
same classes in multiple jars because of signing/sealing and
classloader issues, and if you split the classes out into a new jar
then you've got usability and deployment issues. So what do you do?
I'm wondering if this is really the functionality to be considering
all of this for. Two or three small classes related to localization.
Is it really worth the trouble? Are we really going to make a common
jar file with two classes, without which you don't get error messages?
Really?
And remember, this isn't about reusing someone else's code or
libraries. I don't think anyone is going to be against reusing other
project's code or libraries in order to promote cross-project goodwill
and general open source happiness. We're talking about copying two of
our own classes from one part of the tree to the other.
I think we can all tackle the code-sharing problem with some new
functionality later. For which, by the way, I think a good candidate
would be integrating Lucene for full text indexing as a good trial for
both code reuse from other projects and code sharing among the
different parts of Derby. And personally, I'd like to see David be
able to finish localizing the error messages sometime this decade. :-)
andrew
Re: Modular build, was: VOTE: Approach for sharing code
Posted by David Jencks <da...@yahoo.com>.
Admittedly I approach this mostly from an app server perspective, but I
prefer
--Create a common package and put all common code there
--Make a separate jar with this stuff in it
--Leave the common classes out of derby.jar and derby-client.jar
--For users convenience in standalone apps produce an additional jar
that includes everything from common, derby, derby-client, network,
drda, etc etc. This would not be used if you are concerned about
classpaths and versions or the smallest possible footprint.
--Make sure that all the jars have clear version numbers in their names.
I think you are asking for major classloader trouble including the same
classes in 2 different jars (doesn't this produce a sealing or signing
problem?) and I think the package-name munging approach is really ugly
and twisted.
thanks
david jencks
On Sep 13, 2005, at 3:34 PM, David W. Van Couvering wrote:
> Hi, Kathy, thanks for your email. The timing is actually pretty good,
> I was just talking with Francois trying to understand his concerns
> better.
>
> After quickly describing the two approaches, I'd like to summarize the
> experience/impact of these approaches from the perspectives of the end
> user, the developer/maintainer, and the test developer/runner.
>
> Goal:
> - Reduce code duplication while continuing to support different
> versions of client and embedded driver in the same VM
>
> Approach 1:
> - Create a common package and put all common code there
> (org.apache.derby.common)
> - Use compatibility guidelines to ensure backward compatibility and
> some degree of forward compatibility
> - Common classes are embedded in derby.jar and derby-client.jar
> (based on lots of negative feedback for having a separate jar)
>
> Approach 2:
> - Create a common package and put all common code there
> (org.apache.derby.common)
> - During build process, "clone" all common classes into at least two
> generated packages, one for the engine and one for the network client
> (org.apache.derby.engine.common and org.apache.derby.client.common).
> - This approach avoids having to implement compatiblity
> guidelines/constraints and guarantees, as the engine and client
> continue to be fully isolated
>
> USER EXPERIENCE
>
> Approach 1
> - No new jar files, everything looks the same
> - For the vast percentage of users who don't mix versions in the same
> VM, everything works great
> - For the small percentage of users who mix versions, the order in
> which jar files are loaded. For example, if they want to use
> derby-10.2.jar and derby-client-11.0.jar, then if derby-10.2.jar is
> first in the classpath, they will get an exception saying that the
> client code is incompatible with the common code and that they need to
> re-order their jar files, whereas it will work fine if the order is
> reversed.
>
> Approach 2
> - No new jar files, everything looks the same
> - Ordering of jar files does not matter, everything works fine
>
>
> DEVELOPER EXPERIENCE
>
> Approach 1
> - Pretty much as it is today, nothing surprising
>
> Approach 2
> - When navigating code either during source browsing or debugging,
> they will see the *generated* common code, not the master common code.
> - If a developer is not aware of how things work, or just forgets,
> he/she will try to edit this generated code, and will be confused and
> surprised when his/her changes disappear after a build
> - Stack traces will point to generated code, and the developer will
> have to remember to translate that back to the master version.
> - The generation process must be very careful not to adjust line
> numbers or all stack traces will be off and misleading. This means
> for instance we can't add comments saying "THIS IS GENERATED CODE, DO
> NOT EDIT"
> - We may need to generate more copies if different types of version
> mixing are required (e.g. if the tools.jar and derby.jar need to be at
> different versions)
>
>
> TESTER EXPERIENCE
> Approach 1
> - We would have to build and run compatibility tests to make sure
> that compatible versions run correctly and incompatible ones correcty
> raise the exception
>
> Approach 2
> - No real change, although debugging of tests may be confusing due to
> issues I already listed under developer experience
>
>
> I also am uncomfortable with the "twistiness" of approach 2. There is
> something to be said for a clean architecture and build environment.
> I have seen time and again that a good architecture allows you to
> scale and grow, whereas "twisty" architectures tend to wrap you up and
> tie you down at some point. I think this has to be taken into
> consideration.
>
> My main question is: is it OK to sometimes throw an exception for the
> small set of users who mix versions, and for them to then have to
> rearrange their jar ordering. If the answer is that this is not
> acceptable and would be considered a showstopper regression for some
> part of our user base, then I think we have no choice but to go with
> Approach 2, even if we do risk painting ourselves into an
> architectural corner. Otherwise, it is my strong recommendation to go
> with Approach 1.
>
> Thanks!
>
> David
>
> Kathy Saunders wrote:
>
>> David W. Van Couvering wrote:
>>
>>> Well, we're at a bit of a standoff here. What I'm looking for is a
>>> nail-in-the-coffin data point that would move us in one direction or
>>> the other.
>>
>>
>> I've been following this discussion with some interest as my
>> background is technical support. In fact, I supported Cloudscape for
>> the first 4 years (from release 1.0 of Cloudscape). My experience is
>> that developers (particularly Java developers) really liked
>> Cloudscape (now Derby) because it was so easy to use and deploy.
>> And, I found historically that one of the most common issues to come
>> up were classpath issues (in particular we got in trouble a few times
>> when we introduced something that caused the order of the jar files
>> to be important). You should note that I'm not, and never have been,
>> a Derby developer, so I don't claim to be an expert on what's correct
>> and best from a development perspective.
>>
>> I'm a bit concerned because I see a lot of discussion about what is
>> good from a derby development perspective, but not so much how these
>> changes may affect users of Derby. Although some Derby users have
>> complex applications (like application servers), many are
>> implementing much more simple solutions.
>> Having said that, I'm a bit lost in what is being proposed from the
>> user/functional perspective. David, as soon as you have a more
>> concrete proposal (may not be the time yet), can you post that
>> information? Could you provide information on what the users of
>> Derby will have to do with this change (how would our documentation
>> need to be changed) and maybe footprint, performance, etc. impacts
>> vs. the benefits from making this change. I'd like to be able to
>> provide my input from a usability/documentation perspective.
>>
>> In addition, I work on Derby now in the testing area, so I'd also
>> like to understand the implications for what additional testing might
>> need to be done. If we create more jar files, is there more testing
>> requirements for different combinations?
>>
>> Thanks,
>> Kathy
>>
> <david.vancouvering.vcf>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Hi, Kathey. My responses below...
Kathey Marsden wrote:
>David W. Van Couvering wrote:
>
>
>
>>Hi, Kathy, thanks for your email. The timing is actually pretty good,
>>I was just talking with Francois trying to understand his concerns
>>better.
>>
>>After quickly describing the two approaches, I'd like to summarize the
>>experience/impact of these approaches from the perspectives of the end
>>user, the developer/maintainer, and the test developer/runner.
>>
>>
>>
>
>Thank you David for the summary. I thought "modular build" meant
>adding more jars, so it was good to have that cleared With Approach 1
>I personally am not too keen on the new ordering requirements, testing
>requirements, or the potential for regression with the versioning scheme.
>
>
Note that we already have to do compatibility testing between client and
server because of protocol changes. But yes, this does increase the
amount of testing we have to do.
The ordering requirements are for a very small subset of users. For
most users, ordering does not matter at all.
The regression question to me is the clincher. If we just are not able
to drop in a new version of Derby that could cause existing code to
break because of their classpath order, then Approach 1 is just not
feasible. I am hoping we could document this change and the users to
whom this is important could with very little effort adjust how they set
up the order their jars are loaded.
In general I am concerned we are making architectural compromises for a
small subset of customers. One of my managers at Sybase once told me
his priorities for making decisions around changes to a product: first
the product, then the customer, then the engineers. What he meant was,
you don't want to compromise the overall "solidity" of the product for
the sake of a small set of customers.
>For the USER EXPERIENCE for either approach, how much growth do you
>anticipate in the client due to code sharing with the first round of
>what you want to share and a guesstimate of how big it might get if we
>utilize all that you think the client should use from the engine?
>The earlier thread on the size of the common jar file
>http://mail-archives.apache.org/mod_mbox/db-derby-dev/200507.mbox/%3c42DF40D5.8050605@debrunners.com%3e
>made it sound significant.
>
>
>
That thread was assuming I was migrating the entire Monitor/Service
architecture over into common, and it was getting a bit big. This is
not necessarily going to be required. I have some thoughts, not yet
fully gelled, about how we might be able to introduce a lightweight
service architecture into common, and have this be compatible with the
heavier-weight service architecture currently in the engine.
I'd like to understand what requirements we have for the footprint of
the client jar.
Thanks,
David
>Kathey
>
>
>
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Kathey Marsden <km...@sbcglobal.net>.
David W. Van Couvering wrote:
> Hi, Kathy, thanks for your email. The timing is actually pretty good,
> I was just talking with Francois trying to understand his concerns
> better.
>
> After quickly describing the two approaches, I'd like to summarize the
> experience/impact of these approaches from the perspectives of the end
> user, the developer/maintainer, and the test developer/runner.
>
Thank you David for the summary. I thought "modular build" meant
adding more jars, so it was good to have that cleared With Approach 1
I personally am not too keen on the new ordering requirements, testing
requirements, or the potential for regression with the versioning scheme.
For the USER EXPERIENCE for either approach, how much growth do you
anticipate in the client due to code sharing with the first round of
what you want to share and a guesstimate of how big it might get if we
utilize all that you think the client should use from the engine?
The earlier thread on the size of the common jar file
http://mail-archives.apache.org/mod_mbox/db-derby-dev/200507.mbox/%3c42DF40D5.8050605@debrunners.com%3e
made it sound significant.
Kathey
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
One thing I'd like to raise is that the code cloning is a build process.
If in the future we realize we have painted ourselves into a corner,
unless I'm missing something we can just remove the cloning step and we
have the module-based approach that Jeremy and I have been backing.
Right?
David
Jeremy Boynes wrote:
> David W. Van Couvering wrote:
>
>>
>>
>> Jeremy Boynes wrote:
>> [snip]
>>
>>> Daniel John Debrunner wrote:
>>>
>>>> The issue is that today this is fully supported because the client and
>>>> engine do not share code.
>>>>
>>>> Some of the code sharing approaches regress Derby in this area by not
>>>> supporting this, or require class path ordering for it to be supported.
>>>>
>>>
>>> Some of the others support this by defining compatibility contracts
>>> and eliminate the need for classpath ordering by not duplicating
>>> classes.
>>
>>
>>
>> Can't you have the situation where common 10.2 and common 10.3 are
>> both included in the classpath (by accident, as Dan brings up)?
>> Wouldn't you end up with order dependencies then?
>>
>
> To what extent do we need to cater for accidents? But, yes if you do end
> up with both then the (actual) order would determine which one you ended
> up with. In many cases the user would not notice due to the API
> compatibility between 10.2 and 10.3 (with 10.3 being a pure superset by
> the rules defined). The key thing is that there is no overlap in
> functionality between jars (so you are not getting 10.2 (containing
> client + common) merged with 10.3 (containing engine + common)).
>
>> [snip]
>>
>>> Let's recharacterize this a little. What we are contemplating with
>>> code sharing is extracting common functionality out into a library.
>>> By saying that we are not willing to accept any solution where a
>>> component depends on a library we are shutting ourselves off from
>>> using any external library or any functionality not provided by Derby
>>> itself. This dooms us forever to reinvent any functionality that
>>> could be provided by other projects.
>>>
>>> For example, there are libraries out there that support bytecode
>>> generation, JMX for management, high-performance concurrency on Java
>>> 1.4, regexp processing to support SQL patterns, ... By saying we are
>>> not prepared to incorporate them but instead need our own versions
>>> that can be morphed for client and server we dramatically reduce the
>>> functionality that can be made available to users.
>>>
>>> So let me ask this: do our users want more functionality faster by
>>> allowing the use of libraries, or a completely standalone solution
>>> with tight control over the entire implementation?
>>
>>
>>
>> You make a compelling argument here. I already would like to use
>> stuff in Jakarta Commons (I haven't brought it up yet, one thing at a
>> time). It seems a good Apache Java citizen should make use of what's
>> in Jakarta Commons rather than build stuff themselves. And I think
>> Jeremy's right that we will run into this same configuration situation
>> with these guys.
>>
>> Jeremy, how *do* the users of commons avoid accidentally using a
>> version they are not compatible with (e.g. a consumer depends on new
>> features that aren't available in an older version of the common jar
>> file)?
>>
>
> This can be a problem - Java's variant of DLL hell. The solution
> generally relies on two factors:
> 1) upward compatibility between versions - because they are small-ish
> components then they tend to do one thing and not suffer from API
> scope creep. You just use the latest version and add it to the
> classpath once
>
> 2) multi-classloader loading mechanisms, which are commonplace in
> open source projects, even command line utilities like Ant and Maven
> not just the application servers
>
> It is kind of self-fulfilling - when you adopt a modular structure then
> you expect to be loading modules with libraries at different version
> levels and so use a classloading architecture that fits.
>
> There is also a cultural thing here. For example, suppose I am writing a
> database because I happen to think writing database stuff is fun (yes, I
> do realize that doing this for fun makes one appear somewhat odd). I
> come up with the idea that converting the query plan to executable code
> is a neat idea and so need to generate that code. I can implement a code
> generator myself, or look for one out there that is already written and
> is available under a suitable license. Given my main interest is in
> databases, using someone else's code to generate byte code save me a lot
> of effort (especially when the class format changes). If it doesn't do
> what I want, I could hack my own but it is generally easier to donate
> those changes back so that someone else can maintain them. After all, my
> interest is in databases not in byte code generation.
>
> Basically, a volunteer community scales better by incorporating each
> other's code rather than trying to do it all.
>
> --
> Jeremy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Kathey Marsden <km...@sbcglobal.net>.
Jeremy Boynes wrote:
> Rather than address this ourselves (and just for Derby) by convoluted
> code-rewrites and avoidance of third party libraries we should
> encourage the application providers avoid this entirely by exhibiting
> discipline when using shared resources
You have an extremely important point Jeremy. I think as with any
discipline issue, education is key. We need to make mixed versions
truly be the edge case that we all want it to be. Once that is the
case, it is going to provide us with much more flexibility. What I
need as a person who talks to users and support folks all the time is a
good document and hopefully an example of what what developers should
do to make sure their application load the copy of Derby that they want
even if they find themselves in a JVM with another instance of Derby
loaded by the classpath or if they use a component that might use
another version of Derby.
I think the discipline that we want to be able to expect is not
mentioned at all in our documentation and we need to focus on educating
our users on exactly what to do. Would you be willing to write
something up that we could put in the documentation or provide a white
paper? Helping the Users do the right thing might untie the hands of
the Engineers so they can make the Product better sooner.
Kathey
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Daniel John Debrunner <dj...@debrunners.com>.
Jeremy Boynes wrote:
> Daniel John Debrunner wrote:
>
>>
>> But as an aside, I once downloaded an open source project, which then
>> instructed me to fetch a further five or six common Apache libraries,
>> either at a specific version numbers or higher than a given version. I
>> followed all this correctly, set up the classpath and immediately got an
>> abstract method error.
>>
>> That's an example of badly managed common versions, and I do not wish
>> Derby to fall into the same trap.
>
> And how many open source projects have you downloaded or used without
> such issues? Please don't taint this from one bad experience.
Nothing tainted, I'm just pointing out there are bad ways to handle
common code.
> All of these lead to a simple end-user experience
and that's what we need, any solution needs to continue the simple
end-user experience.
Dan.
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Jeremy Boynes <jb...@apache.org>.
Daniel John Debrunner wrote:
>
> But as an aside, I once downloaded an open source project, which then
> instructed me to fetch a further five or six common Apache libraries,
> either at a specific version numbers or higher than a given version. I
> followed all this correctly, set up the classpath and immediately got an
> abstract method error.
>
> That's an example of badly managed common versions, and I do not wish
> Derby to fall into the same trap.
>
> 1) The ease to download goes away in that case where I had to spend time
> finding and downloading N additional jars.
>
> 2) The ease to use goes away when I have to modify the classpath to add
> six or seven jars.
>
> 3) The desire to use goes away completely when having followed the
> instructions the product fails.
>
> This e-mail isn't to solict ideas on how specifics can be solved, but to
> understand that Derby's ease of use in all aspects is one of its
> greatest strengths, we cannot forget that, or lose it.
>
And how many open source projects have you downloaded or used without
such issues? Please don't taint this from one bad experience.
For build-time dependencies, many of these issues are addressed by Maven
which automatically downloads the dependencies specified in the project.
This would be an immediate solution to the problem raised on the
recent junit thread.
For run-time dependencies, these can be addressed with solutions like
uberjar:
1) for Derby as a standalone server, we can have one executable jar that
loads all dependencies from an internal classloader (no classpath
issues at all)
2) for Derby as a library, the application embedding us should be using
the same type of solution and we would fit in there just like
clogging. With it controlling the classpath there is nothing for the
user to (mis-)configure
All of these lead to a simple end-user experience - whether they are
trying to build from source, use a standalone server, or embed inside
another application.
--
Jeremy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Daniel John Debrunner <dj...@debrunners.com>.
David W. Van Couvering wrote:
> On thinking further about this, I have a couple more questions/concerns
>
> - I think Jeremy has a very good point that has not been directly
> answered. If we do not allow potential compatibility issues by running
> multiple versions in the same VM, then doesn't that mean we are not
> going to be willing to incorporate third-party components like from
> Jakarta Commons?
No. If we have common code, I don't see how it matters if it's written
as part of Derby or elsewhere. We would take the same approach for both.
> I think this is a real problem, and I think a stance like this would not
> be looked upon positively from the rest of the Apache community.
No-one has suggested such a stance.
But as an aside, I once downloaded an open source project, which then
instructed me to fetch a further five or six common Apache libraries,
either at a specific version numbers or higher than a given version. I
followed all this correctly, set up the classpath and immediately got an
abstract method error.
That's an example of badly managed common versions, and I do not wish
Derby to fall into the same trap.
1) The ease to download goes away in that case where I had to spend time
finding and downloading N additional jars.
2) The ease to use goes away when I have to modify the classpath to add
six or seven jars.
3) The desire to use goes away completely when having followed the
instructions the product fails.
This e-mail isn't to solict ideas on how specifics can be solved, but to
understand that Derby's ease of use in all aspects is one of its
greatest strengths, we cannot forget that, or lose it.
Dan.
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Daniel John Debrunner <dj...@debrunners.com>.
David W. Van Couvering wrote:
> - It seems to me that the environments in which mixed versions in the
> same VM occur are:
>
> * app servers, which already deal with this through multiple
> classloaders
I'm not sure that's a completely valid assumption. Some app servers have
different classloaders for different ear/war files, but not for
different JDBC providers. (which would be a Derby client and a Derby
engine for example).
Dan.
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
On thinking further about this, I have a couple more questions/concerns
- I think Jeremy has a very good point that has not been directly
answered. If we do not allow potential compatibility issues by running
multiple versions in the same VM, then doesn't that mean we are not
going to be willing to incorporate third-party components like from
Jakarta Commons?
I think this is a real problem, and I think a stance like this would not
be looked upon positively from the rest of the Apache community.
- It seems to me that the environments in which mixed versions in the
same VM occur are:
* app servers, which already deal with this through multiple
classloaders
* home-built servers in which case we can expect the user to be
expert enough to know how to affect the order in which jars are loaded
- I agree with Jeremy that dealing with shared resources is a well-known
problem and although mistakes can happen good app providers know how to
deal with this. Is it our responsibility to munge our code and paint
ourselves into a corner regarding use of shared components to deal with
app providers who don't know how to work with this?
Thanks,
David
Jeremy Boynes wrote:
> Daniel John Debrunner wrote:
>
>> Jeremy Boynes wrote:
>>
>>
>>> David W. Van Couvering wrote:
>>
>>
>>
>>>> Can't you have the situation where common 10.2 and common 10.3 are
>>>> both included in the classpath (by accident, as Dan brings up)?
>>>> Wouldn't you end up with order dependencies then?
>>
>>
>>
>> I feel my scenario keeps being misrepresented by the choice of terms
>> used to decribe it. Using 'accident' makes it sounds as though it's not
>> an important problem to deal with, as seen in Jeremy's reponse here:
>>
>>
>>> To what extent do we need to cater for accidents?
>>
>>
>>
>> The end user didn't accidently install two applications, they chose to
>> and didn't realise/know that one used client at version 10.2 and one
>> used engine at version 10.3. In many cases the use of Derby engine is
>> hidden by the application developer.
>>
>
> I actually thought "accident" was appropriate here :-) The collision in
> Derby versions is "an unexpected and undesirable event, especially one
> resulting in damage or harm"[1] arising from the installation of two
> applications.
>
> I would assign fault here to the application providers who are not
> allowing for other applications using a common shared resource - the
> system or user classpath; it's like if they both insisted in being
> installed in the same directory.
>
> I would also say that most application providers are aware of this
> problem (usually from some unfortunate prior experience) and take
> explicit control of the classpath using mechanisms described elsewhere.
> Their end-users are unaffected by this scenario.
>
> So whilst there is high impact from your scenario, it applies to
> end-users who install multiple applications that do not exhibit
> classpath discipline and which happen to use different versions of Derby
> client and engine (but not two different engine versions or two
> different client versions). And never mind that these users may be
> affected by any other libraries these applications use.
>
> Rather than address this ourselves (and just for Derby) by convoluted
> code-rewrites and avoidance of third party libraries we should encourage
> the application providers avoid this entirely by exhibiting discipline
> when using shared resources. We can even point them at something like:
> http://classworlds.codehaus.org/
>
> --
> Jeremy
>
> [1] http://dictionary.reference.com/search?q=accident
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Jeremy Boynes <jb...@apache.org>.
Daniel John Debrunner wrote:
> Jeremy Boynes wrote:
>
>
>>David W. Van Couvering wrote:
>
>
>>>Can't you have the situation where common 10.2 and common 10.3 are
>>>both included in the classpath (by accident, as Dan brings up)?
>>>Wouldn't you end up with order dependencies then?
>
>
> I feel my scenario keeps being misrepresented by the choice of terms
> used to decribe it. Using 'accident' makes it sounds as though it's not
> an important problem to deal with, as seen in Jeremy's reponse here:
>
>
>>To what extent do we need to cater for accidents?
>
>
> The end user didn't accidently install two applications, they chose to
> and didn't realise/know that one used client at version 10.2 and one
> used engine at version 10.3. In many cases the use of Derby engine is
> hidden by the application developer.
>
I actually thought "accident" was appropriate here :-) The collision in
Derby versions is "an unexpected and undesirable event, especially one
resulting in damage or harm"[1] arising from the installation of two
applications.
I would assign fault here to the application providers who are not
allowing for other applications using a common shared resource - the
system or user classpath; it's like if they both insisted in being
installed in the same directory.
I would also say that most application providers are aware of this
problem (usually from some unfortunate prior experience) and take
explicit control of the classpath using mechanisms described elsewhere.
Their end-users are unaffected by this scenario.
So whilst there is high impact from your scenario, it applies to
end-users who install multiple applications that do not exhibit
classpath discipline and which happen to use different versions of Derby
client and engine (but not two different engine versions or two
different client versions). And never mind that these users may be
affected by any other libraries these applications use.
Rather than address this ourselves (and just for Derby) by convoluted
code-rewrites and avoidance of third party libraries we should encourage
the application providers avoid this entirely by exhibiting discipline
when using shared resources. We can even point them at something like:
http://classworlds.codehaus.org/
--
Jeremy
[1] http://dictionary.reference.com/search?q=accident
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Daniel John Debrunner <dj...@debrunners.com>.
Jeremy Boynes wrote:
> David W. Van Couvering wrote:
>> Can't you have the situation where common 10.2 and common 10.3 are
>> both included in the classpath (by accident, as Dan brings up)?
>> Wouldn't you end up with order dependencies then?
I feel my scenario keeps being misrepresented by the choice of terms
used to decribe it. Using 'accident' makes it sounds as though it's not
an important problem to deal with, as seen in Jeremy's reponse here:
> To what extent do we need to cater for accidents?
The end user didn't accidently install two applications, they chose to
and didn't realise/know that one used client at version 10.2 and one
used engine at version 10.3. In many cases the use of Derby engine is
hidden by the application developer.
Dan.
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Jeremy Boynes <jb...@apache.org>.
David W. Van Couvering wrote:
>
>
> Jeremy Boynes wrote:
> [snip]
>
>> Daniel John Debrunner wrote:
>>
>>> The issue is that today this is fully supported because the client and
>>> engine do not share code.
>>>
>>> Some of the code sharing approaches regress Derby in this area by not
>>> supporting this, or require class path ordering for it to be supported.
>>>
>>
>> Some of the others support this by defining compatibility contracts
>> and eliminate the need for classpath ordering by not duplicating classes.
>
>
> Can't you have the situation where common 10.2 and common 10.3 are both
> included in the classpath (by accident, as Dan brings up)? Wouldn't you
> end up with order dependencies then?
>
To what extent do we need to cater for accidents? But, yes if you do end
up with both then the (actual) order would determine which one you ended
up with. In many cases the user would not notice due to the API
compatibility between 10.2 and 10.3 (with 10.3 being a pure superset by
the rules defined). The key thing is that there is no overlap in
functionality between jars (so you are not getting 10.2 (containing
client + common) merged with 10.3 (containing engine + common)).
> [snip]
>
>> Let's recharacterize this a little. What we are contemplating with
>> code sharing is extracting common functionality out into a library. By
>> saying that we are not willing to accept any solution where a
>> component depends on a library we are shutting ourselves off from
>> using any external library or any functionality not provided by Derby
>> itself. This dooms us forever to reinvent any functionality that could
>> be provided by other projects.
>>
>> For example, there are libraries out there that support bytecode
>> generation, JMX for management, high-performance concurrency on Java
>> 1.4, regexp processing to support SQL patterns, ... By saying we are
>> not prepared to incorporate them but instead need our own versions
>> that can be morphed for client and server we dramatically reduce the
>> functionality that can be made available to users.
>>
>> So let me ask this: do our users want more functionality faster by
>> allowing the use of libraries, or a completely standalone solution
>> with tight control over the entire implementation?
>
>
> You make a compelling argument here. I already would like to use stuff
> in Jakarta Commons (I haven't brought it up yet, one thing at a time).
> It seems a good Apache Java citizen should make use of what's in Jakarta
> Commons rather than build stuff themselves. And I think Jeremy's right
> that we will run into this same configuration situation with these guys.
>
> Jeremy, how *do* the users of commons avoid accidentally using a version
> they are not compatible with (e.g. a consumer depends on new features
> that aren't available in an older version of the common jar file)?
>
This can be a problem - Java's variant of DLL hell. The solution
generally relies on two factors:
1) upward compatibility between versions - because they are small-ish
components then they tend to do one thing and not suffer from API
scope creep. You just use the latest version and add it to the
classpath once
2) multi-classloader loading mechanisms, which are commonplace in
open source projects, even command line utilities like Ant and Maven
not just the application servers
It is kind of self-fulfilling - when you adopt a modular structure then
you expect to be loading modules with libraries at different version
levels and so use a classloading architecture that fits.
There is also a cultural thing here. For example, suppose I am writing a
database because I happen to think writing database stuff is fun (yes, I
do realize that doing this for fun makes one appear somewhat odd). I
come up with the idea that converting the query plan to executable code
is a neat idea and so need to generate that code. I can implement a code
generator myself, or look for one out there that is already written and
is available under a suitable license. Given my main interest is in
databases, using someone else's code to generate byte code save me a lot
of effort (especially when the class format changes). If it doesn't do
what I want, I could hack my own but it is generally easier to donate
those changes back so that someone else can maintain them. After all, my
interest is in databases not in byte code generation.
Basically, a volunteer community scales better by incorporating each
other's code rather than trying to do it all.
--
Jeremy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Jeremy Boynes wrote:
[snip]
> Daniel John Debrunner wrote:
>
>> The issue is that today this is fully supported because the client and
>> engine do not share code.
>>
>> Some of the code sharing approaches regress Derby in this area by not
>> supporting this, or require class path ordering for it to be supported.
>>
>
> Some of the others support this by defining compatibility contracts
> and eliminate the need for classpath ordering by not duplicating classes.
Can't you have the situation where common 10.2 and common 10.3 are both
included in the classpath (by accident, as Dan brings up)? Wouldn't you
end up with order dependencies then?
[snip]
> Let's recharacterize this a little. What we are contemplating with
> code sharing is extracting common functionality out into a library. By
> saying that we are not willing to accept any solution where a
> component depends on a library we are shutting ourselves off from
> using any external library or any functionality not provided by Derby
> itself. This dooms us forever to reinvent any functionality that could
> be provided by other projects.
>
> For example, there are libraries out there that support bytecode
> generation, JMX for management, high-performance concurrency on Java
> 1.4, regexp processing to support SQL patterns, ... By saying we are
> not prepared to incorporate them but instead need our own versions
> that can be morphed for client and server we dramatically reduce the
> functionality that can be made available to users.
>
> So let me ask this: do our users want more functionality faster by
> allowing the use of libraries, or a completely standalone solution
> with tight control over the entire implementation?
You make a compelling argument here. I already would like to use stuff
in Jakarta Commons (I haven't brought it up yet, one thing at a time).
It seems a good Apache Java citizen should make use of what's in Jakarta
Commons rather than build stuff themselves. And I think Jeremy's right
that we will run into this same configuration situation with these guys.
Jeremy, how *do* the users of commons avoid accidentally using a version
they are not compatible with (e.g. a consumer depends on new features
that aren't available in an older version of the common jar file)?
David
>
> --
> Jeremy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Kathey Marsden <km...@sbcglobal.net>.
Jeremy Boynes wrote:
>
> Let's recharacterize this a little. What we are contemplating with
> code sharing is extracting common functionality out into a library. By
> saying that we are not willing to accept any solution where a
> component depends on a library we are shutting ourselves off from
> using any external library or any functionality not provided by Derby
> itself. This dooms us forever to reinvent any functionality that could
> be provided by other projects.
>
We are not "doomed forever". Requiring a new jar file for new
functionality seems an entirely reasonable thing to me and at that time
we can impose whatever restrictions the community sees fit. Requiring a
new jar file to have the product continue work, is another matter all
together.
We can deprecate product functionality that we don't like for some
reason, but we can't just up and one day take away something that
deployed products depend upon. There are products that are out in the
field now which support and encourage use of Derby and expect the
current separation of jars that we have now. Users should be able to
install applications that embed future versions of Derby and have
things still work as they do today. If they don't it's a *regression*.
I was fascinated by David's priority list: first Product, then User,
then Engineer and think it is really great that we actually seem to have
all permutations of this priority list represented here. All are
important and need advocates, but when you have an installed user
base, sometimes the engineers have to be a little careful and patient.
Of all the approaches presented thus far, I like.
Approach 3) Wait until we have some new set of code that we can use as our first code sharing test case and untangle the existing code later.
This way the big thinking engineers can have fun and do pretty what they want without the installed user base getting in the way.
Kathey
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Jeremy Boynes <jb...@apache.org>.
Daniel John Debrunner wrote:
>
> Derby's client and engine may be in the same classpath and at different
> versions if the JVM is hosting more than one application, or the
> application installers have modified the system/user's classpath to add
> their required jars.
>
In this model, you are assuming that there is a single classpath shared
by all applications run by the system/user. In this mode you cannot have
multiple versions of the client or engine present - you can only have
one version of the client and one version of the engine.
In reality this scenario is useless in anything but the most trivial
installation. As an application can be impacted by any other application
that affects the system/user's classpath reliability requires that they
be isolated from each other. You end up with two modes of operation:
* an application overides the classpath in some script before the JVM
is started to ensure that only the jars it expects are present
* an application runs inside a server that controls the classpath for
it and that server ensures the appropriate libraries are present
> The issue is that today this is fully supported because the client and
> engine do not share code.
>
> Some of the code sharing approaches regress Derby in this area by not
> supporting this, or require class path ordering for it to be supported.
>
Some of the others support this by defining compatibility contracts and
eliminate the need for classpath ordering by not duplicating classes.
> While it is true that multiple class loaders solve the issue, this
> approach is not always possible, I believe, for example, some of the
> major application servers do not support different class loaders for
> different JDBC providers (eg. the Derby client at 10.3 and engine and 10.2).
>
Those application servers also define which versions are supported by
the application server vendor. They also do not support client 10.2 and
client 10.3 which is an equally likely combination.
> Thus the argument really is, are we willing to accept regression in this
> area to gain code sharing, or should the code sharing solution not
> regress Derby?
>
Let's recharacterize this a little. What we are contemplating with code
sharing is extracting common functionality out into a library. By saying
that we are not willing to accept any solution where a component depends
on a library we are shutting ourselves off from using any external
library or any functionality not provided by Derby itself. This dooms us
forever to reinvent any functionality that could be provided by other
projects.
For example, there are libraries out there that support bytecode
generation, JMX for management, high-performance concurrency on Java
1.4, regexp processing to support SQL patterns, ... By saying we are not
prepared to incorporate them but instead need our own versions that can
be morphed for client and server we dramatically reduce the
functionality that can be made available to users.
So let me ask this: do our users want more functionality faster by
allowing the use of libraries, or a completely standalone solution with
tight control over the entire implementation?
--
Jeremy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Kathy Saunders <ka...@mtrad.com>.
Daniel John Debrunner wrote:
>Kathy Saunders wrote:
>
>
>
>
>>Thank you very much for the clear explanation. From a usability
>>perspective, I would vote for approach 2. Requiring a classpath to be
>>in a particular order is always an issue. However, the saving grace is
>>that it sounds like the ordering issue only comes up if you mix versions
>>of the derby.jar and the derbyclient.jar in the same classpath. I don't
>>believe most users put the client and engine in the same classpath
>>(unless there's a new requirement I don't know about), so that
>>definitely helps. Requiring classpath in a specific order can easily
>>lead to complications though, so I'm not in favor of it in general.
>>
>>
>
>Derby's client and engine may be in the same classpath and at different
>versions if the JVM is hosting more than one application, or the
>application installers have modified the system/user's classpath to add
>their required jars.
>
>The issue is that today this is fully supported because the client and
>engine do not share code.
>
>Some of the code sharing approaches regress Derby in this area by not
>supporting this, or require class path ordering for it to be supported.
>
>While it is true that multiple class loaders solve the issue, this
>approach is not always possible, I believe, for example, some of the
>major application servers do not support different class loaders for
>different JDBC providers (eg. the Derby client at 10.3 and engine and 10.2).
>
>Thus the argument really is, are we willing to accept regression in this
>area to gain code sharing, or should the code sharing solution not
>regress Derby?
>
>Dan.
>
>
>
>
>
>
>
>
I don't think we should regress and I don't think we should have
classpath ordering issues (sorry if my rambling mail last time was
misleading). I think that means I would vote for option 2 if we proceed
with code sharing which means no change from a customer
perspective--same number of jars and no ordering issues.
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Ah, I see, yes, you have a point. Well, big swallow, it seems to me
that this is indeed a regression, and a potential usability headache.
Thinking further, this same issue exists for any platform where shared
code can come in through many fronts. I was once told that one
distribution of Linux had three or four different copies of Berkeley
DB. I guess the difference here is that shared libraries are versioned
and you can load a .so of a specific version if you need to, while you
can not do this with JAR files.
Unless someone can convince me otherwise, I am going to have to switch
boats and back Approach 2, the Common Clone approach. I can hold my
architectural purity nose while writing the cloning code.
Thanks,
David
Daniel John Debrunner wrote:
>David W. Van Couvering wrote:
>
>
>
>
>>Note that the main reason these customers want to run with different
>>versions, as I understand it, is so that they can run with a version of
>>the client that matches the server version. If we can guarantee
>>compatibility between client and server versions (so, for instance, you
>>can upgrade the server without having to upgrade the clients, or vise
>>versa), does this requirement goes away?
>>
>>
>
>No.
>
>Client/server compatibility is good, and soft upgrade is good, and all
>help in the area of reducing version compatibility problems.
>
>But I'm not saying anyone *wants* to mix versions, rather that they end
>up mixing versions through circumstances.
>
>I'm assuming worst case, end-user A knows nothing about Java, and is
>using applications from vendor B (client at version 10.2) and vendor C
>(engine at version 10.3).
>
>Dan.
>
>
>
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Daniel John Debrunner <dj...@debrunners.com>.
David W. Van Couvering wrote:
> Note that the main reason these customers want to run with different
> versions, as I understand it, is so that they can run with a version of
> the client that matches the server version. If we can guarantee
> compatibility between client and server versions (so, for instance, you
> can upgrade the server without having to upgrade the clients, or vise
> versa), does this requirement goes away?
No.
Client/server compatibility is good, and soft upgrade is good, and all
help in the area of reducing version compatibility problems.
But I'm not saying anyone *wants* to mix versions, rather that they end
up mixing versions through circumstances.
I'm assuming worst case, end-user A knows nothing about Java, and is
using applications from vendor B (client at version 10.2) and vendor C
(engine at version 10.3).
Dan.
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
I think Dan has really hit the nail on the head. All debate will
quickly stop if we reach consensus that it's not acceptable to break
compatibility (it's not really a regression, we never had this
constraint on ordering before) in this way as part of a minor release
(e.g. 10.1 to 10.2).
I don't know the environment that the customers who depend this on work
in. I don't know how flexible they can be to ordering how jars are
loaded. It's really hard for us to answer this without their input.
Could those of us who have these customers possibly talk to them and
discuss this and understand what their constraints are? I think it's
reasonable to document this for future customers; I don't personally
think it's a major usability issue, the vast majority of users will work
with a single version of Derby in a given VM.
Note that the main reason these customers want to run with different
versions, as I understand it, is so that they can run with a version of
the client that matches the server version. If we can guarantee
compatibility between client and server versions (so, for instance, you
can upgrade the server without having to upgrade the clients, or vise
versa), does this requirement goes away? I know at Sybase we never had
this issue because the TDS protocol allowed you to negotiate the version
of the protocol you were going to run at -- the server knew how to
"downgrade" its protocol support if it needed to. This type of
negotiation and compatibility guarantee is something Rick Hillegas is
already working on...
Thanks,
David
Daniel John Debrunner wrote:
>Kathy Saunders wrote:
>
>
>
>
>>Thank you very much for the clear explanation. From a usability
>>perspective, I would vote for approach 2. Requiring a classpath to be
>>in a particular order is always an issue. However, the saving grace is
>>that it sounds like the ordering issue only comes up if you mix versions
>>of the derby.jar and the derbyclient.jar in the same classpath. I don't
>>believe most users put the client and engine in the same classpath
>>(unless there's a new requirement I don't know about), so that
>>definitely helps. Requiring classpath in a specific order can easily
>>lead to complications though, so I'm not in favor of it in general.
>>
>>
>
>Derby's client and engine may be in the same classpath and at different
>versions if the JVM is hosting more than one application, or the
>application installers have modified the system/user's classpath to add
>their required jars.
>
>The issue is that today this is fully supported because the client and
>engine do not share code.
>
>Some of the code sharing approaches regress Derby in this area by not
>supporting this, or require class path ordering for it to be supported.
>
>While it is true that multiple class loaders solve the issue, this
>approach is not always possible, I believe, for example, some of the
>major application servers do not support different class loaders for
>different JDBC providers (eg. the Derby client at 10.3 and engine and 10.2).
>
>Thus the argument really is, are we willing to accept regression in this
>area to gain code sharing, or should the code sharing solution not
>regress Derby?
>
>Dan.
>
>
>
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Daniel John Debrunner <dj...@debrunners.com>.
Kathy Saunders wrote:
>
> Thank you very much for the clear explanation. From a usability
> perspective, I would vote for approach 2. Requiring a classpath to be
> in a particular order is always an issue. However, the saving grace is
> that it sounds like the ordering issue only comes up if you mix versions
> of the derby.jar and the derbyclient.jar in the same classpath. I don't
> believe most users put the client and engine in the same classpath
> (unless there's a new requirement I don't know about), so that
> definitely helps. Requiring classpath in a specific order can easily
> lead to complications though, so I'm not in favor of it in general.
Derby's client and engine may be in the same classpath and at different
versions if the JVM is hosting more than one application, or the
application installers have modified the system/user's classpath to add
their required jars.
The issue is that today this is fully supported because the client and
engine do not share code.
Some of the code sharing approaches regress Derby in this area by not
supporting this, or require class path ordering for it to be supported.
While it is true that multiple class loaders solve the issue, this
approach is not always possible, I believe, for example, some of the
major application servers do not support different class loaders for
different JDBC providers (eg. the Derby client at 10.3 and engine and 10.2).
Thus the argument really is, are we willing to accept regression in this
area to gain code sharing, or should the code sharing solution not
regress Derby?
Dan.
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Hi, Kathy, I am a bit confused by your response. You seem to be OK with
the classpath ordering issue given that it's an edge case, but you say
you want to go for approach 2. Approach 2 does not require any
classpath ordering.
Also, note that in general classpath ordering isn't important except in
the case where the user is already consciously futzing with the
classpath to mix versions in the same VM.
David
>>
> Hi David,
>
> Thank you very much for the clear explanation. From a usability
> perspective, I would vote for approach 2. Requiring a classpath to be
> in a particular order is always an issue. However, the saving grace
> is that it sounds like the ordering issue only comes up if you mix
> versions of the derby.jar and the derbyclient.jar in the same
> classpath. I don't believe most users put the client and engine in
> the same classpath (unless there's a new requirement I don't know
> about), so that definitely helps. Requiring classpath in a specific
> order can easily lead to complications though, so I'm not in favor of
> it in general.
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Kathy Saunders <ka...@mtrad.com>.
David W. Van Couvering wrote:
> Hi, Kathy, thanks for your email. The timing is actually pretty good,
> I was just talking with Francois trying to understand his concerns
> better.
>
> After quickly describing the two approaches, I'd like to summarize the
> experience/impact of these approaches from the perspectives of the end
> user, the developer/maintainer, and the test developer/runner.
>
> Goal:
> - Reduce code duplication while continuing to support different
> versions of client and embedded driver in the same VM
>
> Approach 1:
> - Create a common package and put all common code there
> (org.apache.derby.common)
> - Use compatibility guidelines to ensure backward compatibility and
> some degree of forward compatibility
> - Common classes are embedded in derby.jar and derby-client.jar
> (based on lots of negative feedback for having a separate jar)
>
> Approach 2:
> - Create a common package and put all common code there
> (org.apache.derby.common)
> - During build process, "clone" all common classes into at least two
> generated packages, one for the engine and one for the network client
> (org.apache.derby.engine.common and org.apache.derby.client.common).
> - This approach avoids having to implement compatiblity
> guidelines/constraints and guarantees, as the engine and client
> continue to be fully isolated
>
> USER EXPERIENCE
>
> Approach 1
> - No new jar files, everything looks the same
> - For the vast percentage of users who don't mix versions in the same
> VM, everything works great
> - For the small percentage of users who mix versions, the order in
> which jar files are loaded. For example, if they want to use
> derby-10.2.jar and derby-client-11.0.jar, then if derby-10.2.jar is
> first in the classpath, they will get an exception saying that the
> client code is incompatible with the common code and that they need to
> re-order their jar files, whereas it will work fine if the order is
> reversed.
>
> Approach 2
> - No new jar files, everything looks the same
> - Ordering of jar files does not matter, everything works fine
>
>
> DEVELOPER EXPERIENCE
>
> Approach 1
> - Pretty much as it is today, nothing surprising
>
> Approach 2
> - When navigating code either during source browsing or debugging,
> they will see the *generated* common code, not the master common code.
> - If a developer is not aware of how things work, or just forgets,
> he/she will try to edit this generated code, and will be confused and
> surprised when his/her changes disappear after a build
> - Stack traces will point to generated code, and the developer will
> have to remember to translate that back to the master version.
> - The generation process must be very careful not to adjust line
> numbers or all stack traces will be off and misleading. This means
> for instance we can't add comments saying "THIS IS GENERATED CODE, DO
> NOT EDIT"
> - We may need to generate more copies if different types of version
> mixing are required (e.g. if the tools.jar and derby.jar need to be at
> different versions)
>
>
> TESTER EXPERIENCE
> Approach 1
> - We would have to build and run compatibility tests to make sure
> that compatible versions run correctly and incompatible ones correcty
> raise the exception
>
> Approach 2
> - No real change, although debugging of tests may be confusing due to
> issues I already listed under developer experience
>
>
> I also am uncomfortable with the "twistiness" of approach 2. There is
> something to be said for a clean architecture and build environment.
> I have seen time and again that a good architecture allows you to
> scale and grow, whereas "twisty" architectures tend to wrap you up and
> tie you down at some point. I think this has to be taken into
> consideration.
>
> My main question is: is it OK to sometimes throw an exception for the
> small set of users who mix versions, and for them to then have to
> rearrange their jar ordering. If the answer is that this is not
> acceptable and would be considered a showstopper regression for some
> part of our user base, then I think we have no choice but to go with
> Approach 2, even if we do risk painting ourselves into an
> architectural corner. Otherwise, it is my strong recommendation to go
> with Approach 1.
>
> Thanks!
>
> David
>
Hi David,
Thank you very much for the clear explanation. From a usability
perspective, I would vote for approach 2. Requiring a classpath to be
in a particular order is always an issue. However, the saving grace is
that it sounds like the ordering issue only comes up if you mix versions
of the derby.jar and the derbyclient.jar in the same classpath. I don't
believe most users put the client and engine in the same classpath
(unless there's a new requirement I don't know about), so that
definitely helps. Requiring classpath in a specific order can easily
lead to complications though, so I'm not in favor of it in general.
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Hi, Kathy, thanks for your email. The timing is actually pretty good, I
was just talking with Francois trying to understand his concerns better.
After quickly describing the two approaches, I'd like to summarize the
experience/impact of these approaches from the perspectives of the end
user, the developer/maintainer, and the test developer/runner.
Goal:
- Reduce code duplication while continuing to support different
versions of client and embedded driver in the same VM
Approach 1:
- Create a common package and put all common code there
(org.apache.derby.common)
- Use compatibility guidelines to ensure backward compatibility and
some degree of forward compatibility
- Common classes are embedded in derby.jar and derby-client.jar (based
on lots of negative feedback for having a separate jar)
Approach 2:
- Create a common package and put all common code there
(org.apache.derby.common)
- During build process, "clone" all common classes into at least two
generated packages, one for the engine and one for the network client
(org.apache.derby.engine.common and org.apache.derby.client.common).
- This approach avoids having to implement compatiblity
guidelines/constraints and guarantees, as the engine and client continue
to be fully isolated
USER EXPERIENCE
Approach 1
- No new jar files, everything looks the same
- For the vast percentage of users who don't mix versions in the same
VM, everything works great
- For the small percentage of users who mix versions, the order in
which jar files are loaded. For example, if they want to use
derby-10.2.jar and derby-client-11.0.jar, then if derby-10.2.jar is
first in the classpath, they will get an exception saying that the
client code is incompatible with the common code and that they need to
re-order their jar files, whereas it will work fine if the order is
reversed.
Approach 2
- No new jar files, everything looks the same
- Ordering of jar files does not matter, everything works fine
DEVELOPER EXPERIENCE
Approach 1
- Pretty much as it is today, nothing surprising
Approach 2
- When navigating code either during source browsing or debugging,
they will see the *generated* common code, not the master common code.
- If a developer is not aware of how things work, or just forgets,
he/she will try to edit this generated code, and will be confused and
surprised when his/her changes disappear after a build
- Stack traces will point to generated code, and the developer will
have to remember to translate that back to the master version.
- The generation process must be very careful not to adjust line
numbers or all stack traces will be off and misleading. This means for
instance we can't add comments saying "THIS IS GENERATED CODE, DO NOT EDIT"
- We may need to generate more copies if different types of version
mixing are required (e.g. if the tools.jar and derby.jar need to be at
different versions)
TESTER EXPERIENCE
Approach 1
- We would have to build and run compatibility tests to make sure that
compatible versions run correctly and incompatible ones correcty raise
the exception
Approach 2
- No real change, although debugging of tests may be confusing due to
issues I already listed under developer experience
I also am uncomfortable with the "twistiness" of approach 2. There is
something to be said for a clean architecture and build environment. I
have seen time and again that a good architecture allows you to scale
and grow, whereas "twisty" architectures tend to wrap you up and tie you
down at some point. I think this has to be taken into consideration.
My main question is: is it OK to sometimes throw an exception for the
small set of users who mix versions, and for them to then have to
rearrange their jar ordering. If the answer is that this is not
acceptable and would be considered a showstopper regression for some
part of our user base, then I think we have no choice but to go with
Approach 2, even if we do risk painting ourselves into an architectural
corner. Otherwise, it is my strong recommendation to go with Approach 1.
Thanks!
David
Kathy Saunders wrote:
> David W. Van Couvering wrote:
>
>> Well, we're at a bit of a standoff here. What I'm looking for is a
>> nail-in-the-coffin data point that would move us in one direction or
>> the other.
>
>
> I've been following this discussion with some interest as my
> background is technical support. In fact, I supported Cloudscape for
> the first 4 years (from release 1.0 of Cloudscape). My experience is
> that developers (particularly Java developers) really liked Cloudscape
> (now Derby) because it was so easy to use and deploy. And, I found
> historically that one of the most common issues to come up were
> classpath issues (in particular we got in trouble a few times when we
> introduced something that caused the order of the jar files to be
> important). You should note that I'm not, and never have been, a
> Derby developer, so I don't claim to be an expert on what's correct
> and best from a development perspective.
>
> I'm a bit concerned because I see a lot of discussion about what is
> good from a derby development perspective, but not so much how these
> changes may affect users of Derby. Although some Derby users have
> complex applications (like application servers), many are implementing
> much more simple solutions.
> Having said that, I'm a bit lost in what is being proposed from the
> user/functional perspective. David, as soon as you have a more
> concrete proposal (may not be the time yet), can you post that
> information? Could you provide information on what the users of
> Derby will have to do with this change (how would our documentation
> need to be changed) and maybe footprint, performance, etc. impacts vs.
> the benefits from making this change. I'd like to be able to provide
> my input from a usability/documentation perspective.
>
> In addition, I work on Derby now in the testing area, so I'd also like
> to understand the implications for what additional testing might need
> to be done. If we create more jar files, is there more testing
> requirements for different combinations?
>
> Thanks,
> Kathy
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Jeremy Boynes <jb...@apache.org>.
Kathy Saunders wrote:
> Jeremy Boynes wrote:
>
>> I would argue that we are actually making life easier for people
>> implementing simple solutions. To me, a simple environment does not
>> need to cater for multiple versions being concurrently loaded or for
>> multi-classloader operation; it also means being able to select just
>> the functionality you need without having to worry about which jar
>> file a class may have come from.
>>
>> I think for that environment, just adding the component jars to the
>> classpath (without any concern for ordering) is reasonable.
>>
>> To make things even simpler, it has also been proposed that we bundle
>> all components together into one jar (containing everything, client
>> and server). This gives you less flexibility and a larger footprint
>> but is a really simple solution.
>>
>> <snip/>
>>
> I have to say that I don't see how adding more jar files to figure out
> whether you need to deploy and add to your classpath makes things easier
> for the simple case.
For the simplest case, the proposal was to package everything into one
jar. What you don't get is minimal footprint and the ability to support
multiple versions in the same JVM, but then those aren't really
high-priority features for the simple case.
> And, so far, I don't see what our users would
> reasonably be able to pick and choose--what would they be able to leave
> out of our database engine other than how the jar files are already
> separated (embedded, network server, tools...)?
>
> Unless I'm missing
> something, David is currently working on internationalizing error
> messages. Would it really make sense to tell someone they may not need
> that functionality? Will they be able to get error messages for network
> server without having those classes in their classpath? I could imagine
> scenarios in the future where there may be significant pieces of
> functionality that we would want to separate because not everyone
> wants/needs that functionality and it would significantly add to
> footprint, but I can't think of anything in that category that currently
> exists other than what we already have. For example, we do have a
> separate jar file for tools.
>
> Footprint is an interesting argument, but will we really see any
> significant differences there yet? Strictly looking at this from a
> usability perspective, I still believe that having a common.jar file
> which has no real meaning to a Derby user (since I believe you'll always
> need in the network server case at this point), so why have them keep
> track of yet another jar file?
>
> If we do have a separate jar file for these classes, I believe that it
> should only be one at this point and classpath order should not matter.
> Again, I'm not saying there may not be a need for more jar files in the
> future. I'm only looking at what I believe is proposed right now.
>
With the current setup, perhaps not - at this time we are only talking
about a few common classes. However, we are talking about an
architecture and philosophy that relies on duplication and isolation
from any libraries, internal or external.
David has raised this issue in conjunction with messages and some common
network functionality. I have raised it previously in conjunction with
the implementation of our DataSources and user-visible API. That this
keeps coming up indicates that there is something here we need to address.
--
Jeremy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Kathy Saunders <ka...@mtrad.com>.
Jeremy Boynes wrote:
> Kathy Saunders wrote:
>
>>
>
> <snip/>
>
>>
>> I'm a bit concerned because I see a lot of discussion about what is
>> good from a derby development perspective, but not so much how these
>> changes may affect users of Derby. Although some Derby users have
>> complex applications (like application servers), many are
>> implementing much more simple solutions.
>
>
> I would argue that we are actually making life easier for people
> implementing simple solutions. To me, a simple environment does not
> need to cater for multiple versions being concurrently loaded or for
> multi-classloader operation; it also means being able to select just
> the functionality you need without having to worry about which jar
> file a class may have come from.
>
> I think for that environment, just adding the component jars to the
> classpath (without any concern for ordering) is reasonable.
>
> To make things even simpler, it has also been proposed that we bundle
> all components together into one jar (containing everything, client
> and server). This gives you less flexibility and a larger footprint
> but is a really simple solution.
>
> <snip/>
>
I have to say that I don't see how adding more jar files to figure out
whether you need to deploy and add to your classpath makes things easier
for the simple case. And, so far, I don't see what our users would
reasonably be able to pick and choose--what would they be able to leave
out of our database engine other than how the jar files are already
separated (embedded, network server, tools...)? Unless I'm missing
something, David is currently working on internationalizing error
messages. Would it really make sense to tell someone they may not need
that functionality? Will they be able to get error messages for network
server without having those classes in their classpath? I could imagine
scenarios in the future where there may be significant pieces of
functionality that we would want to separate because not everyone
wants/needs that functionality and it would significantly add to
footprint, but I can't think of anything in that category that currently
exists other than what we already have. For example, we do have a
separate jar file for tools.
Footprint is an interesting argument, but will we really see any
significant differences there yet? Strictly looking at this from a
usability perspective, I still believe that having a common.jar file
which has no real meaning to a Derby user (since I believe you'll always
need in the network server case at this point), so why have them keep
track of yet another jar file?
If we do have a separate jar file for these classes, I believe that it
should only be one at this point and classpath order should not matter.
Again, I'm not saying there may not be a need for more jar files in the
future. I'm only looking at what I believe is proposed right now.
>>
>> In addition, I work on Derby now in the testing area, so I'd also
>> like to understand the implications for what additional testing might
>> need to be done. If we create more jar files, is there more testing
>> requirements for different combinations?
>>
>
> I don't think there are any more combinations - in fact probably less
> as you would not need to test all possible classpath orderings. We are
> dealing with the same amount of code, just modularizing its structure.
>
> By modularizing the build we also allow for in-depth testing on each
> individual component in isolation. With a clear definition of the API
> contract for each component and testing (unit, functional,
> compatibility) of that contract we can perform more thorough testing
> on each one before integrating into a whole. Integration and system
> testing can focus on the interfaces between components rather than on
> the entire black box.
>
> Add in too that modularization makes it easier for users and
> developers to come up to speed with the design and implementation of
> that component. More eyes on the code with comprehensible component
> leads to better review and higher quality.
>
> Finally, you can see this pattern at work with many open source
> projects: a common core and then a very modular structure that allows
> people to participate at the component level. Examples of projects
> with this type of structure are:
> * Apache HTTPD + mod_*
> * Apache Maven + plugins
> * Eclipse + plugins
> * Apache Jakarta/Tomcat + Commons
> and many more.
>
> --
> Jeremy
>
>
>
>
Thanks for your perspective on the testing issue.
Kathy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Jeremy Boynes <jb...@apache.org>.
Kathy Saunders wrote:
>
<snip/>
>
> I'm a bit concerned because I see a lot of discussion about what is good
> from a derby development perspective, but not so much how these changes
> may affect users of Derby. Although some Derby users have complex
> applications (like application servers), many are implementing much more
> simple solutions.
I would argue that we are actually making life easier for people
implementing simple solutions. To me, a simple environment does not need
to cater for multiple versions being concurrently loaded or for
multi-classloader operation; it also means being able to select just the
functionality you need without having to worry about which jar file a
class may have come from.
I think for that environment, just adding the component jars to the
classpath (without any concern for ordering) is reasonable.
To make things even simpler, it has also been proposed that we bundle
all components together into one jar (containing everything, client and
server). This gives you less flexibility and a larger footprint but is a
really simple solution.
<snip/>
>
> In addition, I work on Derby now in the testing area, so I'd also like
> to understand the implications for what additional testing might need to
> be done. If we create more jar files, is there more testing
> requirements for different combinations?
>
I don't think there are any more combinations - in fact probably less as
you would not need to test all possible classpath orderings. We are
dealing with the same amount of code, just modularizing its structure.
By modularizing the build we also allow for in-depth testing on each
individual component in isolation. With a clear definition of the API
contract for each component and testing (unit, functional,
compatibility) of that contract we can perform more thorough testing on
each one before integrating into a whole. Integration and system testing
can focus on the interfaces between components rather than on the entire
black box.
Add in too that modularization makes it easier for users and developers
to come up to speed with the design and implementation of that
component. More eyes on the code with comprehensible component leads to
better review and higher quality.
Finally, you can see this pattern at work with many open source
projects: a common core and then a very modular structure that allows
people to participate at the component level. Examples of projects with
this type of structure are:
* Apache HTTPD + mod_*
* Apache Maven + plugins
* Eclipse + plugins
* Apache Jakarta/Tomcat + Commons
and many more.
--
Jeremy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Kathy Saunders <ka...@mtrad.com>.
David W. Van Couvering wrote:
> Well, we're at a bit of a standoff here. What I'm looking for is a
> nail-in-the-coffin data point that would move us in one direction or
> the other.
I've been following this discussion with some interest as my background
is technical support. In fact, I supported Cloudscape for the first 4
years (from release 1.0 of Cloudscape). My experience is that
developers (particularly Java developers) really liked Cloudscape (now
Derby) because it was so easy to use and deploy. And, I found
historically that one of the most common issues to come up were
classpath issues (in particular we got in trouble a few times when we
introduced something that caused the order of the jar files to be
important). You should note that I'm not, and never have been, a Derby
developer, so I don't claim to be an expert on what's correct and best
from a development perspective.
I'm a bit concerned because I see a lot of discussion about what is good
from a derby development perspective, but not so much how these changes
may affect users of Derby. Although some Derby users have complex
applications (like application servers), many are implementing much more
simple solutions.
Having said that, I'm a bit lost in what is being proposed from the
user/functional perspective. David, as soon as you have a more concrete
proposal (may not be the time yet), can you post that information?
Could you provide information on what the users of Derby will have to
do with this change (how would our documentation need to be changed) and
maybe footprint, performance, etc. impacts vs. the benefits from making
this change. I'd like to be able to provide my input from a
usability/documentation perspective.
In addition, I work on Derby now in the testing area, so I'd also like
to understand the implications for what additional testing might need to
be done. If we create more jar files, is there more testing
requirements for different combinations?
Thanks,
Kathy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Well, we're at a bit of a standoff here. What I'm looking for is a
nail-in-the-coffin data point that would move us in one direction or
the other.
Here are my feelings on the issue. Although the cloning approach
helps us avoid having to maintain compatibility, my gut feel is that
it is a hack and it will come back to bite us in the future. I also
think it's not extensible -- it's a particular solution to a particular
problem of sharing code between client and engine. What if we want to
share code in other ways in the future? We'd have to keep trying to
extend this model, making more and more copies. I also don't like the
debugging model of the cloning approach, where what you're looking at
and debugging is not the "real" code but "shadow" code. It just overall
feels like a bandaid type of approach; when I've gone for these types
of solutions in the past I've almost always regretted it (or somebody
else who had to deal with it four years later).
The module principle as Jeremy describes it, is a well-understood approach,
it's already in use within Jakarta commons, and has been used to great
effect in Solaris. It's extensible, and it has been shown to be effective
in allowing a project to be more flexible and scale up better as a project
grows in size and complexity. It's also in line with the approach we
already have in place with the engine code, with the Monitor and Service
pattern.
That said, I recognize that it requires a certain level of discipline
around version compatibility to make this scheme work. For this reason,
I think we should be careful how many "modules" we create.
I don't think it helps to take every subsystem and turn it into a module
unless there is a need for it to be used in multiple places. For now I
would like to propose we stick with just the "common module" as the only
one; if necessary we can break this into smaller pieces in the future,
for example splitting out the DRDA code, but so far I don't see a need
for that.
So, unless someone can show me a nail that puts the module approach in
the grave, this is the approach I'd like to take.
Thanks,
David
Jeremy Boynes wrote:
> Kathey Marsden wrote:
>
>> Jeremy Boynes wrote:
>>
>>> In that context, components that come to mind are engine, client, net,
>>> tools and common and external dependencies for consideration include
>>> logging, configuration and thread management.
>>
>>
>> What type of version interoperability do you propose for these
>> components and how would that be managed?
>>
>
> I think the same as what we have proposed earlier on this thread.
>
> The public interface for any component can only be extended at the
> same major.minor level. So 10.3 may have stuff that 10.1 does not have
> but a consumer built against 10.1 is guaranteed to work with any 10.X
> implementation provided X >= 1. Any incompatible changes require a new
> major version and no compatibility is implied.
>
> Implementation can provide version information about themselves and
> about the interfaces they support; this might be through the Package
> info or through some custom mechanisn. Consumers can query that and
> adapt as appropriate.
>
> This applies within a single classloader; if a user needs a more
> esoteric scheme within a single JVM then they need to isolate the
> versions from each other by using different classloaders. I do not
> consider this an unrealistic expectation given this is a non-trivial
> case in the first place.
>
> One thing I think we should add is separation between in-VM client
> (the JDBC implementation) and the engine implementation. This should
> allow a program in one classloader to access an engine in another;
> this may involve data copies which would make it less performant than
> an in-classloader configuration but it will be better than using a
> network connection which is the only option now.
>
> Across JVMs we support up-down compatibility across a wider spread of
> versions but that can be done with the wire protocol and does not need
> class-level compatibility.
>
> --
> Jeremy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Jeremy Boynes <jb...@apache.org>.
David W. Van Couvering wrote:
> Well, this is an interesting wrench you have thrown in here, Jeremy.
Wrenches are a specialty of the house :-)
<snip/>
>
> The question is, what are the benefits of the approach, and do they
> merit the rigor that we would have to follow to make sure things don't
> break? I have to think about this myself. What do others think?
>
The big benefit I see is not technical but social. AIUI Jakarta Commons
evolved out of the recognition that large projects (specifically Tomcat)
provided many libraries that would be valuable for other projects to
use. By separating them out it made is possible for people to contribute
in a meaningful way without needing to grok the whole of a large project
in its entirity.
Commons itself went through a process of evolution, both in terms of
stability and in terms of modularity (decoupling the libraries from each
other). It now provides a useful resource for open source and
commercial projects alike.
In many cases the rigour is not there to the same extent as in typical
closed source implementations. Instead, the use of classloader
hierarchies allows modules to be loaded into separate domains with
multiple versions present in memory concurrently.
This is analogous to the J2EE model where the application server needs
to be able to load different class versions for the different
applications it is hosting. However, it is also used by frameworks like
Spring, Eclipse, Plexus (Maven), Ant etc. in more specialized niches
than a general purpose application server.
> P.S. Jeremy I don't understand why separate classloaders are needed to
> work with different versions in the same VM, if we meet compatibility
> rules.
>
I was thinking of the case where they do not meet compatibility
requirements for co-location, for example a 11.X client talking to a
10.X engine. The in-VM, cross-classloader mode would allow these to work
together without having to use a network connection.
For most co-location situations though I would anticipate that the
compatibility rules combined with the clear separation of classes into
distinct modules would allow the user to assemble an unordered list of
jars that could be loaded by a single classloader.
--
Jeremy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Well, this is an interesting wrench you have thrown in here, Jeremy. I
personally have always been a believer in the scheme you suggest, with
modules of code being built independently rather than a single
monolithic code base. It allows for greater flexibility, but it does
require a lot more discipline in terms of version compatibility. This
is the way Solaris works, for instance, and they have been quite
successful at it. But you should see the level of rigor they apply here
at Sun to make sure that works. Before this level of rigor was applied,
Solaris was a nasty ball of yarn indeed.
Now, we do have a guide for us in this approach. Jakarta Commons is a
collection of independent modules that are released independently, and
which must meet strong compatibility rules. Take a look at their
versioning guidelines here:
http://jakarta.apache.org/commons/releases/versioning.html
The question is, what are the benefits of the approach, and do they
merit the rigor that we would have to follow to make sure things don't
break? I have to think about this myself. What do others think?
P.S. Jeremy I don't understand why separate classloaders are needed to
work with different versions in the same VM, if we meet compatibility rules.
David
Jeremy Boynes wrote:
> Kathey Marsden wrote:
>
>> Jeremy Boynes wrote:
>>
>>> In that context, components that come to mind are engine, client, net,
>>> tools and common and external dependencies for consideration include
>>> logging, configuration and thread management.
>>
>>
>> What type of version interoperability do you propose for these
>> components and how would that be managed?
>>
>
> I think the same as what we have proposed earlier on this thread.
>
> The public interface for any component can only be extended at the
> same major.minor level. So 10.3 may have stuff that 10.1 does not have
> but a consumer built against 10.1 is guaranteed to work with any 10.X
> implementation provided X >= 1. Any incompatible changes require a new
> major version and no compatibility is implied.
>
> Implementation can provide version information about themselves and
> about the interfaces they support; this might be through the Package
> info or through some custom mechanisn. Consumers can query that and
> adapt as appropriate.
>
> This applies within a single classloader; if a user needs a more
> esoteric scheme within a single JVM then they need to isolate the
> versions from each other by using different classloaders. I do not
> consider this an unrealistic expectation given this is a non-trivial
> case in the first place.
>
> One thing I think we should add is separation between in-VM client
> (the JDBC implementation) and the engine implementation. This should
> allow a program in one classloader to access an engine in another;
> this may involve data copies which would make it less performant than
> an in-classloader configuration but it will be better than using a
> network connection which is the only option now.
>
> Across JVMs we support up-down compatibility across a wider spread of
> versions but that can be done with the wire protocol and does not need
> class-level compatibility.
>
> --
> Jeremy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Jeremy Boynes <jb...@apache.org>.
Kathey Marsden wrote:
> Jeremy Boynes wrote:
>
>>In that context, components that come to mind are engine, client, net,
>>tools and common and external dependencies for consideration include
>>logging, configuration and thread management.
>
> What type of version interoperability do you propose for these
> components and how would that be managed?
>
I think the same as what we have proposed earlier on this thread.
The public interface for any component can only be extended at the same
major.minor level. So 10.3 may have stuff that 10.1 does not have but a
consumer built against 10.1 is guaranteed to work with any 10.X
implementation provided X >= 1. Any incompatible changes require a new
major version and no compatibility is implied.
Implementation can provide version information about themselves and
about the interfaces they support; this might be through the Package
info or through some custom mechanisn. Consumers can query that and
adapt as appropriate.
This applies within a single classloader; if a user needs a more
esoteric scheme within a single JVM then they need to isolate the
versions from each other by using different classloaders. I do not
consider this an unrealistic expectation given this is a non-trivial
case in the first place.
One thing I think we should add is separation between in-VM client (the
JDBC implementation) and the engine implementation. This should allow a
program in one classloader to access an engine in another; this may
involve data copies which would make it less performant than an
in-classloader configuration but it will be better than using a network
connection which is the only option now.
Across JVMs we support up-down compatibility across a wider spread of
versions but that can be done with the wire protocol and does not need
class-level compatibility.
--
Jeremy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Kathey Marsden <km...@sbcglobal.net>.
Jeremy Boynes wrote:
> In that context, components that come to mind are engine, client, net,
> tools and common and external dependencies for consideration include
> logging, configuration and thread management.
>
What type of version interoperability do you propose for these
components and how would that be managed?