You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Andrew McIntyre <mc...@gmail.com> on 2005/09/16 22:20:05 UTC
Re: Modular build, was: VOTE: Approach for sharing code
On 9/15/05, Kathey Marsden <km...@sbcglobal.net> wrote:
> Jeremy Boynes wrote:
>
> >
> > This dooms us forever to reinvent any functionality that could
> > be provided by other projects.
> >
> We are not "doomed forever". Requiring a new jar file for new
> functionality seems an entirely reasonable thing to me and at that time
> we can impose whatever restrictions the community sees fit. Requiring a
> new jar file to have the product continue work, is another matter all
> together.
+1!
It looks like we've got a really intractable situation here. There are
those against source/binary modification because of the inherent
problems with maintenance and debugging, but you can't package the
same classes in multiple jars because of signing/sealing and
classloader issues, and if you split the classes out into a new jar
then you've got usability and deployment issues. So what do you do?
I'm wondering if this is really the functionality to be considering
all of this for. Two or three small classes related to localization.
Is it really worth the trouble? Are we really going to make a common
jar file with two classes, without which you don't get error messages?
Really?
And remember, this isn't about reusing someone else's code or
libraries. I don't think anyone is going to be against reusing other
project's code or libraries in order to promote cross-project goodwill
and general open source happiness. We're talking about copying two of
our own classes from one part of the tree to the other.
I think we can all tackle the code-sharing problem with some new
functionality later. For which, by the way, I think a good candidate
would be integrating Lucene for full text indexing as a good trial for
both code reuse from other projects and code sharing among the
different parts of Derby. And personally, I'd like to see David be
able to finish localizing the error messages sometime this decade. :-)
andrew
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "M. Mortazavi" <Ma...@Sun.COM>.
Daniel John Debrunner wrote:
>Jeremy Boynes wrote:
>
>
>
>>Think of the carnage if it was java_14208_b13.sql_300.Connection
>>
>>
>
>It's actually instructive to look how Java solves this:
>
>- The interface is kept upwards compatible, by following rules such as
>only new methods/fields etc.
>
>Then to look at how a consumer, Derby, deals with the fact of multiple
>versions of the interface:
>
>- Derby only provides and uses the functionality to match the version of
>the interface (java.sql.Connection) loaded, determined at runtime.
>
>I'd thought this was a workable direction being proposed about six
>thousand message ago, upwards compatible apis and the ability for a
>consumer to handle a lower version. Not sure what derailed it.
>
>Dan.
>
>
I'm not sure if this will help or hinder with your considerations but . . .
Compatible type evolution is described in the Java Serialization Spec.
The reason it is described there is because the problem arises when
Class A is compiled against a particular version of Class B but then at
run-time an instance of A encounters (through class loading upon
de-serialization, for example) an instance of some other version of
Class B.
These type compatibility rules can be found here:
http://java.sun.com/j2se/1.4.2/docs/guide/serialization/spec/version.html#wp9419
M.
--
M. Mortazavi
+1 408 276 7205 (office)
+1 408 421 4093 (mobile)
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Jeremy Boynes <jb...@apache.org>.
David W. Van Couvering wrote:
>
> You say Java does this -- is this documented somewhere? I'd love to have
> that as a reference.
>
Do you mean the JVM impl or the general philosophy? I think the
versioning model is defined in the language spec but would have to go
digging.
> First of all, Kathey was concerned we wouldn't be able to follow this
> kind of discipline.
>
> http://mail-archives.apache.org/mod_mbox/db-derby-dev/200509.mbox/%3c43207ABC.8080000@sbcglobal.net%3e
>
It is on us and our users to police ourselves.
>
> But I'll also show where I saw this get tripped up:
>
> > * a consumer expecting X.Y will work with X.Y' when Y' < Y by version
> > detection and degraded functionality (Y' level). If Y level function
> > is required then it will gracefully die (able to detect rather
> > than AbstractMethodError)
>
>
> In my original proposal:
>
> * compatibility will be strongly encouraged but not guaranteed
> against previous minor versions (e.g. a 10.2 consumer works
> with 10.1 common classes, but a 10.3 consumer has a hard
> dependency on new methods, it can not work with 10.2
> common classes).
>
> Perhaps I remember incorrectly, but I remember us (or enough of us)
> generally agreeing that gracefully dying when Y level function was
> required was not acceptable, as this was a regression of existing
> behavior. This was the "nail in the coffin" for my original proposal.
>
> http://mail-archives.apache.org/mod_mbox/db-derby-dev/200509.mbox/%3c43289F58.6000707@mtrad.com%3e
>
> http://mail-archives.apache.org/mod_mbox/db-derby-dev/200509.mbox/%3c4328AC1F.6000504@sun.com%3e
>
>
> Unless you can address these issues I think we're back at square one...
>
I'd put this as part of the discipline. The Y consumer, as the newer
version, knows about earlier Y' versions and hence can be coded to
accept the downrev dependency - it can downgrade itself, it can offer
different behaviour, it can implement the functionality itself, it can
die gracefully, or we can just make it X+1.
Bear in mind there are no guarantees. For example, someone else could
have modified Y' themselves without any regard to compatibility and just
stuck it on the classpath.
Also, let's not forget that this is only in the case where we have
multiple applications with poor classpath discipline installed
concurrently. (And I haven't forgotten the request for a writeup on
stuff like ClassWorlds).
--
Jeremy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Well, that works for me!
David
Daniel John Debrunner wrote:
> David W. Van Couvering wrote:
>
>
>
>>In my original proposal:
>>
>> * compatibility will be strongly encouraged but not guaranteed
>> against previous minor versions (e.g. a 10.2 consumer works
>> with 10.1 common classes, but a 10.3 consumer has a hard
>> dependency on new methods, it can not work with 10.2
>> common classes).
>>
>>Perhaps I remember incorrectly, but I remember us (or enough of us)
>>generally agreeing that gracefully dying when Y level function was
>>required was not acceptable, as this was a regression of existing
>>behavior. This was the "nail in the coffin" for my original proposal.
>
>
> Maybe I'm an optimist, but I think that a consumer of common code can
> always be coded to keep running (in a reduced mode) when faced with an
> older version of the common code. Thus I think this approach can be made
> to work, just start with the mindset that dying is unacceptable, rather
> than inevitable.
>
> Dan.
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Daniel John Debrunner <dj...@debrunners.com>.
David W. Van Couvering wrote:
> In my original proposal:
>
> * compatibility will be strongly encouraged but not guaranteed
> against previous minor versions (e.g. a 10.2 consumer works
> with 10.1 common classes, but a 10.3 consumer has a hard
> dependency on new methods, it can not work with 10.2
> common classes).
>
> Perhaps I remember incorrectly, but I remember us (or enough of us)
> generally agreeing that gracefully dying when Y level function was
> required was not acceptable, as this was a regression of existing
> behavior. This was the "nail in the coffin" for my original proposal.
Maybe I'm an optimist, but I think that a consumer of common code can
always be coded to keep running (in a reduced mode) when faced with an
older version of the common code. Thus I think this approach can be made
to work, just start with the mindset that dying is unacceptable, rather
than inevitable.
Dan.
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Thanks for writing up a proposed solution, I was getting a little tired
of putting up targets for others to shoot down.
I agree with this wholeheartedly this was what I have wanted to do
(although perhaps not as well communicated and there are some details
here that are new and which seem really good/interesting).
You say Java does this -- is this documented somewhere? I'd love to have
that as a reference.
First of all, Kathey was concerned we wouldn't be able to follow this
kind of discipline.
http://mail-archives.apache.org/mod_mbox/db-derby-dev/200509.mbox/%3c43207ABC.8080000@sbcglobal.net%3e
But I'll also show where I saw this get tripped up:
> * a consumer expecting X.Y will work with X.Y' when Y' < Y by version
> detection and degraded functionality (Y' level). If Y level function
> is required then it will gracefully die (able to detect rather
> than AbstractMethodError)
In my original proposal:
* compatibility will be strongly encouraged but not guaranteed
against previous minor versions (e.g. a 10.2 consumer works
with 10.1 common classes, but a 10.3 consumer has a hard
dependency on new methods, it can not work with 10.2
common classes).
Perhaps I remember incorrectly, but I remember us (or enough of us)
generally agreeing that gracefully dying when Y level function was
required was not acceptable, as this was a regression of existing
behavior. This was the "nail in the coffin" for my original proposal.
http://mail-archives.apache.org/mod_mbox/db-derby-dev/200509.mbox/%3c43289F58.6000707@mtrad.com%3e
http://mail-archives.apache.org/mod_mbox/db-derby-dev/200509.mbox/%3c4328AC1F.6000504@sun.com%3e
Unless you can address these issues I think we're back at square one...
David
Further asides below:
> As a quick braindump:
> * there are rules that restrict changes to the interface going forward
> * Sun (et al) stick to those rules even when it is painful
> * no class/method is removed without being deprecated first
> * the JVM supports version skew a little by not requiring an
> implementation to implement all methods in a interface (at runtime)
That's nice to know, I didn't know that
> * the interfaces often provide a mechanism to determine what features of
> an API the implementation implements
> * frameworks can utilize multiple classloaders to load multiple versions
> of a class into the VM - tools have developed to simplify this
>
I don't know much about such frameworks and tools, but I don't think
this solves Dan's problem of "accidentally" loading multiple versions.
> I think we can support a scenario where:
> * a comsumer expecting version X.Y will work with any implementation
> X.Y' when Y' >= Y
> * X.Y defines the interface version, i.e. X.Y.Z works with X.Y.Z' for
> all Z'
> * X defines the compatibility level i.e. X and X' are not guaranteed
> to work together (although we will try to ensure they do)
>
> Basic rules:
> * at the Z level, no API change is allowed, just implementation changes
> * at the Y level, additive API changes are allowed but must be
> accompanied by support in the versioning mechanism so that they can be
> detected. Things can be deprecated
> * at the Z level, incompatible changes are allowed. Items deprecated in
> version X can be removed in version X+2 (implicitly (X+2).0.0).
>
> This all applies to operation in a single classloader. Where multiple
> classloaders are involved (inside one VM or in multiple VMs) a different
> set of versioning behaviours applies to the wire protocol.
>
> --
> Jeremy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Sorry, pilot error, I needed to log in.
David W. Van Couvering wrote:
> Hm, the front page appears to be an "Immutable Page." Is this perhaps
> because someone started editing it and didn't commit/back out their
> changes?
>
> David
>
> Daniel John Debrunner wrote:
>
>> David W. Van Couvering wrote:
>>
>>
>>> Hi, all. I haven't gotten any more comments on the proposal at
>>> http://wiki.apache.org/db-derby/ModuleVersioningGuidelines, so I'm going
>>> to update this page based on the comments I have received so far and
>>> then we'll try for another vote.
>>
>>
>>
>> Is there any reason this (and other wiki pages added by folks) are not
>> linked into the front page of the wiki? Is this just standard wiki
>> practice or are we just missing making the pages more visible?
>>
>> It seems like unless you know of the page then it's hard to join in any
>> discussion about it, the automated wiki diffs sent to the committer list
>> are basically unreadable.
>>
>>
>>
>> Dan.
>>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Hm, the front page appears to be an "Immutable Page." Is this perhaps
because someone started editing it and didn't commit/back out their changes?
David
Daniel John Debrunner wrote:
> David W. Van Couvering wrote:
>
>
>>Hi, all. I haven't gotten any more comments on the proposal at
>>http://wiki.apache.org/db-derby/ModuleVersioningGuidelines, so I'm going
>>to update this page based on the comments I have received so far and
>>then we'll try for another vote.
>
>
> Is there any reason this (and other wiki pages added by folks) are not
> linked into the front page of the wiki? Is this just standard wiki
> practice or are we just missing making the pages more visible?
>
> It seems like unless you know of the page then it's hard to join in any
> discussion about it, the automated wiki diffs sent to the committer list
> are basically unreadable.
>
>
>
> Dan.
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Kathey Marsden <km...@sbcglobal.net>.
David W. Van Couvering wrote:
> I can gladly add this to the front page, and perhaps we can make this
> a general policy. I would like to suggest that we have a sub-page off
> the front page for proposals that are being discussed/reviewed on the
> Wiki site.
>
That sounds like a good idea.
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
I can gladly add this to the front page, and perhaps we can make this a
general policy. I would like to suggest that we have a sub-page off the
front page for proposals that are being discussed/reviewed on the Wiki site.
David
Daniel John Debrunner wrote:
> David W. Van Couvering wrote:
>
>
>>Hi, all. I haven't gotten any more comments on the proposal at
>>http://wiki.apache.org/db-derby/ModuleVersioningGuidelines, so I'm going
>>to update this page based on the comments I have received so far and
>>then we'll try for another vote.
>
>
> Is there any reason this (and other wiki pages added by folks) are not
> linked into the front page of the wiki? Is this just standard wiki
> practice or are we just missing making the pages more visible?
>
> It seems like unless you know of the page then it's hard to join in any
> discussion about it, the automated wiki diffs sent to the committer list
> are basically unreadable.
>
>
>
> Dan.
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Daniel John Debrunner <dj...@debrunners.com>.
David W. Van Couvering wrote:
> Hi, all. I haven't gotten any more comments on the proposal at
> http://wiki.apache.org/db-derby/ModuleVersioningGuidelines, so I'm going
> to update this page based on the comments I have received so far and
> then we'll try for another vote.
Is there any reason this (and other wiki pages added by folks) are not
linked into the front page of the wiki? Is this just standard wiki
practice or are we just missing making the pages more visible?
It seems like unless you know of the page then it's hard to join in any
discussion about it, the automated wiki diffs sent to the committer list
are basically unreadable.
Dan.
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Thanks, Kathey, I missed that, I'll add that to this page.
David
Kathey Marsden wrote:
> David W. Van Couvering wrote:
>
>
>>Hi, all. I haven't gotten any more comments on the proposal at
>>http://wiki.apache.org/db-derby/ModuleVersioningGuidelines, so I'm
>>going to update this page based on the comments I have received so far
>>and then we'll try for another vote.
>>
>
>
> David will there be another document that describes the packaging and
> any external impact of this change or will it be included in this one?
>
> Thanks
>
> Kathey
>
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Kathey Marsden <km...@sbcglobal.net>.
David W. Van Couvering wrote:
> Hi, all. I haven't gotten any more comments on the proposal at
> http://wiki.apache.org/db-derby/ModuleVersioningGuidelines, so I'm
> going to update this page based on the comments I have received so far
> and then we'll try for another vote.
>
David will there be another document that describes the packaging and
any external impact of this change or will it be included in this one?
Thanks
Kathey
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Hi, all. I haven't gotten any more comments on the proposal at
http://wiki.apache.org/db-derby/ModuleVersioningGuidelines, so I'm going
to update this page based on the comments I have received so far and
then we'll try for another vote.
Thanks,
David
David W. Van Couvering wrote:
> So, I'll wait for another overnight (over-day) period, but it sounds
> like we may have a level of consensus here (fingers crossed).
>
> What I propose to do is take Jeremy's quick brain dump (thanks again for
> doing this Jeremy) and turn it into a more complete policy around this.
> Hopefully I'll capture the essence of this correctly, but I'm sure
> you'll correct me if I miss something :)
>
> I'll put this on the Derby Wiki, to make it more readable/reviewable.
>
> Thanks all,
>
> David
>
> Jeremy Boynes wrote:
>
>> Daniel John Debrunner wrote:
>>
>>> Jeremy Boynes wrote:
>>>
>>>
>>>> Think of the carnage if it was java_14208_b13.sql_300.Connection
>>>
>>>
>>>
>>>
>>> It's actually instructive to look how Java solves this:
>>>
>>> - The interface is kept upwards compatible, by following rules such as
>>> only new methods/fields etc.
>>>
>>> Then to look at how a consumer, Derby, deals with the fact of multiple
>>> versions of the interface:
>>>
>>> - Derby only provides and uses the functionality to match the version of
>>> the interface (java.sql.Connection) loaded, determined at runtime.
>>>
>>> I'd thought this was a workable direction being proposed about six
>>> thousand message ago, upwards compatible apis and the ability for a
>>> consumer to handle a lower version. Not sure what derailed it.
>>>
>>
>> Kathey asked if I would be willing to write something up and I started
>> thinking about it but haven't had the bandwidth to write anything.
>>
>> As a quick braindump:
>> * there are rules that restrict changes to the interface going forward
>> * Sun (et al) stick to those rules even when it is painful
>> * no class/method is removed without being deprecated first
>> * the JVM supports version skew a little by not requiring an
>> implementation to implement all methods in a interface (at runtime)
>> * the interfaces often provide a mechanism to determine what features of
>> an API the implementation implements
>> * frameworks can utilize multiple classloaders to load multiple versions
>> of a class into the VM - tools have developed to simplify this
>>
>> I think we can support a scenario where:
>> * a comsumer expecting version X.Y will work with any implementation
>> X.Y' when Y' >= Y
>> * a consumer expecting X.Y will work with X.Y' when Y' < Y by version
>> detection and degraded functionality (Y' level). If Y level function
>> is required then it will gracefully die (able to detect rather
>> than AbstractMethodError)
>> * X.Y defines the interface version, i.e. X.Y.Z works with X.Y.Z' for
>> all Z'
>> * X defines the compatibility level i.e. X and X' are not guaranteed
>> to work together (although we will try to ensure they do)
>>
>> Basic rules:
>> * at the Z level, no API change is allowed, just implementation changes
>> * at the Y level, additive API changes are allowed but must be
>> accompanied by support in the versioning mechanism so that they can be
>> detected. Things can be deprecated
>> * at the Z level, incompatible changes are allowed. Items deprecated in
>> version X can be removed in version X+2 (implicitly (X+2).0.0).
>>
>> This all applies to operation in a single classloader. Where multiple
>> classloaders are involved (inside one VM or in multiple VMs) a
>> different set of versioning behaviours applies to the wire protocol.
>>
>> --
>> Jeremy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
So, I'll wait for another overnight (over-day) period, but it sounds
like we may have a level of consensus here (fingers crossed).
What I propose to do is take Jeremy's quick brain dump (thanks again for
doing this Jeremy) and turn it into a more complete policy around this.
Hopefully I'll capture the essence of this correctly, but I'm sure
you'll correct me if I miss something :)
I'll put this on the Derby Wiki, to make it more readable/reviewable.
Thanks all,
David
Jeremy Boynes wrote:
> Daniel John Debrunner wrote:
>
>> Jeremy Boynes wrote:
>>
>>
>>> Think of the carnage if it was java_14208_b13.sql_300.Connection
>>
>>
>>
>> It's actually instructive to look how Java solves this:
>>
>> - The interface is kept upwards compatible, by following rules such as
>> only new methods/fields etc.
>>
>> Then to look at how a consumer, Derby, deals with the fact of multiple
>> versions of the interface:
>>
>> - Derby only provides and uses the functionality to match the version of
>> the interface (java.sql.Connection) loaded, determined at runtime.
>>
>> I'd thought this was a workable direction being proposed about six
>> thousand message ago, upwards compatible apis and the ability for a
>> consumer to handle a lower version. Not sure what derailed it.
>>
>
> Kathey asked if I would be willing to write something up and I started
> thinking about it but haven't had the bandwidth to write anything.
>
> As a quick braindump:
> * there are rules that restrict changes to the interface going forward
> * Sun (et al) stick to those rules even when it is painful
> * no class/method is removed without being deprecated first
> * the JVM supports version skew a little by not requiring an
> implementation to implement all methods in a interface (at runtime)
> * the interfaces often provide a mechanism to determine what features of
> an API the implementation implements
> * frameworks can utilize multiple classloaders to load multiple versions
> of a class into the VM - tools have developed to simplify this
>
> I think we can support a scenario where:
> * a comsumer expecting version X.Y will work with any implementation
> X.Y' when Y' >= Y
> * a consumer expecting X.Y will work with X.Y' when Y' < Y by version
> detection and degraded functionality (Y' level). If Y level function
> is required then it will gracefully die (able to detect rather
> than AbstractMethodError)
> * X.Y defines the interface version, i.e. X.Y.Z works with X.Y.Z' for
> all Z'
> * X defines the compatibility level i.e. X and X' are not guaranteed
> to work together (although we will try to ensure they do)
>
> Basic rules:
> * at the Z level, no API change is allowed, just implementation changes
> * at the Y level, additive API changes are allowed but must be
> accompanied by support in the versioning mechanism so that they can be
> detected. Things can be deprecated
> * at the Z level, incompatible changes are allowed. Items deprecated in
> version X can be removed in version X+2 (implicitly (X+2).0.0).
>
> This all applies to operation in a single classloader. Where multiple
> classloaders are involved (inside one VM or in multiple VMs) a different
> set of versioning behaviours applies to the wire protocol.
>
> --
> Jeremy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Jeremy Boynes <jb...@apache.org>.
Daniel John Debrunner wrote:
> Jeremy Boynes wrote:
>
>
>>Think of the carnage if it was java_14208_b13.sql_300.Connection
>
>
> It's actually instructive to look how Java solves this:
>
> - The interface is kept upwards compatible, by following rules such as
> only new methods/fields etc.
>
> Then to look at how a consumer, Derby, deals with the fact of multiple
> versions of the interface:
>
> - Derby only provides and uses the functionality to match the version of
> the interface (java.sql.Connection) loaded, determined at runtime.
>
> I'd thought this was a workable direction being proposed about six
> thousand message ago, upwards compatible apis and the ability for a
> consumer to handle a lower version. Not sure what derailed it.
>
Kathey asked if I would be willing to write something up and I started
thinking about it but haven't had the bandwidth to write anything.
As a quick braindump:
* there are rules that restrict changes to the interface going forward
* Sun (et al) stick to those rules even when it is painful
* no class/method is removed without being deprecated first
* the JVM supports version skew a little by not requiring an
implementation to implement all methods in a interface (at runtime)
* the interfaces often provide a mechanism to determine what features of
an API the implementation implements
* frameworks can utilize multiple classloaders to load multiple versions
of a class into the VM - tools have developed to simplify this
I think we can support a scenario where:
* a comsumer expecting version X.Y will work with any implementation
X.Y' when Y' >= Y
* a consumer expecting X.Y will work with X.Y' when Y' < Y by version
detection and degraded functionality (Y' level). If Y level function
is required then it will gracefully die (able to detect rather
than AbstractMethodError)
* X.Y defines the interface version, i.e. X.Y.Z works with X.Y.Z' for
all Z'
* X defines the compatibility level i.e. X and X' are not guaranteed
to work together (although we will try to ensure they do)
Basic rules:
* at the Z level, no API change is allowed, just implementation changes
* at the Y level, additive API changes are allowed but must be
accompanied by support in the versioning mechanism so that they can be
detected. Things can be deprecated
* at the Z level, incompatible changes are allowed. Items deprecated in
version X can be removed in version X+2 (implicitly (X+2).0.0).
This all applies to operation in a single classloader. Where multiple
classloaders are involved (inside one VM or in multiple VMs) a different
set of versioning behaviours applies to the wire protocol.
--
Jeremy
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Hm, that has continued to be the proposal I liked best. I thought what
derailed it was the inability to guarantee forward compatibility, that
is when a new consumer depends upon a new API.
For example, let's say you have an interface in release 1:
public MyInterface
{
int method1(int val);
}
Now in release 2 the consumer wants to add a new method:
public MyInterface
{
int method1(int val);
String method2(String val);
}
This is a backward-compatible change but not a forward-compatible
change, because if the new consumer calls method2 on the original
MyInterface, he'll get an error.
You could argue this is solvable by saying any modifications to an
interface require a new interface:
public MyInterface2 extends MyInterface
{
int method2(String val);
}
This is both backward- and forward-compatible.
The problem with this is the jar sealing problem. If the old jar is
first in the classpath, then MyInterface will be loaded from the old jar
and MyInterface2 will be loaded from the new jar. If these interfaces
are in the same package (org.apache.derby.common) then you'll get a
SecurityException.
I don't think that Java solves this particular problem, as it generally
assumes you have only one version in your classpath at a given time.
That's what I think has continued to derail any solution we propose.
Thus my proposal to rename the package, to solve the jar sealing problem.
David
Daniel John Debrunner wrote:
> Jeremy Boynes wrote:
>
>
>>Think of the carnage if it was java_14208_b13.sql_300.Connection
>
>
> It's actually instructive to look how Java solves this:
>
> - The interface is kept upwards compatible, by following rules such as
> only new methods/fields etc.
>
> Then to look at how a consumer, Derby, deals with the fact of multiple
> versions of the interface:
>
> - Derby only provides and uses the functionality to match the version of
> the interface (java.sql.Connection) loaded, determined at runtime.
>
> I'd thought this was a workable direction being proposed about six
> thousand message ago, upwards compatible apis and the ability for a
> consumer to handle a lower version. Not sure what derailed it.
>
> Dan.
>
>
>
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Daniel John Debrunner <dj...@debrunners.com>.
Jeremy Boynes wrote:
> Think of the carnage if it was java_14208_b13.sql_300.Connection
It's actually instructive to look how Java solves this:
- The interface is kept upwards compatible, by following rules such as
only new methods/fields etc.
Then to look at how a consumer, Derby, deals with the fact of multiple
versions of the interface:
- Derby only provides and uses the functionality to match the version of
the interface (java.sql.Connection) loaded, determined at runtime.
I'd thought this was a workable direction being proposed about six
thousand message ago, upwards compatible apis and the ability for a
consumer to handle a lower version. Not sure what derailed it.
Dan.
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
David W. Van Couvering wrote:
> This proposal was based on the assumption that our common code would
> *never* be exposed to the outside world.
>
> I also never claimed that we should apply this solution to third-party
> components. That's a fish I don't think we need to fry right now.
>
Although I do want to remind us all that, based on Dan's email a while
back, the JDK itself encountered the mixed version problem and solved/is
solving it through package renaming...
http://mail-archives.apache.org/mod_mbox/db-derby-dev/200509.mbox/%3c43275363.6000503@debrunners.com%3e
"The future plan is to rename the org.apache.** packages to be something
like com.sun.org.apache.** to fix this problem."
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
This proposal was based on the assumption that our common code would
*never* be exposed to the outside world.
I also never claimed that we should apply this solution to third-party
components. That's a fish I don't think we need to fry right now.
I think you are probably unhappy with this solution because it goes
against your vision of a modular architecture for Derby.
If you don't assume that any "module" of Derby can go with any other
version of any other module, but instead assume that all of Derby goes
together as a single unit, then it's fine if the client and engine code
for a given version are tied to the common code for the same version.
It actually makes sense -- you have well-defined and predictable
behavior in an unpredictable environment where multiple versions live in
the same classpath.
If you aren't thinking modular, I personally don't think it's that
arcane and confusing, particularly compared to making multiple copies of
an original "common" class.
I also can't think of any other solution besides class cloning (which I
really dislike) that can solve this problem and still give us common code.
David
Jeremy Boynes wrote:
> I find this arcane and confusing; worse, I don't think it really works.
>
> Code that is using the common API becomes tied to a specific version. By
> renaming the package we force all users of the common code to modify
> their code. We can do that for things in our source tree but not in others.
>
> Similarly this forces us to modify any external libraries we use to
> exhibit the same behaviour - for example, renaming o.a.commons.logging
> to o.a.commons.logging_1208. This means we have custom versions of all
> external libraries.
>
> Think of the carnage if it was java_14208_b13.sql_300.Connection
>
> --
> Jeremy
>
> David W. Van Couvering wrote:
>
>> Hi, all. I've been letting this brew in my subconscious this weekend,
>> with the strong belief that *somehow* there is a solution that can meet
>> all of our requirements (simple architecture, ease-of-use,
>> compatibility, no regressions, easy to maintain, reduce code
>> duplication).
>>
>> I think I may have something that will do this, and your comments are
>> much appreciated. There is a bit of overhead for the release process,
>> but I think if we are going to have overhead, that's the place to do it.
>>
>> The principle is that for each release, the common package is unique.
>> This is achieved by appending the release number to the package name.
>> So for the 10.1.2 release the package name is org.apache.common_1012.
>> In this way a consumer of the common classes for a given version is
>> guaranteed to load the right classes and no conflicts or confusion occur.
>>
>> As part of the release process, when you make a branch for a release,
>> you run a script that renames the common package to match your release
>> name. This script does the following:
>>
>> - In the source, replace all instances of
>> org.apache.common_<oldversion> with org.apache.common_<newversion>
>>
>> - Do an svn move of java/common/org/apache/common_<oldversion> to
>> java/common/org/apache/common_<newversion>
>>
>> Because you are using svn move, although it's a bit confusing, all
>> history is maintained and you can still use svn to do merges and ports
>> of changes.
>>
>> The common package is put into both derby.jar and derby-client.jar,
>> and we do not create a new JAR file.
>>
>> Thanks,
>>
>> David
>
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Jeremy Boynes <jb...@apache.org>.
I find this arcane and confusing; worse, I don't think it really works.
Code that is using the common API becomes tied to a specific version. By
renaming the package we force all users of the common code to modify
their code. We can do that for things in our source tree but not in others.
Similarly this forces us to modify any external libraries we use to
exhibit the same behaviour - for example, renaming o.a.commons.logging
to o.a.commons.logging_1208. This means we have custom versions of all
external libraries.
Think of the carnage if it was java_14208_b13.sql_300.Connection
--
Jeremy
David W. Van Couvering wrote:
> Hi, all. I've been letting this brew in my subconscious this weekend,
> with the strong belief that *somehow* there is a solution that can meet
> all of our requirements (simple architecture, ease-of-use,
> compatibility, no regressions, easy to maintain, reduce code duplication).
>
> I think I may have something that will do this, and your comments are
> much appreciated. There is a bit of overhead for the release process,
> but I think if we are going to have overhead, that's the place to do it.
>
> The principle is that for each release, the common package is unique.
> This is achieved by appending the release number to the package name. So
> for the 10.1.2 release the package name is org.apache.common_1012. In
> this way a consumer of the common classes for a given version is
> guaranteed to load the right classes and no conflicts or confusion occur.
>
> As part of the release process, when you make a branch for a release,
> you run a script that renames the common package to match your release
> name. This script does the following:
>
> - In the source, replace all instances of org.apache.common_<oldversion>
> with org.apache.common_<newversion>
>
> - Do an svn move of java/common/org/apache/common_<oldversion> to
> java/common/org/apache/common_<newversion>
>
> Because you are using svn move, although it's a bit confusing, all
> history is maintained and you can still use svn to do merges and ports
> of changes.
>
> The common package is put into both derby.jar and derby-client.jar, and
> we do not create a new JAR file.
>
> Thanks,
>
> David
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Rick Hillegas <Ri...@Sun.COM>.
There are now three cloning proposals for handling shared code:
1) Hand cloning by the developer as is done today.
2) Build-time cloning.
3) Release-time cloning (David's latest proposal).
All of these cloning approaches address Dan's problem case: a client and
server at diffferent rev levels in the same classpath. It's worth
pointing out that none of these proposals addresses the other mixed
usage scenario: wiring into the classpath two applications each of which
embeds a different version of the server. In that scenario we can still
expect Heisenbugs. We've never tested this scenario and so we have no
idea what our current exposure is. I agree with Jeremy that this
scenario is a systemic defect in Java and that we are not responsible
for solving it. The best we can do is point our users at best-of-breed
techniques for containing the problem.
Here's what I think about the cloning proposals:
(1)
+ Requires no changes.
- Brittle, bug-prone.
- Can't cut clone bloat by refactoring jar files.
(2)
+ Simple for bug porting.
- Can't cut clone bloat by refactoring jar files.
- Awkward to code and debug.
(3)
+ Easy to code and debug.
+ Clone bloat not necessary if client and server at same rev level.
- Awkward for bug porting.
In and of themselves, none of these approaches presents the naive
customer any additional complexity out-of-the-box. Additional complexity
would vex the customer if, for option (3) we refactored the jar files to
cut clone bloat. I dislike (3) the least.
Cheers,
-Rick
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Francois Orsini wrote:
> I think you meant 'org.apache.derby.common' in your examples.
Yes, thanks.
>
> Now, if the common package versioning is well automated as part of
> generating a new release, then it is probably ok to have a new version
> being generated for every release.
>
> Are you thinking that we would always see 'org.apache.derby.common' in
> the source tree and that the common pkg version suffix would only be
> added when a (new) release gets generated?
Well, this is a good question, if you're asking about what should the
package name be in the trunk. Yes, I suppose it could just be
'org.apache.derby.common' and the rename happens only when a branch is
made for a release. I am trying to think if this would cause any issues
for people pulling the nightly builds -- I can't think of any, unless
people try to mix versions from old and new nightly builds, in which
case I think they're really on their own :)
An alternate approach is to do the rename on the trunk just prior to
making the branch, so the trunk package name always matches the most
recent release branch. But this doesn't seem as clean...
David
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Francois Orsini <fr...@gmail.com>.
I think you meant 'org.apache.derby.common' in your examples.
Now, if the common package versioning is well automated as part of
generating a new release, then it is probably ok to have a new version being
generated for every release.
Are you thinking that we would always see 'org.apache.derby.common' in the
source tree and that the common pkg version suffix would only be added when
a (new) release gets generated?
--francois
On 9/19/05, David W. Van Couvering <Da...@sun.com> wrote:
>
> Hi, all. I've been letting this brew in my subconscious this weekend,
> with the strong belief that *somehow* there is a solution that can meet
> all of our requirements (simple architecture, ease-of-use,
> compatibility, no regressions, easy to maintain, reduce code duplication).
>
> I think I may have something that will do this, and your comments are
> much appreciated. There is a bit of overhead for the release process,
> but I think if we are going to have overhead, that's the place to do it.
>
> The principle is that for each release, the common package is unique.
> This is achieved by appending the release number to the package name.
> So for the 10.1.2 release the package name is org.apache.common_1012.
> In this way a consumer of the common classes for a given version is
> guaranteed to load the right classes and no conflicts or confusion occur.
>
> As part of the release process, when you make a branch for a release,
> you run a script that renames the common package to match your release
> name. This script does the following:
>
> - In the source, replace all instances of org.apache.common_<oldversion>
> with org.apache.common_<newversion>
>
> - Do an svn move of java/common/org/apache/common_<oldversion> to
> java/common/org/apache/common_<newversion>
>
> Because you are using svn move, although it's a bit confusing, all
> history is maintained and you can still use svn to do merges and ports
> of changes.
>
> The common package is put into both derby.jar and derby-client.jar, and
> we do not create a new JAR file.
>
> Thanks,
>
> David
>
>
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Hi, all. I've been letting this brew in my subconscious this weekend,
with the strong belief that *somehow* there is a solution that can meet
all of our requirements (simple architecture, ease-of-use,
compatibility, no regressions, easy to maintain, reduce code duplication).
I think I may have something that will do this, and your comments are
much appreciated. There is a bit of overhead for the release process,
but I think if we are going to have overhead, that's the place to do it.
The principle is that for each release, the common package is unique.
This is achieved by appending the release number to the package name.
So for the 10.1.2 release the package name is org.apache.common_1012.
In this way a consumer of the common classes for a given version is
guaranteed to load the right classes and no conflicts or confusion occur.
As part of the release process, when you make a branch for a release,
you run a script that renames the common package to match your release
name. This script does the following:
- In the source, replace all instances of org.apache.common_<oldversion>
with org.apache.common_<newversion>
- Do an svn move of java/common/org/apache/common_<oldversion> to
java/common/org/apache/common_<newversion>
Because you are using svn move, although it's a bit confusing, all
history is maintained and you can still use svn to do merges and ports
of changes.
The common package is put into both derby.jar and derby-client.jar, and
we do not create a new JAR file.
Thanks,
David
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
It sounds to me then that it's OK to have the same packages in derby.jar
and derby-client.jar. The classes in the common package would get
loaded from one and only of the jar files. We just need them in both so
the client and engine jars can be independent.
David
Daniel John Debrunner wrote:
> Andrew McIntyre wrote:
>
>
>
>>When a jar is sealed, classes cannot be loaded from multiple if the
>>package they are in is sealed. You can unseal specific packages, so
>>that's less of an issue than for signed jars. I believe attempting to
>>load the same class from different jars if the jars are signed will
>>cause a SecurityException, but I'm not well versed in the ins and outs
>>of signed jars.
>
>
> I think signing and sealing are independent concepts.
>
> If a class has been loaded from a signed jar, then it won't be loaded
> again from any jar (within the same class loader), so I don't think
> there can be any exception there.
>
> A sealed package means that all classes within that package must be
> loaded from the same jar. I assume this is true for a single
> classloader, not across the vm.
>
>
> Dan.
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Daniel John Debrunner <dj...@debrunners.com>.
Andrew McIntyre wrote:
> When a jar is sealed, classes cannot be loaded from multiple if the
> package they are in is sealed. You can unseal specific packages, so
> that's less of an issue than for signed jars. I believe attempting to
> load the same class from different jars if the jars are signed will
> cause a SecurityException, but I'm not well versed in the ins and outs
> of signed jars.
I think signing and sealing are independent concepts.
If a class has been loaded from a signed jar, then it won't be loaded
again from any jar (within the same class loader), so I don't think
there can be any exception there.
A sealed package means that all classes within that package must be
loaded from the same jar. I assume this is true for a single
classloader, not across the vm.
Dan.
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Andrew McIntyre <mc...@gmail.com>.
On 9/16/05, David W. Van Couvering <Da...@sun.com> wrote:
> I appreciate your pragmatic approach, Andrew. The thing is I have seen
> a number of other pieces of functionality queued up for code sharing
> between client and server. These include DRDA network code, potentially
> some higher-level JDBC functionality, logging and tracing, and
> management via JMX. I also discovered that the versioning mechanism in
> org.apache.derby.iapi.services.info is actually reused across tools,
> client and engine.
This is true, I realize that there are a lot of opportunities for code
sharing and reuse. I just don't know if *this* is the one that is
worth it. I think the way these possible code-sharing opportunities
should all be handled is for someone to implement some new
functionality as a module and package it into a new *optional* jar
file. The new code can be used if it is present, and the old code can
be deprecated, but remain packaged for a certain number of releases
before it is removed, if it is removed. This gives users time to
prepare for a change and developers to work through all the issues
without causing any unnecessary pain along the way.
Do we really want to go to this trouble for the client localization
functionality? I don't know. To me, logging seems like a better first
project for that.
> We may still decide this is intractable and throw up our hands, but I
> hope that's not the case. It's been a healthy debate, and has brought
> to bear some good discussions about "who" we want to be, what our key
> principles are, and I think that's been good too.
I agree, it's been a good healthy debate, but I thought it might be
good to reflect upon the task at hand. :-)
> You talked about signing/sealing problems about packaging the classes in
> multiple jars. I saw this mentioned before, can you or someone else
> elaborate?
When a jar is sealed, classes cannot be loaded from multiple if the
package they are in is sealed. You can unseal specific packages, so
that's less of an issue than for signed jars. I believe attempting to
load the same class from different jars if the jars are signed will
cause a SecurityException, but I'm not well versed in the ins and outs
of signed jars.
andrew
Re: Modular build, was: VOTE: Approach for sharing code
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
I appreciate your pragmatic approach, Andrew. The thing is I have seen
a number of other pieces of functionality queued up for code sharing
between client and server. These include DRDA network code, potentially
some higher-level JDBC functionality, logging and tracing, and
management via JMX. I also discovered that the versioning mechanism in
org.apache.derby.iapi.services.info is actually reused across tools,
client and engine.
Given this, I thought that although it would be a tough debate it was
worthwhile having and one which, if we solved, would enable a lot of
other common services to be built across client and engine. So I took
the challenge and decided to try and solve this.
We may still decide this is intractable and throw up our hands, but I
hope that's not the case. It's been a healthy debate, and has brought
to bear some good discussions about "who" we want to be, what our key
principles are, and I think that's been good too.
You talked about signing/sealing problems about packaging the classes in
multiple jars. I saw this mentioned before, can you or someone else
elaborate? If it's really true that we can't package the same classes
in multiple jars, that's a very important data point...
Thanks,
David
Andrew McIntyre wrote:
> On 9/15/05, Kathey Marsden <km...@sbcglobal.net> wrote:
>
>>Jeremy Boynes wrote:
>>
>>
>>>This dooms us forever to reinvent any functionality that could
>>>be provided by other projects.
>>>
>>
>>We are not "doomed forever". Requiring a new jar file for new
>>functionality seems an entirely reasonable thing to me and at that time
>>we can impose whatever restrictions the community sees fit. Requiring a
>>new jar file to have the product continue work, is another matter all
>>together.
>
>
> +1!
>
> It looks like we've got a really intractable situation here. There are
> those against source/binary modification because of the inherent
> problems with maintenance and debugging, but you can't package the
> same classes in multiple jars because of signing/sealing and
> classloader issues, and if you split the classes out into a new jar
> then you've got usability and deployment issues. So what do you do?
>
> I'm wondering if this is really the functionality to be considering
> all of this for. Two or three small classes related to localization.
> Is it really worth the trouble? Are we really going to make a common
> jar file with two classes, without which you don't get error messages?
> Really?
>
> And remember, this isn't about reusing someone else's code or
> libraries. I don't think anyone is going to be against reusing other
> project's code or libraries in order to promote cross-project goodwill
> and general open source happiness. We're talking about copying two of
> our own classes from one part of the tree to the other.
>
> I think we can all tackle the code-sharing problem with some new
> functionality later. For which, by the way, I think a good candidate
> would be integrating Lucene for full text indexing as a good trial for
> both code reuse from other projects and code sharing among the
> different parts of Derby. And personally, I'd like to see David be
> able to finish localizing the error messages sometime this decade. :-)
>
> andrew
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Jeremy Boynes <jb...@apache.org>.
Evolution would start with a small change that was then built on and
expanded. From that perspective, starting with a small change like the
localization would be the beginning of an evolutionary change. Doing it
now, near the start of a feature release, and providing plenty of time
for feedback from users in the community is less disruptive than trying
to rush it in at the end.
This is one area where open source is different than closed source
development - all users get to see changes early and can provide
feedback into the process, not just a selected few during a beta cycle
when it is usually far to late.
As for the size of the common jar, once you go beyond one adding
additional libraries in becomes just more of the same. That leads to a
model where packaging is based on loose vs. close coupling
considerations. Some services may be small, some may be larger, but the
boundaries are determined by the use case and not an arbitrary packaging
scheme.
This was an issue originally with Jakarata Commons. In its early days,
although the libraries were separated out from the original monolith
they were still coupled together; it wasn't until later that the
decoupling of services we have now was attained (and to some extent that
is still a work in progress).
--
Jeremy
Satheesh Bandaram wrote:
> +1. I also vote to solve the common code issues when we have sufficient
> critical mass of it. Currently
> there doesn't seem to be enough to justify a new mechanism, especially a
> solution that is directly visible
> to end users and could cause some disruptions. While the discussion has
> been very useful to evolve a
> possible solution in the future, I don't see an urgent need to address
> this yet. Not sure how end-users
> would see a derbycommon.jar that is only a few kilobytes (tens?) in size
> and wonder why it is even there.
>
> We just released a 10.1 feature release... Not sure when our next
> feature release is going to be, but seems
> like there might be many months of time to address any packaging issues.
> Let us continue to enhance
> Derby in the current framework and as we get closer to another release,
> we could evaluate the packaging
> issues.
>
> Evolution, not revolution... Someone said this before, don't remember
> who ... :-)
>
> Satheesh
>
> Andrew McIntyre wrote:
>
>
>>+1!
>>
>>It looks like we've got a really intractable situation here. There are
>>those against source/binary modification because of the inherent
>>problems with maintenance and debugging, but you can't package the
>>same classes in multiple jars because of signing/sealing and
>>classloader issues, and if you split the classes out into a new jar
>>then you've got usability and deployment issues. So what do you do?
>>
>>I'm wondering if this is really the functionality to be considering
>>all of this for. Two or three small classes related to localization.
>>Is it really worth the trouble? Are we really going to make a common
>>jar file with two classes, without which you don't get error messages?
>>Really?
>>
>>And remember, this isn't about reusing someone else's code or
>>libraries. I don't think anyone is going to be against reusing other
>>project's code or libraries in order to promote cross-project goodwill
>>and general open source happiness. We're talking about copying two of
>>our own classes from one part of the tree to the other.
>>
>>I think we can all tackle the code-sharing problem with some new
>>functionality later. For which, by the way, I think a good candidate
>>would be integrating Lucene for full text indexing as a good trial for
>>both code reuse from other projects and code sharing among the
>>different parts of Derby. And personally, I'd like to see David be
>>able to finish localizing the error messages sometime this decade. :-)
>>
>>andrew
>>
>>
>>
>>
>>
>
>
Re: Modular build, was: VOTE: Approach for sharing code
Posted by Satheesh Bandaram <sa...@Sourcery.Org>.
+1. I also vote to solve the common code issues when we have sufficient
critical mass of it. Currently
there doesn't seem to be enough to justify a new mechanism, especially a
solution that is directly visible
to end users and could cause some disruptions. While the discussion has
been very useful to evolve a
possible solution in the future, I don't see an urgent need to address
this yet. Not sure how end-users
would see a derbycommon.jar that is only a few kilobytes (tens?) in size
and wonder why it is even there.
We just released a 10.1 feature release... Not sure when our next
feature release is going to be, but seems
like there might be many months of time to address any packaging issues.
Let us continue to enhance
Derby in the current framework and as we get closer to another release,
we could evaluate the packaging
issues.
Evolution, not revolution... Someone said this before, don't remember
who ... :-)
Satheesh
Andrew McIntyre wrote:
>+1!
>
>It looks like we've got a really intractable situation here. There are
>those against source/binary modification because of the inherent
>problems with maintenance and debugging, but you can't package the
>same classes in multiple jars because of signing/sealing and
>classloader issues, and if you split the classes out into a new jar
>then you've got usability and deployment issues. So what do you do?
>
>I'm wondering if this is really the functionality to be considering
>all of this for. Two or three small classes related to localization.
>Is it really worth the trouble? Are we really going to make a common
>jar file with two classes, without which you don't get error messages?
>Really?
>
>And remember, this isn't about reusing someone else's code or
>libraries. I don't think anyone is going to be against reusing other
>project's code or libraries in order to promote cross-project goodwill
>and general open source happiness. We're talking about copying two of
>our own classes from one part of the tree to the other.
>
>I think we can all tackle the code-sharing problem with some new
>functionality later. For which, by the way, I think a good candidate
>would be integrating Lucene for full text indexing as a good trial for
>both code reuse from other projects and code sharing among the
>different parts of Derby. And personally, I'd like to see David be
>able to finish localizing the error messages sometime this decade. :-)
>
>andrew
>
>
>
>
>