You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Daniel John Debrunner <dj...@debrunners.com> on 2005/03/04 23:25:21 UTC
Single JDK14 compile model?
The current compile model for Derby uses a JDK 1.3 and JDK 1.4. Those
classes that are required to run in JDK1.3/JDBC 2.0 are compiled against
the JDK 1.3 libraries, and the JDK 1.4/1.5/JDBC 3.0 are compiled against
the JDK 1.4 libraries. This requires developers to download and setup
two JDKs and some extra libraries from JDK 1.3 extensions.
While this model worked well for closed source (all the setup was
handled automatically), it's no so good for open source. This is because
of putting the burden on developers to download extra stuff.
I was trying to expand this model to support J2ME/CDC/Foundation but I
am now having doubts, mainly due to the requirement at Apache to be part
of Gump. Andrew had to modify various build.xml files (adding near
duplicate actions) and make the gump build a multi-pass affair. I don't
see how I can add J2ME support in this mode while also keeping Gump running.
So I looked at an alternate approach where all the code is compiled
using a single JDK, requiring at least JDK 1.4 though I'm only tested
with 1.4 so far. This requires making some classes abstract that must
not be abstract in J2ME or JDK 1.3. EmbedResultSet is an example. This
allows the class to then compile under JDK 1.4.
The trick I then use is to modify the class file sometime after
compilation to mark it concrete by modifying the byte code in the .class
file (using Derby's class file utilities). This does not cause any
problems with the JVM (though there are some issues with the current
version of jikes that Derby uses).
So is this direction ok for Derby? Along with Jeremy checking in the
Apache jar files required by Derby, it would make downloading and
compling Derby much easier.
Looking at the two approaches, here are the trade-offs:
Mulitple JDKs
+ enforces correct sub-setting at development time, enforced by the
compiler, e.g. correct methods in JDBC implementations, not using JDK
1.4 classes in a JDK 1.3 environment, not using non J2ME classes in J2ME
code.
- tricky (maybe impossible with J2ME) to work with Gump
- tricky for the developer to get started on Derby
- J2ME libraries not (easily) available
Single JDK
- correct implementations only enforced by testing in the given
environment, e.g. JDK 1.3 J2ME.
- requires more awareness for contributors working in the code
(e.g. not to use JDK 1.4 classes in code that might be used in J2ME).
+ simple for Gump (hopefully)
+ simple for the developer to set up
Comments?
Dan.
Re: Single JDK14 compile model?
Posted by Daniel John Debrunner <dj...@debrunners.com>.
Kathey Marsden wrote:
> Is there any bytecode incompatibility when you compile under jdk141 and
> run under jdk131? I heard once that there was but it was merely
> anecdote and I don't know if that is really true or not, but thought it
> would be worth confirming that it is ok to compile under jdk141 and run
> under jdk131.
I don't believe so, in the current build model a single compiler is
used, either javac from 1.4.2 or jikes, just different libraries on its
compile classpath.
Dan.
Re: Single JDK14 compile model?
Posted by RPost <rp...@pacbell.net>.
>Daniel John Debrunner wrote:
>- requires more awareness for contributors working in the code
>(e.g. not to use JDK 1.4 classes in code that might be used in J2ME).
Do you know the current state of J2ME as regards support for JDK 1.4?
Is the need to support J2ME the only 'technical' reason that JDK 1.3 needs
to be supported. I mean technical, as opposed to business reasons since I
don't think the community is ready to drop support for JDK 1.3 in general.
Re: Single JDK14 compile model?
Posted by Kathey Marsden <km...@sbcglobal.net>.
Daniel John Debrunner wrote:
>The current compile model for Derby uses a JDK 1.3 and JDK 1.4. Those
>classes that are required to run in JDK1.3/JDBC 2.0 are compiled against
>the JDK 1.3 libraries, and the JDK 1.4/1.5/JDBC 3.0 are compiled against
>the JDK 1.4 libraries. This requires developers to download and setup
>two JDKs and some extra libraries from JDK 1.3 extensions.
>
>While this model worked well for closed source (all the setup was
>handled automatically), it's no so good for open source. This is because
>of putting the burden on developers to download extra stuff.
>
>I was trying to expand this model to support J2ME/CDC/Foundation but I
>am now having doubts, mainly due to the requirement at Apache to be part
>of Gump. Andrew had to modify various build.xml files (adding near
>duplicate actions) and make the gump build a multi-pass affair. I don't
>see how I can add J2ME support in this mode while also keeping Gump running.
>
>So I looked at an alternate approach where all the code is compiled
>using a single JDK, requiring at least JDK 1.4 though I'm only tested
>with 1.4 so far. This requires making some classes abstract that must
>not be abstract in J2ME or JDK 1.3. EmbedResultSet is an example. This
>allows the class to then compile under JDK 1.4.
>
>The trick I then use is to modify the class file sometime after
>compilation to mark it concrete by modifying the byte code in the .class
>file (using Derby's class file utilities). This does not cause any
>problems with the JVM (though there are some issues with the current
>version of jikes that Derby uses).
>
>So is this direction ok for Derby? Along with Jeremy checking in the
>Apache jar files required by Derby, it would make downloading and
>compling Derby much easier.
>
>Looking at the two approaches, here are the trade-offs:
>
>Mulitple JDKs
>
>+ enforces correct sub-setting at development time, enforced by the
>compiler, e.g. correct methods in JDBC implementations, not using JDK
>1.4 classes in a JDK 1.3 environment, not using non J2ME classes in J2ME
>code.
>
>- tricky (maybe impossible with J2ME) to work with Gump
>
>- tricky for the developer to get started on Derby
>
>- J2ME libraries not (easily) available
>
>
>Single JDK
>
>- correct implementations only enforced by testing in the given
>environment, e.g. JDK 1.3 J2ME.
>
>- requires more awareness for contributors working in the code
>(e.g. not to use JDK 1.4 classes in code that might be used in J2ME).
>
>+ simple for Gump (hopefully)
>
>+ simple for the developer to set up
>
>Comments?
>Dan.
>
>
>
>
Is there any bytecode incompatibility when you compile under jdk141 and
run under jdk131? I heard once that there was but it was merely
anecdote and I don't know if that is really true or not, but thought it
would be worth confirming that it is ok to compile under jdk141 and run
under jdk131.
Thanks
Kathey
Re: Single JDK14 compile model?
Posted by Andrew McIntyre <mc...@gmail.com>.
On Fri, 04 Mar 2005 14:25:21 -0800, Daniel John Debrunner
<dj...@debrunners.com> wrote:
> I was trying to expand this model to support J2ME/CDC/Foundation but I
> am now having doubts, mainly due to the requirement at Apache to be part
> of Gump. Andrew had to modify various build.xml files (adding near
> duplicate actions) and make the gump build a multi-pass affair. I don't
> see how I can add J2ME support in this mode while also keeping Gump running.
Just to be clear, the near-duplicate actions were needed to work
around a bug in Jikes 1.22. But yes, because of the way Gump controls
the compilation classpath, it became necessary to split the engine
build across JDBC 2/JDBC 3 lines. I might be able to simplify it
somewhat from the current situation, but I haven't spent much time on
it since I got it working.
> The trick I then use is to modify the class file sometime after
> compilation to mark it concrete by modifying the byte code in the .class
> file (using Derby's class file utilities).
Adjusting how we build in Gump for changing requirements is an
inconvenience, one that I'm not sure really calls for going to such
lengths to mitigate. I'd rather put the time in myself to keeping our
build running in Gump than deal with problems caused by modifying the
classfiles at build time (I'm thinking of all the fun I've had with
obfuscation). :-)
A more serious problem would seem to be:
> - J2ME libraries not (easily) available
Is there any readily available reference implementation for J2ME
(w/JSR169)? How would Gump, or anyone, build Derby if we require J2ME
reference libraries to build, but there's no available reference
implementation? If the answer is: there is none, then I suppose we'll
need to consider this anyway.
Also, Kathey Marsden wrote:
> Suddenly users have to think of which set of jar files to
> download, we need to document when and why to use which and they may
> need to download a different jar when they upgrade their jvm.
If the choice is between J2ME and J2SE, then that's pretty easy, but I
don't think we should have separate jars for different JDK revisions.
That would get confusing indeed, not to mention be a real hassle for
maintenance (and versioning, and, and...)
Even creating a split between J2ME and J2SE shouldn't require
splitting the codeline. The module system allows for creating
different jar files with different sets of modules, although I think
the code for this would need to be revived in the build.
andrew
Re: Single JDK14 compile model?
Posted by Suresh Thalamati <su...@gmail.com>.
Daniel John Debrunner wrote:
>The trick I then use is to modify the class file sometime after
>compilation to mark it concrete by modifying the byte code in the .class
>file (using Derby's class file utilities). This does not cause any
>problems with the JVM (though there are some issues with the current
>version of jikes that Derby uses).
>
>
>
Will the debugging tools like Eclipse map a class to it's source file
properly, if a class file is modified as mentioned above ?
-suresh
Re: Single JDK14 compile model?
Posted by David Van Couvering <Da...@Sun.COM>.
I must confess to not understanding all the issues, but I think in
summary what you're saying is that you are "tricking" the older VMs to
work by modifying byte code compiled under JDK 1.4. Although I see the
ease-of-development value in this, it is a little disconcerting that
we'd be taking VM compatibility into our own hands. Also, what are the
consequences of this approach when we want to support JDK 1.5?
At the same time, I think what you're saying is that if we include
support for J2ME then things as they are become untenable. Are there
any other alternatives?
David
Daniel John Debrunner wrote:
>The current compile model for Derby uses a JDK 1.3 and JDK 1.4. Those
>classes that are required to run in JDK1.3/JDBC 2.0 are compiled against
>the JDK 1.3 libraries, and the JDK 1.4/1.5/JDBC 3.0 are compiled against
>the JDK 1.4 libraries. This requires developers to download and setup
>two JDKs and some extra libraries from JDK 1.3 extensions.
>
>While this model worked well for closed source (all the setup was
>handled automatically), it's no so good for open source. This is because
>of putting the burden on developers to download extra stuff.
>
>I was trying to expand this model to support J2ME/CDC/Foundation but I
>am now having doubts, mainly due to the requirement at Apache to be part
>of Gump. Andrew had to modify various build.xml files (adding near
>duplicate actions) and make the gump build a multi-pass affair. I don't
>see how I can add J2ME support in this mode while also keeping Gump running.
>
>So I looked at an alternate approach where all the code is compiled
>using a single JDK, requiring at least JDK 1.4 though I'm only tested
>with 1.4 so far. This requires making some classes abstract that must
>not be abstract in J2ME or JDK 1.3. EmbedResultSet is an example. This
>allows the class to then compile under JDK 1.4.
>
>The trick I then use is to modify the class file sometime after
>compilation to mark it concrete by modifying the byte code in the .class
>file (using Derby's class file utilities). This does not cause any
>problems with the JVM (though there are some issues with the current
>version of jikes that Derby uses).
>
>So is this direction ok for Derby? Along with Jeremy checking in the
>Apache jar files required by Derby, it would make downloading and
>compling Derby much easier.
>
>Looking at the two approaches, here are the trade-offs:
>
>Mulitple JDKs
>
>+ enforces correct sub-setting at development time, enforced by the
>compiler, e.g. correct methods in JDBC implementations, not using JDK
>1.4 classes in a JDK 1.3 environment, not using non J2ME classes in J2ME
>code.
>
>- tricky (maybe impossible with J2ME) to work with Gump
>
>- tricky for the developer to get started on Derby
>
>- J2ME libraries not (easily) available
>
>
>Single JDK
>
>- correct implementations only enforced by testing in the given
>environment, e.g. JDK 1.3 J2ME.
>
>- requires more awareness for contributors working in the code
>(e.g. not to use JDK 1.4 classes in code that might be used in J2ME).
>
>+ simple for Gump (hopefully)
>
>+ simple for the developer to set up
>
>Comments?
>Dan.
>
>
>
Re: Single JDK14 compile model?
Posted by RPost <rp...@pacbell.net>.
Suresh Thalamati wrote:
> Not always true. Reverting back to an old jvm is easy in case of a
> problem, but that is not always true with the databases :-)
> Once the database is upgraded , it is not easy to revert back to an
> old verions that easily.
I'm not that concerned about anyone wanting to revert back to an old jvm.
The only time I have seen this is when evaluating a new jvm for possible
adoption. Once an evalution and testing of a new jvm is complete and put
into production I've never seen anyone try to go back.
Anyone going backwards is going to face a lot more issues than just jvm and
database.
Re: Single JDK14 compile model?
Posted by Jeremy Boynes <jb...@apache.org>.
Suresh Thalamati wrote:
> Jeremy Boynes wrote:
>> Add to that if someone is upgrading JVM version there is a good chance
>> they are also going to upgrade database version as well, which would
>> be a different jar anyway.
>>
> Not always true. Reverting back to an old jvm is easy in case of a
> problem, but that is not always true with the databases :-)
> Once the database is upgraded , it is not easy to revert back to an
> old verions that easily.
>
If the physical format changes then definitely. But hopefully we would
be able to run a newer version in a mode that kept the old physical
format allowing a quick rollback. However, that's an entirely different
but equally important discussion :-)
--
Jeremy
Re: Single JDK14 compile model?
Posted by Suresh Thalamati <su...@gmail.com>.
Jeremy Boynes wrote:
> Kathey Marsden wrote:
>
>>
>> I think there are not only maintenance and development issues with
>> having multiple branches for the jvm versions, but usability issues as
>> well. Suddenly users have to think of which set of jar files to
>> download, we need to document when and why to use which and they may
>> need to download a different jar when they upgrade their jvm. All in
>> all I am not a big fan of this proposal.
>>
>
> I believe that the decision to change JVM (be it vendor or version) is
> a major decision especially in a production environment. Having to use
> a different jar file for the database would be relatively minor
> concern given the level of proving that would be done with the new JVM.
>
> Add to that if someone is upgrading JVM version there is a good chance
> they are also going to upgrade database version as well, which would
> be a different jar anyway.
>
Not always true. Reverting back to an old jvm is easy in case of a
problem, but that is not always true with the databases :-)
Once the database is upgraded , it is not easy to revert back to an
old verions that easily.
Thanks
-suresht
Re: Single JDK14 compile model?
Posted by Jeremy Boynes <jb...@apache.org>.
Kathey Marsden wrote:
>
> I think there are not only maintenance and development issues with
> having multiple branches for the jvm versions, but usability issues as
> well. Suddenly users have to think of which set of jar files to
> download, we need to document when and why to use which and they may
> need to download a different jar when they upgrade their jvm. All in
> all I am not a big fan of this proposal.
>
I believe that the decision to change JVM (be it vendor or version) is a
major decision especially in a production environment. Having to use a
different jar file for the database would be relatively minor concern
given the level of proving that would be done with the new JVM.
Add to that if someone is upgrading JVM version there is a good chance
they are also going to upgrade database version as well, which would be
a different jar anyway.
--
Jeremy
Re: Single JDK14 compile model?
Posted by RPost <rp...@pacbell.net>.
"Kathey Marsden" wrote:
>Suddenly users have to think of which set of jar files to download
Maybe I misunderstood Jeremy's suggestion. How many choices are we talking
about?
I thought it was something like:
1. download a 1.3x JVM version of Derby
2. download a 1.4x JVM version of Derby
3. download a 1.5x JVM version of Derby
With each option possibly including whether J2ME support is needed.
Re: Single JDK14 compile model?
Posted by Kathey Marsden <km...@sbcglobal.net>.
>
> A lot of the problems here arise because we are trying to support 3
> different JVM revision levels with one output JAR. Is this time to
> consider generating multiple output JARs, one for each VM?
>
> This would also allow us to tune the implementations for each VM
> configuration e.g. eliminating code entirely for features not
> supportable/wanted in J2ME environments (reducing the code footprint),
> replacing SANE/INSANE with assertions in 1.4 and up, native support
> for regex using j.u.regex from 1.4, or leveraging features only
> available in JDK1.5 such as j.u.concurrent?
>
> The testing burden for this is no different as production releases
> would need to be tested on all VMs anyway. It is a higher burden for
> developers as fixes would need to be done in all code-lines (assuming
> some divergance).
>
> --
> Jeremy
>
I think there are not only maintenance and development issues with
having multiple branches for the jvm versions, but usability issues as
well. Suddenly users have to think of which set of jar files to
download, we need to document when and why to use which and they may
need to download a different jar when they upgrade their jvm. All in
all I am not a big fan of this proposal.
Thanks
Kathey
Re: Single JDK14 compile model?
Posted by RPost <rp...@pacbell.net>.
Daniel John Debrunner wrote:
> Now Y is compiled, but *not* X.
>
> The VM correectly handles this at runtime, loading X correctly, even
> though it does not implement all the methods of Y. It has to, in order
> to support upwards compatibility of java code.
>
> My marking the class as not abstract is just mimicing this situation.
>
Maybe so, but don't we always rebuild everything anyway? In which case X
won't compile because it doesn't implement all of Y's methods.
I think Jeremy's concern, at least my concern if it wasn't his, is whether
it is really necessary to use such a trick. I have to side with Jeremy on
this one. An expert such as yourself might be able to understand the
implications of such implementations and when, or more importantly when not,
to use them but the traditional developer won't be able to.
So I would ask if there isn't a more traditional solution that could be used
instead.
I ask about another 'trick' in
http://mail-archives.eu.apache.org/mod_mbox/db-derby-dev/200502.mbox/%3c0006
01c516d3$a8382710$1401a8c0@rpws002%3e where a low level interface,
DataValueDescriptor, has methods that return data types of BooleanDataValue
which is an interface that extends DataValueDescriptor. This may also be
necessary, although no one has responded to say so, but I don't think a
circular reference such as this is recommended.
Re: Single JDK14 compile model?
Posted by Daniel John Debrunner <dj...@debrunners.com>.
Jeremy Boynes wrote:
> Daniel John Debrunner wrote:
>
>>
>> The trick I then use is to modify the class file sometime after
>> compilation to mark it concrete by modifying the byte code in the .class
>> file (using Derby's class file utilities). This does not cause any
>> problems with the JVM (though there are some issues with the current
>> version of jikes that Derby uses).
>>
>
> This really really worries me and I am very uncomfortable with it.
My justification is that it is exactly the same as if the class file
implementing the interface has been compiled against one version of the
interface, and then run with a newer version. JVM's handle this correctly.
Take a simple example.
public interface Y
{
public void a();
}
public class X implements Y
{
public void a() {}
}
Now I compile X and Y, everything is correct.
Now Y is modified to add a method.
public interface Y
{
public void a();
public void b();
}
Now Y is compiled, but *not* X.
The VM correectly handles this at runtime, loading X correctly, even
though it does not implement all the methods of Y. It has to, in order
to support upwards compatibility of java code.
My marking the class as not abstract is just mimicing this situation.
Dan.
Re: Single JDK14 compile model?
Posted by David Van Couvering <Da...@Sun.COM>.
I must concur; in the past Derby had a single focus (embedded use) and a
single set of developers building to that focus. Now that it's in open
source, any number of folks will want/need to take it in different
directions, both smaller (CLDC) and larger (enterprise environments).
We'd either have to be vigilant about a single focus, risking smaller
takeup, or the development environment/model will need to adapt to that
reality...
David
Jeremy Boynes wrote:
> Daniel John Debrunner wrote:
>
>>
>> Also Jeremy said this approach (using modules) would lead to an ever
>> growing number of modules, that tends not to be the case, since as I
>> think Rick pointed out, at some point environments are dropped from
>> support, such as Derby used to support JDK 1.1 and no longer does. Most
>> of the split between 1.1 and 1.3 code has been removed.
>>
>
> Sure - but with JDK1.5 we are again reentering an era where there are
> major differences between VM levels. The only bytecode level
> difference between 1.3 and 1.4 was assert which IMHO was not
> compelling; however, with 1.5 we have generics, annotations, changes
> to String concatenation, foreach, autoboxing and more, never mind the
> changes in the libraries. I'm sure JDBC4.0 will use some of these
> features so what will we do next year?
>
> What I also meant by the growing number of modules was that we would
> be using them for features that a good number of users consider
> optional or unnecessary. Back to the example of analytics - wanted by
> OLAP users, pretty much irrelevant to embedded users. Or take JMX
> support - wanted by enterprise users, but 1.3 and 1.4 would require an
> additional library, and it wouldn't even run on CLDC J2ME due to the
> need for reflection. Or spatial - that might be useful in a location
> aware embedded device but not others. Object types - could that reduce
> the O/R complexity in applications? In-tx replication, useful only
> with a good pipe between servers. External security policy
> definitions? XML data types? And so on ...
>
> To me, the number of modules is only going to grow and
> one-size-fits-all is not a good long-term approach.
>
> --
> Jeremy
Re: Single JDK14 compile model?
Posted by Jeremy Boynes <jb...@apache.org>.
Daniel John Debrunner wrote:
>
> Also Jeremy said this approach (using modules) would lead to an ever
> growing number of modules, that tends not to be the case, since as I
> think Rick pointed out, at some point environments are dropped from
> support, such as Derby used to support JDK 1.1 and no longer does. Most
> of the split between 1.1 and 1.3 code has been removed.
>
Sure - but with JDK1.5 we are again reentering an era where there are
major differences between VM levels. The only bytecode level difference
between 1.3 and 1.4 was assert which IMHO was not compelling; however,
with 1.5 we have generics, annotations, changes to String concatenation,
foreach, autoboxing and more, never mind the changes in the libraries.
I'm sure JDBC4.0 will use some of these features so what will we do next
year?
What I also meant by the growing number of modules was that we would be
using them for features that a good number of users consider optional or
unnecessary. Back to the example of analytics - wanted by OLAP users,
pretty much irrelevant to embedded users. Or take JMX support - wanted
by enterprise users, but 1.3 and 1.4 would require an additional
library, and it wouldn't even run on CLDC J2ME due to the need for
reflection. Or spatial - that might be useful in a location aware
embedded device but not others. Object types - could that reduce the O/R
complexity in applications? In-tx replication, useful only with a good
pipe between servers. External security policy definitions? XML data
types? And so on ...
To me, the number of modules is only going to grow and one-size-fits-all
is not a good long-term approach.
--
Jeremy
Re: Single JDK14 compile model? (patch attached)
Posted by Jeremy Boynes <jb...@apache.org>.
Andrew McIntyre wrote:
>
> On Mar 8, 2005, at 6:00 AM, Jeremy Boynes wrote:
>
>> I ran into something similar where by compiling under 1.4 one of the
>> classes in ij uses the new StringBuffer.append(StringBuffer) method.
>>
>> Here's how I fixed it in the mavenized version.
>> <snip>
>
>
> Yep! That fixed the problem. Thanks, Jeremy!
>
> With the addition of that snippit, the jars built with the previous
> patch using jdk142 only (compiled with javac on Mac OS X, btw), work on
> jdk131 as well.
>
NP. I'm still not keen on the class file modification approach though.
--
Jeremy
Re: Single JDK14 compile model? (patch attached)
Posted by Andrew McIntyre <mc...@gmail.com>.
On Mar 8, 2005, at 6:00 AM, Jeremy Boynes wrote:
> I ran into something similar where by compiling under 1.4 one of the
> classes in ij uses the new StringBuffer.append(StringBuffer) method.
>
> Here's how I fixed it in the mavenized version.
> <snip>
Yep! That fixed the problem. Thanks, Jeremy!
With the addition of that snippit, the jars built with the previous
patch using jdk142 only (compiled with javac on Mac OS X, btw), work on
jdk131 as well.
andrew
Re: Single JDK14 compile model? (patch attached)
Posted by Jeremy Boynes <jb...@apache.org>.
Andrew McIntyre wrote:
>
> On Mar 6, 2005, at 7:50 AM, Daniel John Debrunner wrote:
>
>> No, we require the class files modified to be sub-classed (see my other
>> long e-mail from this morning) and there is no issue with it.
>
>
> For some reason I had it in my head that any subclass would know that
> its superclass was abstract, and that subclassing a class that was once
> abstract, but now concrete, would lead to problems. This is not the
> case, of course, since the subclass has no record of the superclass'
> abstract status.
>
> Attached is a patch to build with JDK 1.4.2 only. This works as expected
> when running under JDK 1.4.2. However, when I attempt to use the jars
> created with this build with JDK 1.3.1, I get a NoSuchMethodError on a
> select (create table and insert works fine). What have I missed here?
>
> andrew
>
I ran into something similar where by compiling under 1.4 one of the
classes in ij uses the new StringBuffer.append(StringBuffer) method.
Here's how I fixed it in the mavenized version.
--
Jeremy
Index:
modules/tools/src/main/java/org/apache/derby/tools/JDBCDisplayUtil.java
===================================================================
---
modules/tools/src/main/java/org/apache/derby/tools/JDBCDisplayUtil.java
(revision 156382)
+++
modules/tools/src/main/java/org/apache/derby/tools/JDBCDisplayUtil.java
(working copy)
@@ -556,7 +556,7 @@
for (int k=blanks.length(); k<w; k++)
blanks.append(' ');
- buf.append(blanks);
+ buf.append((Object)blanks);
// REMIND: could do more cleverness,
like keep around
// past buffers to reuse...
}
Re: Single JDK14 compile model? (patch attached)
Posted by Daniel John Debrunner <dj...@debrunners.com>.
Andrew McIntyre wrote:
> Attached is a patch to build with JDK 1.4.2 only. This works as expected
> when running under JDK 1.4.2. However, when I attempt to use the jars
> created with this build with JDK 1.3.1, I get a NoSuchMethodError on a
> select (create table and insert works fine). What have I missed here?
This is basically what I was doing. Were you using jikes or javac? Jikes
1.14 has some issues, in that it creates abstract methods in an abstract
class from its declared interfaces. This was fixed by jukes 1.22 though
from their change history I couldn't see any explanation for the change
in the Jikes release files. I sent a query to the jikes list asking for
clarification around this issue.
Dan.
Re: Single JDK14 compile model? (patch attached)
Posted by Andrew McIntyre <mc...@gmail.com>.
On Mar 6, 2005, at 7:50 AM, Daniel John Debrunner wrote:
> No, we require the class files modified to be sub-classed (see my other
> long e-mail from this morning) and there is no issue with it.
For some reason I had it in my head that any subclass would know that
its superclass was abstract, and that subclassing a class that was once
abstract, but now concrete, would lead to problems. This is not the
case, of course, since the subclass has no record of the superclass'
abstract status.
Attached is a patch to build with JDK 1.4.2 only. This works as
expected when running under JDK 1.4.2. However, when I attempt to use
the jars created with this build with JDK 1.3.1, I get a
NoSuchMethodError on a select (create table and insert works fine).
What have I missed here?
andrew
Re: Single JDK14 compile model?
Posted by Daniel John Debrunner <dj...@debrunners.com>.
Andrew McIntyre wrote:
> - any classfile modified in this way probably should not be subclassed,
> to avoid possible problems compiling or verifying classes in past or
> future JVMs. This could be achieved in the source by clearly documenting
> classes to be modified this way, and in the binary by setting ACC_FINAL
> at the same time that we unset ACC_ABSTRACT.
No, we require the class files modified to be sub-classed (see my other
long e-mail from this morning) and there is no issue with it.
The JVM/verifier does not and must not check at runtime that a class
implements all the the required methods from its declared interfaces.
This would break the Java model of backwards compatibility, e.g. running
a simple JDBC 2.0 driver in JDK 1.4. There's even the
AbstractMethodError and NoSuchMethod errors to handle this case.
Dan.
Re: Single JDK14 compile model?
Posted by Andrew McIntyre <mc...@gmail.com>.
On Mar 5, 2005, at 8:54 AM, Daniel John Debrunner wrote:
> The old Cloudscape obfuscation tool did a huge amount of manipulation
> of
> the class file, including modifying the byte code stream. Because of
> that experience I intentionally made sure this process was zero risk
> and
> didn't require huge changes in the class file.
I certainly agree that this is a much more minimal change to the
classfiles than the obfuscation process. After looking at the spec for
the class file structure, there are only two (very) minor concerns that
I see with marking a classfile not abstract after compiling it as
abstract:
- we should be careful to *just* clear the bit ACC_ABSTRACT, in case
other bits in that byte are used for other purposes in the future.
- any classfile modified in this way probably should not be subclassed,
to avoid possible problems compiling or verifying classes in past or
future JVMs. This could be achieved in the source by clearly
documenting classes to be modified this way, and in the binary by
setting ACC_FINAL at the same time that we unset ACC_ABSTRACT.
> Or maybe even simpler,
> just write a explicit program that only modifies that byte at the
> correct offset in the file, in place.
Well, we can't just change a particular offset in the classfile, since
a classfile's constant pool structure comes before the class' access
flags. I suppose it would be fairly straightforward to write a program
that accounts for that as well, but I believe you said you already had
this working using Derby's classfile utilities, yes? :-)
On Mar 5, 2005, at 8:46 AM, Daniel John Debrunner wrote:
> 2) GUMP - It seems Derby has to be part of Gump, but Gump's philosphy
> is
> very different to Derby's, it basically says the product build rules
> are
> wrong and it must build using gump rules. Maybe I'm wrong on this?
I think that we should definitely make an effort to build in the Gump
environment. Since other projects could (hopefully, will :-) ) be added
to Gump that require Derby as a dependency and want to build as a part
of the Gump nightly process, we should make an effort to build properly
there for the sake of being part of the community. But. I'm sure we'll
be forgiven for occasional Gump breakage due to major changes happening
here.
Also, I wouldn't say that Gump's build process implies that each
product's build rules are wrong. Gump just overrides the compilation
classpath as an expediency to cross-product integration. Instead of
having to know how each project expects to find its dependencies, it
puts the latest version of each dependency on the compilation classpath
and expects the compiler to take care of the rest.
In fact, I think Derby is the first project built by Gump to require
more than one version of the Java runtime libraries at a time, which
accounts somewhat for the contortions that were necessary to fit our
build into the current Gump build model.
andrew
Re: Single JDK14 compile model?
Posted by Jeremy Boynes <jb...@apache.org>.
Daniel John Debrunner wrote:
> Jeremy Boynes wrote:
>
> In order to support J2ME, JDK 1.3, JDK 1.4 and JDK 1.5 we have to build
> code that works in those environments, how it is packaged to run is
> separate.
>
The challenge we face is taking advantage of the capabilities of each
environment; I do not think we want to be constrained by the lowest
common denominator. There are features in the later platforms that are
simply not available in the earlier ones, features not just in the
runtime environment but in the bytecode itself; we can gain advantages
by compiling for a specific platform level but the price is a loss of
compatibility with prior levels.
In the end it comes down to what users want, and we need to adapt the
development process to meet those requirements.
If the most important factor for users is a single jar for all platform
levels, then we can continue to do that. Minor issues here are a more
complex build and perhaps no integration with Gump (although surely that
can be fixed). The big issues though are the constraint of 1.3 level
bytecode, code bloat due to multiple implementations, longer development
times to provide all features on all VMs, and a huge monolithic project
that contains the kitchen sink collection of modules.
On the other hand users may prefer a model where there are jars for each
VM platform containing identical feature sets but tuned for that
platform. This is going to be a lot of work for developers as we need to
implement every feature on each platform, and some will be easier than
others. It also leaves users with the kitchen sink problem in that the
configuration will contain major modules they don't need/want.
A third alternative is that we make it easy to generate custom
configurations and as part of the build make produce the ones that users
actually want. For example, one build could be a single-user embedded
J2ME with JSR169 containing just core database functions; another a
standalone enterprise server with network access, LDAP integration, high
concurrency, JMX support, ... These two builds may need to use different
JVMs, different bytecode levels, different library classes, and
different Derby modules.
I prefer the third option - it provides users with what they want
(whatever that is), and we get to choose on a feature by feature basis
whether to stick with the lowest common platform, to use all the
features of a newer one, or to experiment in the new world and work with
users on the effort of backporting. I think this provides a reasonable
balance that allows a rapid development process for new features that
users want but at the same time continues to support the existing user base.
Above all, let users decide.
--
Jeremy
Re: Single JDK14 compile model?
Posted by Daniel John Debrunner <dj...@debrunners.com>.
Jeremy Boynes wrote:
>
> A lot of the problems here arise because we are trying to support 3
> different JVM revision levels with one output JAR. Is this time to
> consider generating multiple output JARs, one for each VM?
I'm not so sure this is true. I think we can separate the build issues
from the packaging issues.
In order to support J2ME, JDK 1.3, JDK 1.4 and JDK 1.5 we have to build
code that works in those environments, how it is packaged to run is
separate.
Can we focus on the build issues first, and then maybe have a packaging
disucssion. If we can't solve the build then the packing discussion
doesn't matter.
Currently we require developers to download the correct environments and
use them. I think is is fine to continue like this, but there are two or
three problems.
1) The availability of J2ME libraries
2) GUMP - It seems Derby has to be part of Gump, but Gump's philosphy is
very different to Derby's, it basically says the product build rules are
wrong and it must build using gump rules. Maybe I'm wrong on this?
3) The burden on the developers of having to download 4 environments.
Also Jeremy said this approach (using modules) would lead to an ever
growing number of modules, that tends not to be the case, since as I
think Rick pointed out, at some point environments are dropped from
support, such as Derby used to support JDK 1.1 and no longer does. Most
of the split between 1.1 and 1.3 code has been removed.
Dan.
Re: Single JDK14 compile model?
Posted by RPost <rp...@pacbell.net>.
Jeremy Boynes wrote:
> To me the big issue is setting expectations with users - are they always
> going to be getting one jar or are they going to be able to choose
> between profiles based on JVM version, platform level, or SQL feature set.
>
> I believe that some features are going to be big and complex (e.g.
> object type support, charsets/collations, geo/spatial, full-text
> indexing, analytics, ...) and very few users will want/need all of
> these. Ultimately we are going to want a mechanism that allows users to
> easily build the module configuration they want.
And I know I am concerned about two types of user.
An 'end user' is going to use a pre-built version of Derby as an embedded
database. These users are concerned about JVM version and the speed and size
of the database and could care less about how many different code bases
might be available.
An 'integrator user' will be doing their own builds. They will be concerned
with what tools and environment are needed to perform the build, where to
get the tools (if the Derby site doesn't, or can't include them all), and
how to customize the build for the intended customer or use.
I'm sick and tired of bloated software such as word processors that have
support for 97 fonts when I only need 3 and can't integrate with
spreadsheets and do charts, etc when I don't need any of those features.
Unfortunately there is no way to strip out the unwanted stuff. One of the
old goals of modular software has long fallen by the wayside with the major
vendors because there is no profit in providing a software update that
includes the new features in a new separate DLL or SO; there is profit in
forcing users to uninstall the old and purchase an entire new version even
when the new version is 95% of what the old one was.
Let's no do that with Derby.
Re: Single JDK14 compile model?
Posted by Jeremy Boynes <jb...@apache.org>.
Daniel John Debrunner wrote:
> David Van Couvering wrote:
>
>
>>If the current situation (as I am beginning to understand) prevents us
>>from taking advantage of numerous JDK 1.4 and 1.5 features, then I think
>>this going to become more and more of a burden. As well as what Jeremy
>>mentioned, there is exception chaining, and NIO support, and others I'm
>>sure...
>
>
> The current Derby architecture explictly allows modules to take
> advantage of features in various JDKs. SO there is no issue there. Derby
> can dynamically load different modules for different VM environments.
Sure but as we take more and more advantage of features in newer VMs
then the number of different modules is going to grow and grow.
>
>
>>If the current situation also constrains our ability to add features
>>because we're worried about overall size of the JAR for J2ME, this is a
>>problem too.
>
>
> Maybe an issue. The bulk of the code is currently in the SQL engine and
> compiler, which is required on J2ME. At the moment, splitting the code
> up to have a jar that just supports J2ME is not worth the effort. It may
> become an issue if analytics or other features are added, but I'll worry
> about that when it happens.
>
> While I would like to see the current derby.jar get smaller, say 1Mb,
> I'm also half not convinced it's worth the effort. Having just got my
> wife an IPod mini with 4Gb storage, I was thinking given the really
> small physical size of such a device, what's the difference between 1Mb
> and 2Mb.
>
Drop it and see what still works (just don't tell the wife) - there is a
big difference between rotating bits of plastic and flash. How much RAM
does that iPod have? No more than 32MB I would guess, most of which will
be buffer to let them spin down the drive to reduce shock risk and power
drain.
To me the big issue is setting expectations with users - are they always
going to be getting one jar or are they going to be able to choose
between profiles based on JVM version, platform level, or SQL feature set.
I believe that some features are going to be big and complex (e.g.
object type support, charsets/collations, geo/spatial, full-text
indexing, analytics, ...) and very few users will want/need all of
these. Ultimately we are going to want a mechanism that allows users to
easily build the module configuration they want.
--
Jeremy
Re: Single JDK14 compile model?
Posted by Daniel John Debrunner <dj...@debrunners.com>.
David Van Couvering wrote:
> If the current situation (as I am beginning to understand) prevents us
> from taking advantage of numerous JDK 1.4 and 1.5 features, then I think
> this going to become more and more of a burden. As well as what Jeremy
> mentioned, there is exception chaining, and NIO support, and others I'm
> sure...
The current Derby architecture explictly allows modules to take
advantage of features in various JDKs. SO there is no issue there. Derby
can dynamically load different modules for different VM environments.
> If the current situation also constrains our ability to add features
> because we're worried about overall size of the JAR for J2ME, this is a
> problem too.
Maybe an issue. The bulk of the code is currently in the SQL engine and
compiler, which is required on J2ME. At the moment, splitting the code
up to have a jar that just supports J2ME is not worth the effort. It may
become an issue if analytics or other features are added, but I'll worry
about that when it happens.
While I would like to see the current derby.jar get smaller, say 1Mb,
I'm also half not convinced it's worth the effort. Having just got my
wife an IPod mini with 4Gb storage, I was thinking given the really
small physical size of such a device, what's the difference between 1Mb
and 2Mb.
Dan.
Re: Single JDK14 compile model?
Posted by David Van Couvering <Da...@Sun.COM>.
If the current situation (as I am beginning to understand) prevents us
from taking advantage of numerous JDK 1.4 and 1.5 features, then I think
this going to become more and more of a burden. As well as what Jeremy
mentioned, there is exception chaining, and NIO support, and others I'm
sure...
If the current situation also constrains our ability to add features
because we're worried about overall size of the JAR for J2ME, this is a
problem too.
David
Jeremy Boynes wrote:
> Daniel John Debrunner wrote:
>
>>
>> The trick I then use is to modify the class file sometime after
>> compilation to mark it concrete by modifying the byte code in the .class
>> file (using Derby's class file utilities). This does not cause any
>> problems with the JVM (though there are some issues with the current
>> version of jikes that Derby uses).
>>
>
> This really really worries me and I am very uncomfortable with it.
>
>> So is this direction ok for Derby? Along with Jeremy checking in the
>> Apache jar files required by Derby, it would make downloading and
>> compling Derby much easier.
>>
>> Looking at the two approaches, here are the trade-offs:
>>
>> Mulitple JDKs
>>
>> + enforces correct sub-setting at development time, enforced by the
>> compiler, e.g. correct methods in JDBC implementations, not using JDK
>> 1.4 classes in a JDK 1.3 environment, not using non J2ME classes in J2ME
>> code.
>>
>> - tricky (maybe impossible with J2ME) to work with Gump
>>
>> - tricky for the developer to get started on Derby
>>
>> - J2ME libraries not (easily) available
>>
>>
>> Single JDK
>>
>> - correct implementations only enforced by testing in the given
>> environment, e.g. JDK 1.3 J2ME.
>>
>> - requires more awareness for contributors working in the code
>> (e.g. not to use JDK 1.4 classes in code that might be used in J2ME).
>>
>> + simple for Gump (hopefully)
>>
>> + simple for the developer to set up
>>
>> Comments?
>
>
> A lot of the problems here arise because we are trying to support 3
> different JVM revision levels with one output JAR. Is this time to
> consider generating multiple output JARs, one for each VM?
>
> This would also allow us to tune the implementations for each VM
> configuration e.g. eliminating code entirely for features not
> supportable/wanted in J2ME environments (reducing the code footprint),
> replacing SANE/INSANE with assertions in 1.4 and up, native support
> for regex using j.u.regex from 1.4, or leveraging features only
> available in JDK1.5 such as j.u.concurrent?
>
> The testing burden for this is no different as production releases
> would need to be tested on all VMs anyway. It is a higher burden for
> developers as fixes would need to be done in all code-lines (assuming
> some divergance).
>
> --
> Jeremy
Re: Single JDK14 compile model?
Posted by RPost <rp...@pacbell.net>.
>"Jeremy Boynes" wrote:
> I was under the impression that the entire codebase currently ran under
> 1.3 (with the exception of the stuff specifically compiled for 1.4 which
> can be extracted from the build).
An example of what I meant by how the current codebase breaks out in terms
of JVM version is which specific derby source files will ONLY work with a
specific JVM version. That is, set of source files are currently 1.3
specific, 1.4 specific, etc.
Dan made a comment about a single JVM would need developers to be careful
not to include 1.4 code in classes to be run in J2ME. So my question is -
how would a new contributor know when they are changing code that might be
specific to a particular version..
I like your idea for segmenting the versions in separate jars. I would think
that most installations will only use one version and it makes it easier to
take older versions out of service.
Re: Single JDK14 compile model?
Posted by Jeremy Boynes <jb...@apache.org>.
RPost wrote:
>>"Jeremy Boynes" wrote:
>
>
>>A lot of the problems here arise because we are trying to support 3
>>different JVM revision levels with one output JAR. Is this time to
>>consider generating multiple output JARs, one for each VM?
>>
>
>
>>This would also allow us to tune the implementations for each VM
>>configuration e.g. eliminating code entirely for features not
>>supportable/wanted in J2ME environments (reducing the code footprint),
>>replacing SANE/INSANE with assertions in 1.4 and up, native support for
>>regex using j.u.regex from 1.4, or leveraging features only available in
>>JDK1.5 such as j.u.concurrent?
>
>
> Do you know how the current codebase break out in terms of JVM version?
> Which code is currently needed support 1.3 environments?
>
I was under the impression that the entire codebase currently ran under
1.3 (with the exception of the stuff specifically compiled for 1.4 which
can be extracted from the build).
Given Sun have their version of JDK1.3 in EOL and are dropping support
as of 3/30/2006 I would expect most users to be in the process of
migrating away from it; those that aren't are not likely to be
considering database upgrades either.
I see support for J2ME being more important than 1.3, and in those
environments eliminating redundant code is useful. As we go further and
add features to Derby then the size of the codebase is only going to
increase. Some of those features may not be necessary in a J2ME
environment (analytics as an extreme example) so some profiling will be
to their benefit. I don't see this as any different to an embedded
distribution without DRDA support.
> How would you handle interfaces or base classes that are common to more than
> one JVM version if the descendent class is specific to, say, 1.3? Would you
> duplicate the base classes in each jar?
>
To a large extent yes. The build for each JVM would result in a single
JAR targeted at that JVM so users would just include the correct JAR for
their VM.
Ideally there would also be one version of the interface/base-class in
the source tree shared by all versions, with the VM-specific descendent
just included in the appropriate build.
--
Jeremy
Re: Single JDK14 compile model?
Posted by RPost <rp...@pacbell.net>.
>"Jeremy Boynes" wrote:
> A lot of the problems here arise because we are trying to support 3
> different JVM revision levels with one output JAR. Is this time to
> consider generating multiple output JARs, one for each VM?
>
>This would also allow us to tune the implementations for each VM
>configuration e.g. eliminating code entirely for features not
>supportable/wanted in J2ME environments (reducing the code footprint),
>replacing SANE/INSANE with assertions in 1.4 and up, native support for
>regex using j.u.regex from 1.4, or leveraging features only available in
>JDK1.5 such as j.u.concurrent?
Do you know how the current codebase break out in terms of JVM version?
Which code is currently needed support 1.3 environments?
How would you handle interfaces or base classes that are common to more than
one JVM version if the descendent class is specific to, say, 1.3? Would you
duplicate the base classes in each jar?
Re: Single JDK14 compile model?
Posted by Daniel John Debrunner <dj...@debrunners.com>.
Daniel John Debrunner wrote:
> If the community wants Derby to support J2ME that Derby developers have
> to have an easy way to produce code that works in J2ME.
The key point & I mess it up, "then", not "that". :-(
If the community wants Derby to support J2ME then Derby developers have
to have an easy way to produce code that works in J2ME.
Dan.
Re: Single JDK14 compile model?
Posted by RPost <rp...@pacbell.net>.
>"Daniel John Debrunner" wrote:
> . . . A good detailed explanation of exactly what the issues are . . .
Thanks for taking the time to provide the details. It is really helpful for
making sure we focus on the real need.
Re: Single JDK14 compile model?
Posted by Daniel John Debrunner <dj...@debrunners.com>.
RPost wrote:
>>Jeremy Boynes wrote:
>>
>>Having one JAR for the database is nice, but not if the price is
>>post-compilation bytecode hacks.
>
>
> So how does everyone feelf about pre-compilation sourcecode hacks?
>
> Would someone please state the need a little more clearly? Exactly what
> class/classes need to be abstract for one purpose and concrete for another
> and why?
>
> It seems that the issue is:
>
> IF targetJVM=1.3 then classX needs to be concrete
> ELSEIF targetJVM=1.4 then classX needs to be abstract
That's not quite correct. Really it would be
In JDK 1.3 classX needs to be concrete
But in JDK 1.4 classX would be seen as abstract when *compiled*
This is only a build issue, not a run-time issue, not a packaging issue.
Let's take real code.
Derby needs implementations of java.sql.* classes for JSR169, JDBC 2.1
and JDBC 3.0. JSR169 and JDBC 2.1 are (different) subsets of JDBC 3.0.
The simple case is where a single implementation class can satisfy all
of these JDBC requirements. Examples are EmbedStatement and
EmbedDatabaseMetaData. In these cases the additional methods in JDBC 3.0
only use classes (or primitives) that exist in JSR169 and JDBC 2.1
environments. E.g. when running in JDBC 2.1, EmbedStatement has more
methods (the JDBC 3.0 ones) than it needs to, but they are not
accessible to applications since they are accessing the class through
the JDBC 2.1 java.sql.Statement interface.
The case that causes the problem is where a method in JDBC 3.0
references a class that does not exist in JSR169 and/or JDBC 2.1
environments. This could either be a class added by JDBC 3.0, added by
JDK 1.4 or not available in the JSR 169 J2ME environment. Examples of
these methods are:
PreparedStatement.getParameterMetaData - returns ParameterMetaData
interface, which is added in JDBC 3.0
ResultSet.getBigDecimal - returns BigDecimal which does not exist in
the JSR169 environment.
So Derby implements PreparedStatement (and the others are similar) by
having:
EmbedPreparedStatement - JSR169
EmbedPreparedStatement20 extends EmbedPreparedStatement - JDBC 2.1
EmbedPreparedStatement30 extends EmbedPreparedStatement20 - JDBC 3.0
All of these need to be concrete so that instances can be created in
their required environments (J2ME/CDC/Foundation, JDK 1.3 and JDK 1.4).
[note as checked in (in svn) EmbedPreparedStatement is abstract as
that's what I'm trying to solve with this build discussion]
These classes correctly implement the contract for
java.sql.PreparedStatement in their own environment, but would cause
*compile* issues in the other environments, e.g.
compiling EmbedPreparedStatement20 in JDK 1.4 will fail as the
compiler will indicate it does not implement all the required methods
for PreparedStatement interfaces, "it should be abstract".
compiling EmbedPreparedStatement30 in a J2ME or JDK 1.3 environment
will fail as it depends on classes that do not exist in that environment.
Soooooo, I can think of two solutions, maybe there are others, but this
is what I can think of :-)
A) Compile each file using the correct jars files for its environment,
and control the build order to be:
1) EmbedPreparedStatement with J2ME/CDC/Foundation & JSR169 jars
2) EmbedPreparedStatement20 with JDK 1.3 jars
3) EmbedPreparedStatement30 with JDK 1.4 jars
This is what is done today, though the first step is with the JDK 1.3
jars as the J2ME is a work in progress.
B) Compile in a single environment JDK 1.4 and to avoid the compiliation
failures make the classes abstract in the code and modify the byte code
after compilation.
I'm willing to go with A) or B) and I prefer A) with the main issue
being ii) below. The reason I proposed B as an alternative, was for two
reasons:
i) to simplify life for new developers of Derby, rather than requiring
them to download basically three JDK setups, just require a single one.
And continuing with this approach will mean four environments shortly,
as Derby takes advantage of JDK 1.5 features.
ii) I don't see a way for developers to easily download the required
J2ME environment, there don't seem to be reference SDK's for this setup
yet. Thus I can't check in final J2ME changes, which will delay adoption
of Derby in the J2ME market.
So, to repeat, this is a build environment issue for Derby, not a
packaging or multiple code line issue.
If the community wants Derby to support J2ME that Derby developers have
to have an easy way to produce code that works in J2ME.
That's the core issue, and it's the same even if have a single codeline,
or a special codeline for J2ME, and it's the same issue if we have a
single jar for Derby or a jar per environment.
For the full details of how Derby will implement JDBC classes, see the
design doc in this Jira bug.
http://issues.apache.org/jira/browse/DERBY-97
Dan.
Re: Single JDK14 compile model?
Posted by Satheesh Bandaram <sa...@Sourcery.Org>.
I am not sure if each platform module would be small... Unless we are
talking about a really small number of files, this could cause
maintenance issues. Even in closed development enviornments, products
take extreme care to avoid code duplication.. It might be a bigger
challenge in open source setup... Having three (ME, JDK1.3, JDK1.4) or
even four (with JDK1.5) versions of the code doesn't seem right.
It is a good goal to have developpers develop on one platform. Not sure
what it the right way to get there ...
Satheesh
Jeremy Boynes wrote:
> If we had three small modules for the JDBC API impls then they could
> be compiled once under the appropriate JDK and then just referenced by
> others as needed (e.g. during packaging). The amount of code in each
> would be small and stagnant so there would be no need to build from
> source all the time.
>
> Under this model, I could do all my development and testing using 1.4
> and the JDCB3.0 modules and you would do all your development and
> testing using J2ME and the JSR169 modules. We take care in shared code
> not to use features that are not available in the other platform. We
> both make progress, and rapidly because we can concentrate on the
> feature we're working on rather than the cross-platform build.
>
> Every so often, say nightly, an automated process builds all versions
> on all platforms so as a whole we catch any issues with inappropriate
> dependencies or non-portable code. That process may be Gump, it may be
> something else.
>
> For this to work, we would need to break things down so that a
> developer could work with just one platform at a time. I think that
> means moving away from the one-jar-fits-all model to a
> package-by-platform model.
>
> --
> Jeremy
>
>
>
Re: Single JDK14 compile model?
Posted by Jeremy Boynes <jb...@apache.org>.
Daniel John Debrunner wrote:
> Jeremy Boynes wrote:
>
>
>>Daniel John Debrunner wrote:
>
>
>>>Hopefully I made it clear in my other e-mail, but the issue is not how
>>>the code is laid out, but do we want to require Derby contributors &
>>>committers to download multiple JDKs to build Derby, or a single one?
>>>And how do I make progress on J2ME if developers cannot download a J2ME
>>>environment?
>>>
>>
>>I still see this as an effect of the one-jar-fits-all model that
>>requires everything to be built all the time.
>
>
> I think we are getting closer to understanding each other, I still
> disagree that it's a packaging issue, but it does arise from the
> requirement that a single compile environment produce all the code for
> all the platforms.
>
:-)
>
>>Under this model, I could do all my development and testing using 1.4
>>and the JDCB3.0 modules and you would do all your development and
>>testing using J2ME and the JSR169 modules.
>
>
> So this would be a change from the current Derby model. I'm new to open
> source so I'm unclear how this would work in practise. If a contributor
> or committer working only in JDK 1.4 submits or commits a patch from
> that breaks the build in the JDK 1.3 and/or J2ME, what happens?
>
> Would the patch be veto'ed until it compiles in all the environments? If
> so, then there is an implicit requirement for all developers to have all
> the compile environments and the process is more manual, thus more error
> prone.
>
> Or is the build in the other environments simply broken until someone
> who cares about that platform fixes it? Bad for quality as other
> undetected problems can be added to that environment while the build is
> failing.
>
> That situation cannot arise today because of the single compile model.
>
Yes it is a change in the model.
It deals with the issue that many developers will not have access to all
platforms or environments. Often they are working from a single machine
(like now, all I have access to is a laptop) and that may not even run
all environments.
Often the only way a change can be tested is for it to be committed and
then built and run by some volunteer on a platform they own; a few core
(tier 1) platforms may be available on a continuous basis (a la gump or
continuum) but others may not be available until someone decides to
download and test a beta release.
One factor here is the time "until someone who cares about that platform
fixes it" is usually very very short - broken builds don't linger
because its damned inconvenient and also rather embarassing. No veto is
needed, its cultural. Usually the fix is done by the original developer.
Is quality affected? I don't believe so because everyone sees the
problem and any defect is corrected quickly.
>
>
>>We take care in shared code
>>not to use features that are not available in the other platform. We
>>both make progress, and rapidly because we can concentrate on the
>>feature we're working on rather than the cross-platform build.
>
>
> Once you downloaded the initial requirements to build Derby, how much
> time have you spent worrying about the cross-platform build, which
> occurs automatically? Beyond the initial download I believe it is not a
> time drain.
>
Once it's all up and running then things are fine. However, there is a
fair bit of complexity in the build to deal with compiling specific bits
for specific JVMs; as we do more and more to deal with J2ME issues or to
leverage bits of 1.4 and 1.5 then this is only going to get worse. It
may well be that eventually the everything-in-one-build model may not
work at all - for example, is there even an J2ME or JDK1.5 JDK available
for OSX (a platform used by a lot of open source developers)?
These types of issues affect all sizable open source projects and
breaking them down into separately developable modules is very common
practise - e.g. HTTPD, Tomcat/Jakarta Commons, Geronimo, Ant, Maven are
examples just at Apache alone.
--
Jeremy
Re: Single JDK14 compile model?
Posted by Daniel John Debrunner <dj...@debrunners.com>.
Jeremy Boynes wrote:
> Daniel John Debrunner wrote:
>> Hopefully I made it clear in my other e-mail, but the issue is not how
>> the code is laid out, but do we want to require Derby contributors &
>> committers to download multiple JDKs to build Derby, or a single one?
>> And how do I make progress on J2ME if developers cannot download a J2ME
>> environment?
>>
>
> I still see this as an effect of the one-jar-fits-all model that
> requires everything to be built all the time.
I think we are getting closer to understanding each other, I still
disagree that it's a packaging issue, but it does arise from the
requirement that a single compile environment produce all the code for
all the platforms.
> Under this model, I could do all my development and testing using 1.4
> and the JDCB3.0 modules and you would do all your development and
> testing using J2ME and the JSR169 modules.
So this would be a change from the current Derby model. I'm new to open
source so I'm unclear how this would work in practise. If a contributor
or committer working only in JDK 1.4 submits or commits a patch from
that breaks the build in the JDK 1.3 and/or J2ME, what happens?
Would the patch be veto'ed until it compiles in all the environments? If
so, then there is an implicit requirement for all developers to have all
the compile environments and the process is more manual, thus more error
prone.
Or is the build in the other environments simply broken until someone
who cares about that platform fixes it? Bad for quality as other
undetected problems can be added to that environment while the build is
failing.
That situation cannot arise today because of the single compile model.
> We take care in shared code
> not to use features that are not available in the other platform. We
> both make progress, and rapidly because we can concentrate on the
> feature we're working on rather than the cross-platform build.
Once you downloaded the initial requirements to build Derby, how much
time have you spent worrying about the cross-platform build, which
occurs automatically? Beyond the initial download I believe it is not a
time drain.
Dan.
Re: Single JDK14 compile model?
Posted by Jeremy Boynes <jb...@apache.org>.
Daniel John Debrunner wrote:
> Jeremy Boynes wrote:
>
>>I think there are two basic problems here:
>>
>>1) How do we provide implementations of the JDBC interfaces?
>> The problem here is that JDBC3.0 (JDK1.4/1.5) added classes
>> to the API in java.sql that are not present in JDK1.3 or JSR169
>> JVMs.
>>
>> Perhaps the solution here is to have two separate modules that
>> implement the APIs and which are compiled with the appropriate VM.
>> We can avoid post-compilation hacks and although we could use source
>> code pre-processing I would suggest it may not be necessary as the
>> modules would mostly just be delegating to internal Derby classes
>> making it trivial to support two separate code trees.
>
>
> Hopefully I made it clear in my other e-mail, but the issue is not how
> the code is laid out, but do we want to require Derby contributors &
> committers to download multiple JDKs to build Derby, or a single one?
> And how do I make progress on J2ME if developers cannot download a J2ME
> environment?
>
I still see this as an effect of the one-jar-fits-all model that
requires everything to be built all the time.
If we had three small modules for the JDBC API impls then they could be
compiled once under the appropriate JDK and then just referenced by
others as needed (e.g. during packaging). The amount of code in each
would be small and stagnant so there would be no need to build from
source all the time.
Under this model, I could do all my development and testing using 1.4
and the JDCB3.0 modules and you would do all your development and
testing using J2ME and the JSR169 modules. We take care in shared code
not to use features that are not available in the other platform. We
both make progress, and rapidly because we can concentrate on the
feature we're working on rather than the cross-platform build.
Every so often, say nightly, an automated process builds all versions on
all platforms so as a whole we catch any issues with inappropriate
dependencies or non-portable code. That process may be Gump, it may be
something else.
For this to work, we would need to break things down so that a developer
could work with just one platform at a time. I think that means moving
away from the one-jar-fits-all model to a package-by-platform model.
--
Jeremy
Re: Single JDK14 compile model?
Posted by Daniel John Debrunner <dj...@debrunners.com>.
Jeremy Boynes wrote:
> I think there are two basic problems here:
>
> 1) How do we provide implementations of the JDBC interfaces?
> The problem here is that JDBC3.0 (JDK1.4/1.5) added classes
> to the API in java.sql that are not present in JDK1.3 or JSR169
> JVMs.
>
> Perhaps the solution here is to have two separate modules that
> implement the APIs and which are compiled with the appropriate VM.
> We can avoid post-compilation hacks and although we could use source
> code pre-processing I would suggest it may not be necessary as the
> modules would mostly just be delegating to internal Derby classes
> making it trivial to support two separate code trees.
Hopefully I made it clear in my other e-mail, but the issue is not how
the code is laid out, but do we want to require Derby contributors &
committers to download multiple JDKs to build Derby, or a single one?
And how do I make progress on J2ME if developers cannot download a J2ME
environment?
Dan.
Re: Single JDK14 compile model?
Posted by Jeremy Boynes <jb...@apache.org>.
RPost wrote:
>>Jeremy Boynes wrote:
>>
>>Having one JAR for the database is nice, but not if the price is
>>post-compilation bytecode hacks.
>
>
> So how does everyone feelf about pre-compilation sourcecode hacks?
>
> Would someone please state the need a little more clearly?
I think there are two basic problems here:
1) How do we provide implementations of the JDBC interfaces?
The problem here is that JDBC3.0 (JDK1.4/1.5) added classes
to the API in java.sql that are not present in JDK1.3 or JSR169
JVMs.
Perhaps the solution here is to have two separate modules that
implement the APIs and which are compiled with the appropriate VM.
We can avoid post-compilation hacks and although we could use source
code pre-processing I would suggest it may not be necessary as the
modules would mostly just be delegating to internal Derby classes
making it trivial to support two separate code trees.
This is very similar to how the build works now. We may be able to
simplify this by building the API implementations totally separately
so most of the time people would just use the one for their platform.
2) How do we provide different configurations for distribution?
Essentially, are we sticking with the one-jar-fits-all model or
should we break things down into separate modules that can be built
and packaged separately. In the separate module model, different
versions could be built for different platforms and assembled into
distributable configurations as needed but users would need to
select the configuration applicable to their environment.
If done right, the separate module approach can work very well e.g.
Apache HTTPD, Eclipse, Maven and is well suited to the distributed
development model that is open source; it can also go horribly wrong if
not managed properly.
Back last year I did some experimentation with Maven about breaking down
the build into a module structure. If anyone is interested I could try
and finish that up so folks can have a look.
--
Jeremy
Re: Single JDK14 compile model?
Posted by RPost <rp...@pacbell.net>.
>Jeremy Boynes wrote:
>
> Having one JAR for the database is nice, but not if the price is
> post-compilation bytecode hacks.
So how does everyone feelf about pre-compilation sourcecode hacks?
Would someone please state the need a little more clearly? Exactly what
class/classes need to be abstract for one purpose and concrete for another
and why?
It seems that the issue is:
IF targetJVM=1.3 then classX needs to be concrete
ELSEIF targetJVM=1.4 then classX needs to be abstract
Assuming the above is the issue I would prefer a hack that introduces
conditional compilation using a mechanism like Ant or C or other:
//IF target=1.3
public class X . . .
//ELSEIF target=1.4
public abstract class X . . .
//ENDIF
The a precompilation filter can select the declaration that is needed.
At least this documents the source code with exactly what the problem is and
how it is being solved.
Re: Single JDK14 compile model?
Posted by Jeremy Boynes <jb...@apache.org>.
Daniel John Debrunner wrote:
>>Daniel John Debrunner wrote:
>>
>>>The trick I then use is to modify the class file sometime after
>>>compilation to mark it concrete by modifying the byte code in the .class
>>>file (using Derby's class file utilities). This does not cause any
>>>problems with the JVM (though there are some issues with the current
>>>version of jikes that Derby uses).
>
> Jeremy Boynes wrote:
>
>>This really really worries me and I am very uncomfortable with it.
>
> Can you expand on your concerns?
>
Having grown up patching object code on mainframes, any form of
manipulation of the binary concerns me. Yes, this is a fairly simple
modification but nevertheless it is the principle that is troublesome.
The obvious issue is that the object will no longer match the source -
if I examine the class I see that it is concrete, if I examine the
source then it is abstract. All the doco in the world is just a band-aid
over that difference.
There is considerable reluctance in enterprise environments about any
form of byte code manipulation. Enterprise users do not like obfuscation
as it makes support and diagnostics harder and has no benefit to them.
They have serious concerns about frameworks that manipulate bytecode
directly, for example the enhancers in JDO or the instrumentation done
by AOP frameworks.
In this case all is doing is allowing us to support JDK1.3 and JDK1.4
(or specifically JSR169, JDBC2.0 and JDBC3.0) in one jar. Most users,
embedded, ISV or enterprise, are running in a configuration where the
JVM version is known and can be matched to a suitable database JAR.
Having one JAR for the database is nice, but not if the price is
post-compilation bytecode hacks.
--
Jeremy
Re: Single JDK14 compile model?
Posted by Daniel John Debrunner <dj...@debrunners.com>.
> Daniel John Debrunner wrote:
>
>>
>> The trick I then use is to modify the class file sometime after
>> compilation to mark it concrete by modifying the byte code in the .class
>> file (using Derby's class file utilities). This does not cause any
>> problems with the JVM (though there are some issues with the current
>> version of jikes that Derby uses).
>>
Jeremy Boynes wrote:
> This really really worries me and I am very uncomfortable with it.
Can you expand on your concerns?
And Andrew McIntyre added:
> I'd rather put the time in myself to keeping our
> build running in Gump than deal with problems caused by modifying the
> classfiles at build time (I'm thinking of all the fun I've had with
> obfuscation).
Marking an existing class file as not abstract is a zero risk operation
to the structure of an existing classfile. The operation is simply
changing one bit in the access_flags field in the header of the class.
The old Cloudscape obfuscation tool did a huge amount of manipulation of
the class file, including modifying the byte code stream. Because of
that experience I intentionally made sure this process was zero risk and
didn't require huge changes in the class file.
For an existing class file the on-disk structure (the class file format)
is deterministic, thus a double check could be added to the "mark as
concrete" process to ensure every byte of the class file matches the old
version with the exception of the access_flags. Or maybe even simpler,
just write a explicit program that only modifies that byte at the
correct offset in the file, in place.
Dan.
Re: Single JDK14 compile model?
Posted by Jeremy Boynes <jb...@apache.org>.
Daniel John Debrunner wrote:
>
> The trick I then use is to modify the class file sometime after
> compilation to mark it concrete by modifying the byte code in the .class
> file (using Derby's class file utilities). This does not cause any
> problems with the JVM (though there are some issues with the current
> version of jikes that Derby uses).
>
This really really worries me and I am very uncomfortable with it.
> So is this direction ok for Derby? Along with Jeremy checking in the
> Apache jar files required by Derby, it would make downloading and
> compling Derby much easier.
>
> Looking at the two approaches, here are the trade-offs:
>
> Mulitple JDKs
>
> + enforces correct sub-setting at development time, enforced by the
> compiler, e.g. correct methods in JDBC implementations, not using JDK
> 1.4 classes in a JDK 1.3 environment, not using non J2ME classes in J2ME
> code.
>
> - tricky (maybe impossible with J2ME) to work with Gump
>
> - tricky for the developer to get started on Derby
>
> - J2ME libraries not (easily) available
>
>
> Single JDK
>
> - correct implementations only enforced by testing in the given
> environment, e.g. JDK 1.3 J2ME.
>
> - requires more awareness for contributors working in the code
> (e.g. not to use JDK 1.4 classes in code that might be used in J2ME).
>
> + simple for Gump (hopefully)
>
> + simple for the developer to set up
>
> Comments?
A lot of the problems here arise because we are trying to support 3
different JVM revision levels with one output JAR. Is this time to
consider generating multiple output JARs, one for each VM?
This would also allow us to tune the implementations for each VM
configuration e.g. eliminating code entirely for features not
supportable/wanted in J2ME environments (reducing the code footprint),
replacing SANE/INSANE with assertions in 1.4 and up, native support for
regex using j.u.regex from 1.4, or leveraging features only available in
JDK1.5 such as j.u.concurrent?
The testing burden for this is no different as production releases would
need to be tested on all VMs anyway. It is a higher burden for
developers as fixes would need to be done in all code-lines (assuming
some divergance).
--
Jeremy
Re: Single JDK14 compile model?
Posted by Daniel John Debrunner <dj...@debrunners.com>.
Jeremy Boynes wrote:
> Daniel John Debrunner wrote:
>
>>
>> So I looked at an alternate approach where all the code is compiled
>> using a single JDK, requiring at least JDK 1.4 though I'm only tested
>> with 1.4 so far. This requires making some classes abstract that must
>> not be abstract in J2ME or JDK 1.3. EmbedResultSet is an example. This
>> allows the class to then compile under JDK 1.4.
>>
>
> I did some playing around came up with a small IdeaJ project that
> implements a couple of the API classes (CallableStatement and
> ResultSet). This compiles under 1.4 (with target=1.3) and runs under 1.4
> (with 3.0 features) and 1.3 (without).
>
> Code can be downloaded from http://www.apache.org/~jboynes/vmtest.jar
>
> Is this a solution to the VM compatibility issue or is there a case I am
> missing?
This is somewhat similar to what Derby does today, with one exception.
Your CallableStatement implementation references a class,
ParameterMetaData, that will not exist in jdk 1.3 environments.
The issue then becomes, is it *guaranteed* that any 1.3 JVM will not
throw a class not found exception when it loads this implementation?
With Cloudscape we saw in the past problems with this approach, with
JVMs that did aggressive pre-loading of classes (as they are allowed to)
and when byte code verification is enabled.
That's why the approach was taken to ensure that in any enviromnent that
only classes that could be loaded through code were ones that were
guaranteed to be there.
Dan.
Re: Single JDK14 compile model?
Posted by Jeremy Boynes <jb...@apache.org>.
Daniel John Debrunner wrote:
>
> So I looked at an alternate approach where all the code is compiled
> using a single JDK, requiring at least JDK 1.4 though I'm only tested
> with 1.4 so far. This requires making some classes abstract that must
> not be abstract in J2ME or JDK 1.3. EmbedResultSet is an example. This
> allows the class to then compile under JDK 1.4.
>
I did some playing around came up with a small IdeaJ project that
implements a couple of the API classes (CallableStatement and
ResultSet). This compiles under 1.4 (with target=1.3) and runs under 1.4
(with 3.0 features) and 1.3 (without).
Code can be downloaded from http://www.apache.org/~jboynes/vmtest.jar
Is this a solution to the VM compatibility issue or is there a case I am
missing?
--
Jeremy
Re: Single JDK14 compile model?
Posted by Shreyas Kaushik <Sh...@Sun.COM>.
My answers are inline.
~ Shreyas
Daniel John Debrunner wrote:
>The current compile model for Derby uses a JDK 1.3 and JDK 1.4. Those
>classes that are required to run in JDK1.3/JDBC 2.0 are compiled against
>the JDK 1.3 libraries, and the JDK 1.4/1.5/JDBC 3.0 are compiled against
>the JDK 1.4 libraries. This requires developers to download and setup
>two JDKs and some extra libraries from JDK 1.3 extensions.
>
>While this model worked well for closed source (all the setup was
>handled automatically), it's no so good for open source. This is because
>of putting the burden on developers to download extra stuff.
>
>I was trying to expand this model to support J2ME/CDC/Foundation but I
>am now having doubts, mainly due to the requirement at Apache to be part
>of Gump. Andrew had to modify various build.xml files (adding near
>duplicate actions) and make the gump build a multi-pass affair. I don't
>see how I can add J2ME support in this mode while also keeping Gump running.
>
>So I looked at an alternate approach where all the code is compiled
>using a single JDK, requiring at least JDK 1.4 though I'm only tested
>with 1.4 so far. This requires making some classes abstract that must
>not be abstract in J2ME or JDK 1.3. EmbedResultSet is an example. This
>allows the class to then compile under JDK 1.4.
>
>The trick I then use is to modify the class file sometime after
>compilation to mark it concrete by modifying the byte code in the .class
>file (using Derby's class file utilities). This does not cause any
>problems with the JVM (though there are some issues with the current
>version of jikes that Derby uses).
>
>
I am not comfortable with doing this. What will be the impact on
portability of code.
If there are some things in the class files that may differ from 1.3.*
to 1.4.*, how would
this be handled ?
We also need to see the impact with 1.5.* .
>So is this direction ok for Derby? Along with Jeremy checking in the
>Apache jar files required by Derby, it would make downloading and
>compling Derby much easier.
>
>Looking at the two approaches, here are the trade-offs:
>
>Mulitple JDKs
>
>+ enforces correct sub-setting at development time, enforced by the
>compiler, e.g. correct methods in JDBC implementations, not using JDK
>1.4 classes in a JDK 1.3 environment, not using non J2ME classes in J2ME
>code.
>
>- tricky (maybe impossible with J2ME) to work with Gump
>
>- tricky for the developer to get started on Derby
>
>- J2ME libraries not (easily) available
>
>
>Single JDK
>
>- correct implementations only enforced by testing in the given
>environment, e.g. JDK 1.3 J2ME.
>
>- requires more awareness for contributors working in the code
>(e.g. not to use JDK 1.4 classes in code that might be used in J2ME).
>
>+ simple for Gump (hopefully)
>
>+ simple for the developer to set up
>
>Comments?
>Dan.
>
>
>