Implementation comparisons

[George Harley1 <GH...@uk.ibm.com>]

Hi, 

I would like to ask for some opinions on measuring how "correct" a new 
implementation of a J2SE class is. Consider if someone develops a clean 
implementation of fictitious J2SE type java.foo.Bar which is based on 
their interpretation of the reference API documentation. Like the 
deliverable code, we can expect the unit test code to also be written 
based on assumptions made from the reference API. Is this working model 
good enough to ensure that the resulting type can be used as a one-for-one 
replacement for the equivalent implementation in the reference 
distribution ? What about the behaviour that is not adequately captured in 
the existing documentation ? For example, the precise layout of the result 
from the toString() method ; the order in which items are returned when 
iterating through a collection property etc. 

It is likely safe to assume that users running an application on 
alternative class libraries will not be expecting to make many (any ?) 
changes to that application code if it was previously working as desired. 
So in the general case then how can subtle behavioural incompatibilities 
*between* API implementations be discovered, recorded and remedied ? I 
would be interested in hearing about any tooling or utilities that can 
help with this issue. 

Or perhaps this is not an issue at all .....? 

Best regards, 
George

Re: Implementation comparisons

[Dalibor Topic <ro...@kaffe.org>]

Tim Ellison wrote:
> Dalibor Topic wrote:
> 
>>On Sat, Nov 05, 2005 at 05:58:20PM +0000, Tim Ellison wrote:
>>
>>
>>>>Luckily there are several in depth books about
>>>>various parts of the core library. O'Reilly and Addison Wesley publish
>>>>some very good titles. Since real programmers use these books and the
>>>>examples they give they are often a more solid base to work from.
>>>
>>>At the risk of sounding boring, it is worth noting that the books'
>>>material is usually copyrighted and licensed too -- so we have to be
>>>careful not to copy examples from any reference material into Harmony's
>>>implementation or test suites where the license is incompatible with the
>>>ASL.
>>
>>
>>Licenses in real, printed books?
>>
>>I've only got three words: first sale doctrine. 
> 
> 
> It's not a restriction on the dissemintation of the info in the books
> that is limited, but copying verbatim a book's sample code into a test
> suite or as an implementation of the class library is probably bad form.
> 

Full ACK.

I am sorry for having misinterpreted your original comment to think it'd
be about EULAs on printed books.

cheers,
dalibor topic

Re: Implementation comparisons

[Tim Ellison <t....@gmail.com>]

Dalibor Topic wrote:
> On Sat, Nov 05, 2005 at 05:58:20PM +0000, Tim Ellison wrote:
> 
>>>Luckily there are several in depth books about
>>>various parts of the core library. O'Reilly and Addison Wesley publish
>>>some very good titles. Since real programmers use these books and the
>>>examples they give they are often a more solid base to work from.
>>
>>At the risk of sounding boring, it is worth noting that the books'
>>material is usually copyrighted and licensed too -- so we have to be
>>careful not to copy examples from any reference material into Harmony's
>>implementation or test suites where the license is incompatible with the
>>ASL.
> 
> 
> Licenses in real, printed books?
> 
> I've only got three words: first sale doctrine. 

It's not a restriction on the dissemintation of the info in the books
that is limited, but copying verbatim a book's sample code into a test
suite or as an implementation of the class library is probably bad form.

To produce a good compatible implementation includes all of following
the JavaDoc spec, behaving to the language paradigms as described in the
books, passing the JCK, implementing the functional (and non-functional)
 expectations of applications.

By the same token, we should expect to have to occasionally run
compatibility comparison tests on the reference implementation (RI) APIs
to determine behavior that is not covered well in the specification --
but certainly not infringe on the RI by reverse engineering.

Regards,
Tim



-- 

Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.

Re: Implementation comparisons

[Dalibor Topic <ro...@kaffe.org>]

Mark Wielaard wrote:
> Hi Dalibor,
> 
> On Sat, 2005-11-05 at 11:02 -0800, Dalibor Topic wrote:
> 
>>On Sat, Nov 05, 2005 at 05:58:20PM +0000, Tim Ellison wrote:
>>
>>>>Luckily there are several in depth books about
>>>>various parts of the core library. O'Reilly and Addison Wesley publish
>>>>some very good titles. Since real programmers use these books and the
>>>>examples they give they are often a more solid base to work from.
>>>
>>>At the risk of sounding boring, it is worth noting that the books'
>>>material is usually copyrighted and licensed too -- so we have to be
>>>careful not to copy examples from any reference material into Harmony's
>>>implementation or test suites where the license is incompatible with the
>>>ASL.
>>
>>Licenses in real, printed books?
>>
>>I've only got three words: first sale doctrine. 
> 
> 
> O, come on. You know what Tim means. Normally books don't allow copying
> of their text for arbitrary purposes. You cannot just copy large chunks
> of descriptions or example code from books without the agreement of the
> publisher first. The main point is that the javadoc as published isn't
> really a spec (even if some people call it that way). You do need real
> books that describe what the stuff does that we want to be compatible
> with. In the end it only matters what real programs do and real programs
> are based on real descriptions of the core library as described in these
> books.

Yep. Sorry for the confusion, it was a misunderstanding on my side. :(

cheers,
dalibor topic

Re: Implementation comparisons

[Mark Wielaard <ma...@klomp.org>]

Hi Dalibor,

On Sat, 2005-11-05 at 11:02 -0800, Dalibor Topic wrote:
> On Sat, Nov 05, 2005 at 05:58:20PM +0000, Tim Ellison wrote:
> > > Luckily there are several in depth books about
> > > various parts of the core library. O'Reilly and Addison Wesley publish
> > > some very good titles. Since real programmers use these books and the
> > > examples they give they are often a more solid base to work from.
> > 
> > At the risk of sounding boring, it is worth noting that the books'
> > material is usually copyrighted and licensed too -- so we have to be
> > careful not to copy examples from any reference material into Harmony's
> > implementation or test suites where the license is incompatible with the
> > ASL.
> 
> Licenses in real, printed books?
> 
> I've only got three words: first sale doctrine. 

O, come on. You know what Tim means. Normally books don't allow copying
of their text for arbitrary purposes. You cannot just copy large chunks
of descriptions or example code from books without the agreement of the
publisher first. The main point is that the javadoc as published isn't
really a spec (even if some people call it that way). You do need real
books that describe what the stuff does that we want to be compatible
with. In the end it only matters what real programs do and real programs
are based on real descriptions of the core library as described in these
books.

Cheers,

Mark

-- 
Escape the Java Trap with GNU Classpath!
http://www.gnu.org/philosophy/java-trap.html

Join the community at http://planet.classpath.org/

Re: Implementation comparisons

[George Harley1 <GH...@uk.ibm.com>]

> I've only got three words: first sale doctrine. 

Hi Dalibor, 

IANAL but from the little I know on first sale doctrine (how little ? very 
little) it seems a bit of a gamble to cite it as a defence since it 
appears to be applied so arbitrarily. Regardless, I'm not sure *any* 
project would benefit from getting entangled in such a case. Something 
else to consider is that book example code is often not the best/most 
efficient way to accomplish a given task but has usually been carefully 
constructed to aid discussion of a number of key points that the author 
wishes to make. As Mark wrote, they help give a "more solid base" to 
reason from - they should not be taken as an answer or as an extension of 
a specification in themselves. 

My 2c anyway...

Best regards, 
George





Dalibor Topic <ro...@kaffe.org> 
05/11/2005 19:02
Please respond to
harmony-dev@incubator.apache.org


To
harmony-dev@incubator.apache.org
cc

Subject
Re: Implementation comparisons






On Sat, Nov 05, 2005 at 05:58:20PM +0000, Tim Ellison wrote:
> > Luckily there are several in depth books about
> > various parts of the core library. O'Reilly and Addison Wesley publish
> > some very good titles. Since real programmers use these books and the
> > examples they give they are often a more solid base to work from.
> 
> At the risk of sounding boring, it is worth noting that the books'
> material is usually copyrighted and licensed too -- so we have to be
> careful not to copy examples from any reference material into Harmony's
> implementation or test suites where the license is incompatible with the
> ASL.

Licenses in real, printed books?

I've only got three words: first sale doctrine. 

cheers,
dalibor topic

Re: Implementation comparisons

[Dalibor Topic <ro...@kaffe.org>]

On Sat, Nov 05, 2005 at 05:58:20PM +0000, Tim Ellison wrote:
> > Luckily there are several in depth books about
> > various parts of the core library. O'Reilly and Addison Wesley publish
> > some very good titles. Since real programmers use these books and the
> > examples they give they are often a more solid base to work from.
> 
> At the risk of sounding boring, it is worth noting that the books'
> material is usually copyrighted and licensed too -- so we have to be
> careful not to copy examples from any reference material into Harmony's
> implementation or test suites where the license is incompatible with the
> ASL.

Licenses in real, printed books?

I've only got three words: first sale doctrine. 

cheers,
dalibor topic

Re: Implementation comparisons

[Tim Ellison <t....@gmail.com>]

Mark Wielaard wrote:
> Hi George,
> 
> On Thu, 2005-11-03 at 15:41 -0600, Archie Cobbs wrote:
> 
>>George Harley1 wrote:
>>
>>>I would like to ask for some opinions on measuring how "correct" a new 
>>>implementation of a J2SE class is. Consider if someone develops a clean 
>>
>>Very good question and one that is asked (and sometimes answered)
>>in various forms all the time on the Classpath mailing list. The
>>approach there has been pragmatic.. basically the comparision is
>>of course always versus what the JDK does. If the spec says nothing
>>but JDK behaves like X, then Classpath typically behaves like X, etc.
>>If the spec says X and the JDK does Y, then Classpath usually does X.
>>
>>The harder cases are when the JDK behavior and spec disagree, or the
>>JDK has a bug but applications rely on that buggy behavior, etc.
>>These are taken case by case.
> 
> 
> Yes, and you should always work from real books and not just the javadoc
> that IBM and Sun publish publicly describing the core library.
> Unfortunately those descriptions are not really concise and clear enough
> to be used as spec. 

I agree with the observation, and yet they are 'the spec'.

> Luckily there are several in depth books about
> various parts of the core library. O'Reilly and Addison Wesley publish
> some very good titles. Since real programmers use these books and the
> examples they give they are often a more solid base to work from.

At the risk of sounding boring, it is worth noting that the books'
material is usually copyrighted and licensed too -- so we have to be
careful not to copy examples from any reference material into Harmony's
implementation or test suites where the license is incompatible with the
ASL.

> And it involves Testing, Testing and Testing!
> In the end it matters what real world applications do and need. So
> before all you should test against applications (preferably Free
> Software applications that you can study the source code of). Then
> whenever you find the documentation unclear or when writing new code you
> should write a Mauve unit test to make sure any corner cases are handled
> correctly and no regressions in behavior are introduced in the future.
> And of course whenever you find a real application that depends on a
> particular way the core library works and you decide to change the
> behavior of the library you have to clearly document that and again
> write a Mauve unittest.

Fully agree, of course to be called 'java' we have to pass the JCK too.
 There is no substitute for running key applications though.

Regards,
Tim

> Cheers,
> 
> Mark 
> 

-- 

Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.

Re: Implementation comparisons

[Mark Wielaard <ma...@klomp.org>]

Hi,

On Sat, 2005-11-05 at 11:25 +0100, Mark Wielaard wrote:
> Hi George,
>
> And it involves Testing, Testing and Testing!

Since I think this is very important I should probably add that lots of
applications these days come with unit tests themselves. We are trying
to run lots of autobuilders to catch any unexpected failures. See
http://lists.gnu.org/mailman/listinfo/classpath-testresults

Especially larger applications such as Eclipse or JOnAS give important
clues. http://people.redhat.com/overholt/eclipse/eclipse-tests/ and
http://people.redhat.com/~aph/current-jonas-on-gcj-conformance-test-results.html

Another nice framework is GUMP which can now be run against kaffe and
makes sure a whole bunch of apache programs build and pass their test
suites against the free compilers, runtimes and tools.
http://gump.zones.apache.org/gump/kaffe/

It often seems we are chasing numbers and GNU Classpath snapshot
releases often have humorous names like "95% and counting" because
people love tracking numbers about api coverage. But in reality it is
all about compatibility with real world applications!

Cheers,

Mark

-- 
Escape the Java Trap with GNU Classpath!
http://www.gnu.org/philosophy/java-trap.html

Join the community at http://planet.classpath.org/

Re: Implementation comparisons

[Mark Wielaard <ma...@klomp.org>]

Hi George,

On Thu, 2005-11-03 at 15:41 -0600, Archie Cobbs wrote:
> George Harley1 wrote:
> > I would like to ask for some opinions on measuring how "correct" a new 
> > implementation of a J2SE class is. Consider if someone develops a clean 
> 
> Very good question and one that is asked (and sometimes answered)
> in various forms all the time on the Classpath mailing list. The
> approach there has been pragmatic.. basically the comparision is
> of course always versus what the JDK does. If the spec says nothing
> but JDK behaves like X, then Classpath typically behaves like X, etc.
> If the spec says X and the JDK does Y, then Classpath usually does X.
> 
> The harder cases are when the JDK behavior and spec disagree, or the
> JDK has a bug but applications rely on that buggy behavior, etc.
> These are taken case by case.

Yes, and you should always work from real books and not just the javadoc
that IBM and Sun publish publicly describing the core library.
Unfortunately those descriptions are not really concise and clear enough
to be used as spec. Luckily there are several in depth books about
various parts of the core library. O'Reilly and Addison Wesley publish
some very good titles. Since real programmers use these books and the
examples they give they are often a more solid base to work from.

And it involves Testing, Testing and Testing!
In the end it matters what real world applications do and need. So
before all you should test against applications (preferably Free
Software applications that you can study the source code of). Then
whenever you find the documentation unclear or when writing new code you
should write a Mauve unit test to make sure any corner cases are handled
correctly and no regressions in behavior are introduced in the future.
And of course whenever you find a real application that depends on a
particular way the core library works and you decide to change the
behavior of the library you have to clearly document that and again
write a Mauve unittest.

Cheers,

Mark 

-- 
Escape the Java Trap with GNU Classpath!
http://www.gnu.org/philosophy/java-trap.html

Join the community at http://planet.classpath.org/

Re: Implementation comparisons

[Archie Cobbs <ar...@dellroad.org>]

George Harley1 wrote:
> I would like to ask for some opinions on measuring how "correct" a new 
> implementation of a J2SE class is. Consider if someone develops a clean 

Very good question and one that is asked (and sometimes answered)
in various forms all the time on the Classpath mailing list. The
approach there has been pragmatic.. basically the comparision is
of course always versus what the JDK does. If the spec says nothing
but JDK behaves like X, then Classpath typically behaves like X, etc.
If the spec says X and the JDK does Y, then Classpath usually does X.

The harder cases are when the JDK behavior and spec disagree, or the
JDK has a bug but applications rely on that buggy behavior, etc.
These are taken case by case.

-Archie

__________________________________________________________________________
Archie Cobbs      *        CTO, Awarix        *      http://www.awarix.com