You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by Geir Magnusson Jr <ge...@pobox.com> on 2006/03/22 12:41:56 UTC
Re: Unit testing revisited
Leo Simons wrote:
> Just two cents (or a little more) from the peanut gallery...
>
> On Tue, Mar 21, 2006 at 10:45:42PM -0500, Geir Magnusson Jr wrote:
>> Tim Ellison wrote:
>>> Just to clarify terminology -- unit tests are a 'style' of test that
>>> focus on particular units of functionality. Unit tests can be both
>>> implementation tests and API tests. Implementation tests are specific
>>> to our implementation (the mechanism, hidden to the end user, by which
>>> we chose to implement the APIs); and API tests are common to all
>>> conformant implementations (they test the APIs used by the end user).
>> So can we refer to "implementation tests" as "unit tests", because I
>> would bet that's a well-understood useage, and refer to things that are
>> strictly testing the API as "API tests".
>
> Thinking more about all this verbiage, and looking at a bunch of "unit
> tests" in many apache packages, I think the definitions are inherently
> too vague to get consensus on. It comes down to "what is a unit", and
> this is an age-old discussion (see: metric system vs inches) we should
> not try and have.
>
> It gets us into arguments like "that is not a proper unit test". 'Why
> not?' "The unit is too big." 'Well, our units are just bigger than yours,
> you silly Brits!' "Why you little...!"
>
> So I will suggest we don't go and try to define "unit test" and stop using
> the phrase when we want to make distinctions between stuff.
>
> Eg I would suggest that we bite the bullet and go something like this:
>
> "unit test" --> any test runnable by a "unit testing framework" such as
> JUnit or Cactus.
>
> "implementation test" --> a test run to verify that a specific piece
> of code, preferably as small a piece as is seperately
> testable, behaves as expected.
>
> "specification test" --> a test run to verify that an implementation is
> conformant with some specification, prefereably as small a piece
> of the specification for which a test can be defined.
>
> "API test" --> a specification test where the specification takes the
> form of an API definition (perhaps a java interface with
> supporting javadocs, perhaps just javadocs, perhaps IDL...)
>
> "tck test" --> any test defined as part of something that is called a
> "TCK" or technology compatibility kit. TCK tests are
> supposed to be specification tests.
>
You forgot one - "integration test", which is a unit test that's been
around long enough to shave. :) (It's actually not a unit test...)
These definitions are fine. The key is to separate out "implementation"
from spec/API, IMO.
>>> Geir Magnusson Jr wrote:
>>>> Good unit tests are going to be testing things that are package
>>>> protected. You can't do that if you aren't in the same package
>>>> (obviously).
>>> We have implementation tests that require package private, and maybe
>>> even private access to our implementation classes both in the java.* and
>>> o.a.h.* packages.
>
> This seems correct.
>
>>> The 'problem' is that we cannot define classes in java.* packages that
>>> are loaded by the application classloader. That is counter to
>>> specification and prohibited by the VM.
>>>
>>> We also have API tests that should not have access to package private
>>> and even private types in the implementation.
>
> This seems correct too.
>
>>> The 'problem' is that running API tests in java.* packages does provide
>>> such access, and worse runs those tests on the bootclassloader which
>>> gives them special security access not afforded to our users.
>
> This makes sense.
>
>>> I've said this lots of times before.
>
> Usually that means one is not coming across well, not that people aren't
> trying to listen or anything like that :-)
>
>>> We already see lots of errors caused by
>>> oversight of the classloader differences.
>> Right. And I think the solution is to think about this in some other
>> way than just running things in a VM, like a test harness that does the
>> right thing in terms of the classes being tested (what would be in the
>> boot classloader) and the classes doing the testing.
>
> I don't know about that. I'm sure that if the problem is well-defined
> enough solutions will become apparent, and I still don't quite get why it
> is the subject of continuous debate (eg can't someone just go out and try
> and do what you propose and show it works?).
The problem is 'completeness' because we have multiple problems to
solve. The .test.* solution works - it gets the test off the boot
classpath (and associated namespaces) so the API tests can function
properly - in the right security context, namely the same as user code.
>
>>>> With the "custom" of putting in things in o.a.h.t are we
>>>> implicitly discouraging good testing practice?
>>> This is laughable.
>> You are going to have to explain why it's "laughable". If you are
>> testing a.b.c.Foo and you have to do it from a.b.c.test.FooTest, how can
>> you ever do implementation testing of Foo? It's not an unreasonable
>> question. Certainly not "laughable".
>
> In general casting something someone else thinks as laughable is not very
> conductive to working together. I thought the question was phrased in a
> very thought-provoking manner :-).
>
> In any case, the obvious answer to the question is that you can do it by
> writing your implementation so that it is implementation testable in that
> manner. This means not (or allmost not) using package-private access
> definitions anywhere. If "protected" can make sense, you get to do things
> such as
>
> public class MyTestCase extends TestCase
> {
> public static class MyExtended extends My
> {
> public My m;
>
> public MyExtended( My m )
> {
> this.m = m;
> }
>
> public Object exposedFoo()
> {
> return m.foo();
> }
> }
> }
>
> If "protected" does not make sense, you can put the "real" implementation
> in some other package, and then the package-private stuff is nothing more
> than a facade for that real implementation (you still can't
> implementation-test the facade. What you can do is to use code generation
> to create the facade, and then implementation test the code generation.
> Or just not bother). Eg
>
> --
> package java.foo;
>
> import o.a.h.j.foo.FooImpl;
>
> class Foo { /* package private */
> private final FooImpl f = new FooImpl();
>
> void foo()
> {
> f.foo();
> }
> }
> --
> package o.a.h.j.foo;
>
> public class FooImpl
> {
> public void foo() // readily testable, cuz public
> {
> /* ... */
> }
> }
> --
>
> The last option I'm aware of is to resort to using reflection,
> since the runtime type system can bypass any and all access restrictions
> if you have the appropriate security manager, but that leads to rather
> painful test coding and makes the test coding error prone.
I think that both of these solutions are
a) messy - since only XP psycho's really *enjoy* creating unit tests, we
want to make it as painless as possible as to not disincentivize
developers. Look at what we have so far. IBM had to go off to the Unit
Test Mines they run in a Secret Undisclosed Location in the Principality
of BigBlueLand to provide unit tests for stuff they had already donated!
:) [Thanks, btw]
b) subject to "mechanical failure" - we're doing all sorts of unnatural
acts on code that is usually the "rock solid" basis for doing these
unnatural things to other code (like in app servers), and I worry that
such complexity will lead to very hard or impossible to find failures or
bugs
>
> There is also the possibility that all the package-private materials in
> reality are fully exercised if you test the public parts of the package
> thoroughly enough. A coverage utility like clover can show that. XP
> (extreme programming) purists (like me) might argue that if you have
> package-private stuff that is not exerciseable through the public API
> that the package-private stuff needs to be factored out. But lets try not
> to argue too much :-)
I agree with the latter part. What I worry about though is that despite
the best of intentions, unit testing tends not to ever be complete and
thorough. I don't know if things like clover indicate the quality of
the coverage - but simply having coverage just isn't enough, IMO, as you
may not exercise completely enough so that all internal functionality is
directly exercised. Dunno.
>
>>>> Given that this
>>>> o.a.h.t.* pattern comes from Eclipse-land, how do they do it?
>
> I doubt it comes from Eclipse-land. If ViewCVS wasn't locked for CVS
> I could probably find you code from 1997 at the ASF that has a .test.
> package in the middle.
Tim referenced Eclipse as the source of the practice. That's why I was
asking for how they solved the problem of implementation testing.
>
>>>> I
>>>> couldn't imagine that the Eclipse tests don't test package protected
>>>> things.
>>> The only thing shared with Eclipse-land here is the *.tests.* package
>>> name element, hardly significant or unique I expect.
>> Well, it is around here. While I haven't done a survey, I'm used to
>> projects keeping things in parallel trees to make it easy to test.
>
> If with "here" you mean "the ASF" I'm happy to challenge the assertion :-)
Please point me to it! I always want to see new ways of doing this.
Challenge away!
>
>> Granted, projects don't have the problem we have.
>>
>> The thing I'm asking for is this - how in Eclipse-land do they test
>> package protected stuff? How do they do implementation tests?
>
> I suspects its one or more of the above. For my own code, I tend to
> design it so that implementation tests are not neccessary - eg I build
> a large amount of specification tests (API tests) and verify that the
> code coverage from running the API tests is 100%. Of course we don't
> have that luxury (the API is already defined, and most of it probably
> wasn't designed with this whole "purist" testing thing in mind).
>
>>> Eclipse testing does not have the java.* namespace issues with
>>> classloaders that we have got.
>> Right, but that's a classloader and security manager issue for the
>> testing framework, isn't it?
>>
>> Hypothetically....suppose we decided (for whatever reason) that we
>> weren't going to test in situ to get better control of the environment.
>> What would you do?
>
> What does "in situ" mean?
"in the original place". IOW, we test this code not in isolation, like
in a testing framework, but in the VM itself, which IMO is the problem.
It's the problem because (as Tim rightly points out) any in-package unit
tests are running "incorrectly" - they are not running as user code that
uses the target classes would be running.
So the problem boils down to the fact that we are implicitly doing
integration testing. That's why I've been suggesting the framework -
let us test the code in isolation first, using "implementation tests".
Then, if our isolation framework is sexy enough, lets try to reproduce
the same classloader/security model we would experience in a VM, and do
spec/API testing. *Then* we can do integration testing by running the
code in the VM ("in situ") and do the proper (aka (*.test.*) )
spec/API/tck testing.
I'll post this as a separate message because this one is way too woolly
at this point.
>
>>>> I've been short of Round Tuits lately, but I still would like to
>>>> investigate a test harness that helps us by mitigating the security
>>>> issues...
>>> Today we run all our tests in one suite on the classpath. They are API
>>> tests.
>> I hope they are more than API tests.
>
> See above for why one could hope they don't need to more than API tests (I
> doubt it, but in terms of what would be *nice*...)
>
>>> I expect that we will at least have another test suite of implementation
>>> tests.
>>>
>>> However, over the last few weeks we have been discussing the other
>>> 'dimensions' of testing that we want to embody, and we haven't settled
>>> on a suitable way of representing those different dimensions. Filenames
>>> for testcases may do it if we can squeeze in enough information into a
>>> filename (I don't like that approach, BTW)
>> I don't either.
>>
>> , or explicitly defining
>>> different suites of tests.
>> Which makes sense.
>
> Yup. It could even make sense to build some rather large extensions to JUnit
> to make all this stuff more manageable (eg we *can* do stuff like
>
> MyApiTest extends AbstractHarmonyTestCase
> {
> static { markTestStyle(API); }
>
> /* ... */
> }
>
> MyApiTest extends AbstractHarmonyTestCase
> {
> static { markTestStyle(IMPL); }
>
> /* ... */
> }
>
> , or similar things using 1.5 annotations).
>
>
> cheers!
>
>
> Leo
>
>
Re: [OT] Re: Unit testing revisited
Posted by Stepan Mishura <st...@gmail.com>.
On 3/22/06, Geir Magnusson Jr wrote:
>
>
>
> Leo Simons wrote:
> > On Wed, Mar 22, 2006 at 07:34:16AM -0500, Geir Magnusson Jr wrote:
> >>> LEO :
> >>> I'll point out that every time you restrict to an ordered sequence of
> >>> taking care of things in an open souce community you do slow them down
> just
> >>> a little (hey, that's an interesting assertion. Hmm. Should write a
> book
> >>> about it I guess) so make sure its what you want :-).
> >> Huh?
> >
> > You didn't say "let us test the code in isolation [using a smart
> framework]",
> > you said "let us test the code in isolation *first* [using a smart
> > framework]". I need to write a book about why I think the difference
> > matters, and it needs to be a book because I'll need many many words...
>
> Oh - no, I didn't mean that. Sorry. All three are independent. You can
> do them in parallel. We can build our mechanism to do the
> implementation tests correctly while we continue to do everything else.
>
> I just wanted to see if I could get through the fog and be clear what
> the issues are and stop confusing #1 and #2, both of which are important.
>
> To test java.util.Foo, I believe it's important to have BOTH
>
> java.util.FooTest
>
> AND
>
> org.apache.harmony.test.java.util.FooTest
>
> as they are intended to test different things (the first as a
> 'un-integrated' implementation test and the second as an 'in-situ'
> API/spec test).
Yes, indeed. We should admit that we need both tests rather then arguing
which test is the right.
Thanks,
Stepan.
If we agree on that and recognize that, I suspect the test debates will
> come to rapid closure, and we'll have a mini-roadmap of what we want to
> do in the testing area that is parallelizable and doesn't hold anyone up.
>
> geir
>
>
>
--
Thanks,
Stepan Mishura
Intel Middleware Products Division
Re: [OT] Re: Unit testing revisited
Posted by Geir Magnusson Jr <ge...@pobox.com>.
Leo Simons wrote:
> On Wed, Mar 22, 2006 at 07:34:16AM -0500, Geir Magnusson Jr wrote:
>>> LEO :
>>> I'll point out that every time you restrict to an ordered sequence of
>>> taking care of things in an open souce community you do slow them down just
>>> a little (hey, that's an interesting assertion. Hmm. Should write a book
>>> about it I guess) so make sure its what you want :-).
>> Huh?
>
> You didn't say "let us test the code in isolation [using a smart framework]",
> you said "let us test the code in isolation *first* [using a smart
> framework]". I need to write a book about why I think the difference
> matters, and it needs to be a book because I'll need many many words...
Oh - no, I didn't mean that. Sorry. All three are independent. You can
do them in parallel. We can build our mechanism to do the
implementation tests correctly while we continue to do everything else.
I just wanted to see if I could get through the fog and be clear what
the issues are and stop confusing #1 and #2, both of which are important.
To test java.util.Foo, I believe it's important to have BOTH
java.util.FooTest
AND
org.apache.harmony.test.java.util.FooTest
as they are intended to test different things (the first as a
'un-integrated' implementation test and the second as an 'in-situ'
API/spec test).
If we agree on that and recognize that, I suspect the test debates will
come to rapid closure, and we'll have a mini-roadmap of what we want to
do in the testing area that is parallelizable and doesn't hold anyone up.
geir
[OT] Re: Unit testing revisited
Posted by Leo Simons <ma...@leosimons.com>.
On Wed, Mar 22, 2006 at 07:34:16AM -0500, Geir Magnusson Jr wrote:
> >Heh. You find *those* by running the app server tests :-). I suspect that
> >running the J2EE TCK against geronimo running on harmony and comparing it
> >with running the J2EE TCK against geronimo running on the sun jdk is
> >going to be pretty insightful...
>
> Like a mortar attack is insightful. :)
LOL. There might be a building or two left standing, and we can give the
people that built those a standing ovation :-)
> It will be an interesting test of "The Algebra of TCK-ness"
>
> If A = Sun JDK passes Java SE TCK
> If B(A) = Geronimo passes Java EE TCK on compliant Sun JDK
> If C = Harmony JDK passes on Java SE TCK
>
> then it should be true that B(C). No need to test!
>
> :)
LOL. I think you've just proved the whole discussion is fruitless.
> >>Please point me to it! I always want to see new ways of doing this.
> >>Challenge away!
> >
> >Okay :-), top-of-head,
> >
> >http://svn.apache.org/repos/asf/excalibur/trunk/framework/impl/src/test/org/apache/avalon/framework/context/test/ContextTestCase.java
> >
> >(one of the last remaining bits of code that can be traced back to apache
> >jserv which was tested using testlet which was around before JUnit). In
> >general, the parts of jakarta and what grew out of it that are derivants of
> >the JServ branch of working (including avalon, now excalibur, cocoon) often
> >do thingsl ike this.
> >
> >The fact I typed that URL from memory and was right is kinda scary, isn't
> >it? I've not worked on that code for *years* and its moved a few dozen
> >times...
>
> That is scary. It's also scary that you proposed Avalon as an example :)
LOL. That project has some of the best frigging brilliant ideas in its
codebase, some of which most of the java world *still* hasn't figured out.
Ever heard of classloader hell? We /solved/ that, in 1999, before the J2EE
world knew it would become a problem (all the people at JBoss still haven't
figured it out and I hear Geronimo is having trouble too). I still view OSGi
as a broken committee-designed version of half of avalon. The core of the
code that makes apache cocoon go as fast as it goes lived in avalon and now
lives in excalibur. No-one hardly touches it but no-one needs to because its
as rock solid as, well, the JDK :-)
Quality of the code was never a problem with avalon. Another one of my
assertions is that as code quality approaches infinity the number of
people that can work on it together approaches 0. Lisp hackers don't even
try (you might spot a rephrased version of the "all code continuously evolves
to become like a broken version of Common Lisp", above), in avalon, we did...
...doesn't mean reading the code is a bad idea. Go read :-)
> >>So the problem boils down to the fact that we are implicitly doing
> >>integration testing. That's why I've been suggesting the framework -
> >>let us test the code in isolation first, using "implementation tests".
> >>Then, if our isolation framework is sexy enough, lets try to reproduce
> >>the same classloader/security model we would experience in a VM, and do
> >>spec/API testing. *Then* we can do integration testing by running the
> >>code in the VM ("in situ") and do the proper (aka (*.test.*) )
> >>spec/API/tck testing.
> >>
> >>I'll post this as a separate message because this one is way too woolly
> >>at this point.
> >
> >Okay, this does sound like "the core" of the matter. There you go.
> >
> >I'll point out that every time you restrict to an ordered sequence of
> >taking care of things in an open souce community you do slow them down just
> >a little (hey, that's an interesting assertion. Hmm. Should write a book
> >about it I guess) so make sure its what you want :-).
>
> Huh?
You didn't say "let us test the code in isolation [using a smart framework]",
you said "let us test the code in isolation *first* [using a smart
framework]". I need to write a book about why I think the difference
matters, and it needs to be a book because I'll need many many words...
LSD
Re: Unit testing revisited
Posted by Geir Magnusson Jr <ge...@pobox.com>.
Leo Simons wrote:
> On Wed, Mar 22, 2006 at 06:41:56AM -0500, Geir Magnusson Jr wrote:
>>
[SNIP]
>> You forgot one - "integration test", which is a unit test that's been
>> around long enough to shave. :) (It's actually not a unit test...)
>
> "integration test" --> any test that is not an implementation test or
> specification test. Typically these test the interactions between
> multiple pieces rather than the correct behaviour of a single
> piece.
>
> I forgot another one:
>
> "gump run using harmony" --> the biggest frigging integration test you
> can think of. Tests the interaction between harmony and millions
> of lines of userland code.
"frigging integration test" --> A kind of integration test that
uses a "frig", or "functional rig". See
http://gump.apache.org/
:)
>
>>>>> We already see lots of errors caused by
>>>>> oversight of the classloader differences.
>>>> Right. And I think the solution is to think about this in some other
>>>> way than just running things in a VM, like a test harness that does the
>>>> right thing in terms of the classes being tested (what would be in the
>>>> boot classloader) and the classes doing the testing.
>>> I don't know about that. I'm sure that if the problem is well-defined
>>> enough solutions will become apparent, and I still don't quite get why it
>>> is the subject of continuous debate (eg can't someone just go out and try
>>> and do what you propose and show it works?).
>> The problem is 'completeness' because we have multiple problems to
>> solve.
>
> Uh-oh. Completeness is a scary word. I didn't see that coming.
>
> <snip a couple of hackiness details />
>> I think that both of these solutions are
>>
>> a) messy - since only XP psycho's really *enjoy* creating unit tests, we
>> want to make it as painless as possible as to not disincentivize
>> developers. Look at what we have so far. IBM had to go off to the Unit
>> Test Mines they run in a Secret Undisclosed Location in the Principality
>> of BigBlueLand to provide unit tests for stuff they had already donated!
>> :) [Thanks, btw]
>
> The class library design is messy. Testing it will, one way or another, be
> a messy subject.
>
>> b) subject to "mechanical failure" - we're doing all sorts of unnatural
>> acts on code that is usually the "rock solid" basis for doing these
>> unnatural things to other code (like in app servers), and I worry that
>> such complexity will lead to very hard or impossible to find failures or
>> bugs
>
> Heh. You find *those* by running the app server tests :-). I suspect that
> running the J2EE TCK against geronimo running on harmony and comparing it
> with running the J2EE TCK against geronimo running on the sun jdk is
> going to be pretty insightful...
Like a mortar attack is insightful. :)
It will be an interesting test of "The Algebra of TCK-ness"
If A = Sun JDK passes Java SE TCK
If B(A) = Geronimo passes Java EE TCK on compliant Sun JDK
If C = Harmony JDK passes on Java SE TCK
then it should be true that B(C). No need to test!
:)
>
>>> There is also the possibility that all the package-private materials in
>>> reality are fully exercised if you test the public parts of the package
>>> thoroughly enough. A coverage utility like clover can show that. XP
>>> (extreme programming) purists (like me) might argue that if you have
>>> package-private stuff that is not exerciseable through the public API
>>> that the package-private stuff needs to be factored out. But lets try not
>>> to argue too much :-)
>> I agree with the latter part. What I worry about though is that despite
>> the best of intentions, unit testing tends not to ever be complete and
>> thorough. I don't know if things like clover indicate the quality of
>> the coverage - but simply having coverage just isn't enough, IMO, as you
>> may not exercise completely enough so that all internal functionality is
>> directly exercised. Dunno.
>
> You've never had the pleasure of being part of a project that was fully
> XP-run from the start, have you? Its not a pipe dream but its also not
> likely to be attainable for harmony (if we want to get anything running
> before 2020).
No, I haven't. I don't think you could do Jave SE as XP because design
and planning is needed :)
>
>>>>>> I
>>>>>> couldn't imagine that the Eclipse tests don't test package protected
>>>>>> things.
>>>>> The only thing shared with Eclipse-land here is the *.tests.* package
>>>>> name element, hardly significant or unique I expect.
>>>> Well, it is around here. While I haven't done a survey, I'm used to
>>>> projects keeping things in parallel trees to make it easy to test.
>>> If with "here" you mean "the ASF" I'm happy to challenge the assertion :-)
>> Please point me to it! I always want to see new ways of doing this.
>> Challenge away!
>
> Okay :-), top-of-head,
>
> http://svn.apache.org/repos/asf/excalibur/trunk/framework/impl/src/test/org/apache/avalon/framework/context/test/ContextTestCase.java
>
> (one of the last remaining bits of code that can be traced back to apache
> jserv which was tested using testlet which was around before JUnit). In
> general, the parts of jakarta and what grew out of it that are derivants of
> the JServ branch of working (including avalon, now excalibur, cocoon) often
> do thingsl ike this.
>
> The fact I typed that URL from memory and was right is kinda scary, isn't
> it? I've not worked on that code for *years* and its moved a few dozen
> times...
That is scary. It's also scary that you proposed Avalon as an example :)
>
>> So the problem boils down to the fact that we are implicitly doing
>> integration testing. That's why I've been suggesting the framework -
>> let us test the code in isolation first, using "implementation tests".
>> Then, if our isolation framework is sexy enough, lets try to reproduce
>> the same classloader/security model we would experience in a VM, and do
>> spec/API testing. *Then* we can do integration testing by running the
>> code in the VM ("in situ") and do the proper (aka (*.test.*) )
>> spec/API/tck testing.
>>
>> I'll post this as a separate message because this one is way too woolly
>> at this point.
>
> Okay, this does sound like "the core" of the matter. There you go.
>
> I'll point out that every time you restrict to an ordered sequence of
> taking care of things in an open souce community you do slow them down just
> a little (hey, that's an interesting assertion. Hmm. Should write a book
> about it I guess) so make sure its what you want :-).
Huh?
geir
Re: Unit testing revisited
Posted by Leo Simons <ma...@leosimons.com>.
On Wed, Mar 22, 2006 at 06:41:56AM -0500, Geir Magnusson Jr wrote:
> >Eg I would suggest that we bite the bullet and go something like this:
> >
> > "unit test" --> any test runnable by a "unit testing framework" such as
> > JUnit or Cactus.
> >
> > "implementation test" --> a test run to verify that a specific piece
> > of code, preferably as small a piece as is seperately
> > testable, behaves as expected.
> >
> > "specification test" --> a test run to verify that an implementation is
> > conformant with some specification, prefereably as small a piece
> > of the specification for which a test can be defined.
> >
> > "API test" --> a specification test where the specification takes the
> > form of an API definition (perhaps a java interface with
> > supporting javadocs, perhaps just javadocs, perhaps IDL...)
> >
> > "tck test" --> any test defined as part of something that is called a
> > "TCK" or technology compatibility kit. TCK tests are
> > supposed to be specification tests.
>
> You forgot one - "integration test", which is a unit test that's been
> around long enough to shave. :) (It's actually not a unit test...)
"integration test" --> any test that is not an implementation test or
specification test. Typically these test the interactions between
multiple pieces rather than the correct behaviour of a single
piece.
I forgot another one:
"gump run using harmony" --> the biggest frigging integration test you
can think of. Tests the interaction between harmony and millions
of lines of userland code.
> >>>We already see lots of errors caused by
> >>>oversight of the classloader differences.
> >>Right. And I think the solution is to think about this in some other
> >>way than just running things in a VM, like a test harness that does the
> >>right thing in terms of the classes being tested (what would be in the
> >>boot classloader) and the classes doing the testing.
> >
> >I don't know about that. I'm sure that if the problem is well-defined
> >enough solutions will become apparent, and I still don't quite get why it
> >is the subject of continuous debate (eg can't someone just go out and try
> >and do what you propose and show it works?).
>
> The problem is 'completeness' because we have multiple problems to
> solve.
Uh-oh. Completeness is a scary word. I didn't see that coming.
<snip a couple of hackiness details />
> I think that both of these solutions are
>
> a) messy - since only XP psycho's really *enjoy* creating unit tests, we
> want to make it as painless as possible as to not disincentivize
> developers. Look at what we have so far. IBM had to go off to the Unit
> Test Mines they run in a Secret Undisclosed Location in the Principality
> of BigBlueLand to provide unit tests for stuff they had already donated!
> :) [Thanks, btw]
The class library design is messy. Testing it will, one way or another, be
a messy subject.
> b) subject to "mechanical failure" - we're doing all sorts of unnatural
> acts on code that is usually the "rock solid" basis for doing these
> unnatural things to other code (like in app servers), and I worry that
> such complexity will lead to very hard or impossible to find failures or
> bugs
Heh. You find *those* by running the app server tests :-). I suspect that
running the J2EE TCK against geronimo running on harmony and comparing it
with running the J2EE TCK against geronimo running on the sun jdk is
going to be pretty insightful...
> >There is also the possibility that all the package-private materials in
> >reality are fully exercised if you test the public parts of the package
> >thoroughly enough. A coverage utility like clover can show that. XP
> >(extreme programming) purists (like me) might argue that if you have
> >package-private stuff that is not exerciseable through the public API
> >that the package-private stuff needs to be factored out. But lets try not
> >to argue too much :-)
>
> I agree with the latter part. What I worry about though is that despite
> the best of intentions, unit testing tends not to ever be complete and
> thorough. I don't know if things like clover indicate the quality of
> the coverage - but simply having coverage just isn't enough, IMO, as you
> may not exercise completely enough so that all internal functionality is
> directly exercised. Dunno.
You've never had the pleasure of being part of a project that was fully
XP-run from the start, have you? Its not a pipe dream but its also not
likely to be attainable for harmony (if we want to get anything running
before 2020).
> >>>>I
> >>>>couldn't imagine that the Eclipse tests don't test package protected
> >>>>things.
> >>>The only thing shared with Eclipse-land here is the *.tests.* package
> >>>name element, hardly significant or unique I expect.
> >>Well, it is around here. While I haven't done a survey, I'm used to
> >>projects keeping things in parallel trees to make it easy to test.
> >
> >If with "here" you mean "the ASF" I'm happy to challenge the assertion :-)
>
> Please point me to it! I always want to see new ways of doing this.
> Challenge away!
Okay :-), top-of-head,
http://svn.apache.org/repos/asf/excalibur/trunk/framework/impl/src/test/org/apache/avalon/framework/context/test/ContextTestCase.java
(one of the last remaining bits of code that can be traced back to apache
jserv which was tested using testlet which was around before JUnit). In
general, the parts of jakarta and what grew out of it that are derivants of
the JServ branch of working (including avalon, now excalibur, cocoon) often
do thingsl ike this.
The fact I typed that URL from memory and was right is kinda scary, isn't
it? I've not worked on that code for *years* and its moved a few dozen
times...
> So the problem boils down to the fact that we are implicitly doing
> integration testing. That's why I've been suggesting the framework -
> let us test the code in isolation first, using "implementation tests".
> Then, if our isolation framework is sexy enough, lets try to reproduce
> the same classloader/security model we would experience in a VM, and do
> spec/API testing. *Then* we can do integration testing by running the
> code in the VM ("in situ") and do the proper (aka (*.test.*) )
> spec/API/tck testing.
>
> I'll post this as a separate message because this one is way too woolly
> at this point.
Okay, this does sound like "the core" of the matter. There you go.
I'll point out that every time you restrict to an ordered sequence of
taking care of things in an open souce community you do slow them down just
a little (hey, that's an interesting assertion. Hmm. Should write a book
about it I guess) so make sure its what you want :-).
LSD