You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by Geir Magnusson Jr <ge...@pobox.com> on 2006/03/22 13:15:28 UTC
[classlib] Testing
Pulling out of the various threads where we have been discussing, can we
agree on the problem :
We have unique problems compared to other Java projects because we need
to find a way to reliably test the things that are commonly expected to
be a solid point of reference - namely the core class library.
Further, we've been implicitly doing "integration testing" because - so
far - the only way we've been testing our code has been 'in situ' in the
VM - not in an isolated test harness. To me, this turns it into an
integration test.
Sure, we're using JUnit, but because of the fact we are implmenting core
java.* APIs, we aren't testing with a framework that has been
independently tested for correctness, like we would when testing any
other code.
I hope I got that idea across - I believe that we have to go beyond
normal testing approaches because we don't have a normal situation.
So I think there are three things we want to do (adopting the
terminology that came from the discussion with Tim and Leo ) :
1) implementation tests
2) spec/API tests (I'll bundle together)
3) integration/functional tests
I believe that for #1, the issues related to being on the bootclasspath
don't matter, because we aren't testing that aspect of the classes
(which is how they behave integrated w/ the VM and security system) but
rather the basic internal functioning.
I'm not sure how to approach this, but I'll try. I'd love to hear how
Sun, IBM or BEA deals with this, or be told why it isn't an issue :)
Implementation tests : I'd like to see us be able to do #1 via the
standard same-package technique (i.e. testing a.b.C w/ a.b.CTest) but
we'll run into a tangle of classloader problems, I suspect, becuase we
want to be testing java.* code in a system that already has java.* code.
Can anyone see a way we can do this - test the classlibrary from the
integration point of view - using some test harness + any known-good
JRE, like Sun's or IBM's?
Spec/API tests : these are, IMO, a kind of integration test, because
proper spec/API behavior *is* dependent on factors beyond the actual
code itself (like classloader configuration, and security context).
Because of this, the *.test.* pattern makes perfect sense. Assuming we
could produce something useful for #1 (i.e. a test harness/framework),
could we then augment it to simulate the classloader config + security
config that we'd get in a real VM? That will give us the ability to
test in isolation of the VM, and also let us 'break' the environment to
ensure that the code fails in a predictable way.
Intgration/functional : this is a whole range of things, from doing the
Spec/API tests in an actual VM, to the tests that exercise the code
through interaction with external systems (like network, RMI, GUI, etc)
***
Now, it might be suggested that we just ignore the implementation
testing (#1) and just do #2 and #3 as we are now, and hope we have a
good enough test suite. It could be argued that when Sun started, they
didn't have a known-good platform to do implementation testing on like
we do now. I don't know if that's true.
The difference is that we need to produce something of the same quality
as Sun's Java 5, not Sun's Java 1.0. We've had 11 years since 1.0 to
learn about testing, but they've had 11 years to get things solid.
What to do....
geir
Re: [classlib] Testing
Posted by George Harley <ge...@googlemail.com>.
George Harley wrote:
> Geir Magnusson Jr wrote:
>>
>>
>> George Harley wrote:
>>> Geir Magnusson Jr wrote:
>>>>
>>>> Geir Magnusson Jr wrote:
>>>>>
>>>>> George Harley wrote:
>>>> >>
>>>>>> Of course, the text module has only "implementation-independent
>>>>>> tests that designed to be run from classpath". For modules that
>>>>>> have got implementation-specific tests then I suppose we could
>>>>>> use something like
>>>>>> "org.apache.harmony.[module].tests.impl.[package under test]" or
>>>>>> "org.apache.harmony.[module].tests.internal.[package under test]"
>>>>>> etc. I've got no preference.
>>>>>
>>>>> -1
>>>>>
>>>>> Put them in the same package as the class being tested, please, if
>>>>> there is no problem being run that way.
>>>>
>>>> Sorry to be so snippy. Been a long day.
>>>
>>> Sure. No worries.
>>>
>>>>
>>>> Why would we want this convoluted package name? Why wouldn't we
>>>> want to let them be in the same package as the class being tested
>>>> since it's implementation testing?
>>>
>>> OK, point taken about the long package names that may result for
>>> tests of org.apache.harmony.* stuff. If the class under test is an
>>> internal type (i.e. in the org.apache.harmony.* package space) then
>>> I am sure we can all agree on a sensible abbreviation so that the
>>> org.apache.harmony part does not appear twice. So, for instance, if
>>> we are testing public methods on
>>> org.apache.harmony.security.foo.bar.MyType from the classpath then
>>> we could have the test package be
>>> org.apache.harmony.security.tests.foo.bar.
>>>
>>> I am concerned that we are all agree on what we mean by
>>> "implementation"-specific testing and "internal"-specific testing. I
>>> know it has been confusing me a lot.
>>
>> Yah - we should probably bottom out there...
>>
>>>
>>> If we want to run API tests that are specific to our implementation
>>> then there is no need to have the test types in the same package
>>> name as the class under test and no need for those tests to be on
>>> the bootclasspath. Orthogonal to that are tests that are actually
>>> testing the internals of a type - these are what I take to be
>>> candidates for being in the same package and on the bootclasspath at
>>> test time.
>>
>> So your suggestion is? I agree with you, but when reading the above,
>> I was expecting some kind of "therefore, I think we should...". it
>> could just be that I have no coffee this a.m.
>>
>> geir
>
> Hi,
>
> Actually it was me who was suffering the effects of caffeine
> deficiency. All rectified now :-)
>
> Right, here is a sort of "therefore" that you correctly noticed was
> missing from my previous note. It's probably more a cry for help as I
> try and grasp towards simplicity.
>
> I really think that there are only two kinds of unit test in the
> context of this discussion and I think that where in they live in the
> file system is key.
>
> First, those that are testing public/API methods (both Java API and
> our implementation-specific org.apache.harmony.* API) and do not need
> to be run on the bootclasspath. For tests of Java API types, our test
> classes are in packages called o.a.h.<module>.tests.<package under
> test>. Those tests that are dealing with API methods of
> implementation-specific types (o.a.h.*) can be in test packages that
> are something like o.a.h.<module>.tests.<appropriate package fragment
> minus any o.a.h duplication. On the file system all of these API tests
> can live under <module>/src/test/java and be built to a bin location
> such as bin/main that can be added as a classpath element in our test
> scripts.
Sorry, I meant that all tests intended to run on the classpath should be
compiled to bin/test.
>
> Next, we have "internal" tests that are intended to access non-public
> properties of a type irrespective of whether it is a public API type
> or an implementation-specific type. These tests are in the same
> package as the type under test and are intended to run on the
> bootclasspath. They can live under a separate source folder (e.g.
> <module>/src/test/java-internal or any other of Paulex's suggestions)
> and get built to a separate output location (e.g. bin/main-internal)
> that gets added to the bootclasspath. I like using the "-internal"
> suffix since it makes it clear that the code under those locations has
> been written explicitly to reach into the internals of a type.
...likewise, I meant that tests intended for the bootclasspath should be
compiled to something like a bin/test-internal
>
> Of course the merits of having tests in the same package as the thing
> under test is another story for another thread but it seems pretty
> obvious that a lot of folks go for it so I'll respect that.
>
> OK, my coffee cup is empty now.
>
> Best regards,
> George
>
>>
>>>
>>> Best regards,
>>> George
>>>
>>>>
>>>> geir
>>>>
>>>> ---------------------------------------------------------------------
>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by George Harley <ge...@googlemail.com>.
Geir Magnusson Jr wrote:
>
>
> George Harley wrote:
>> Geir Magnusson Jr wrote:
>>>
>>> Geir Magnusson Jr wrote:
>>>>
>>>> George Harley wrote:
>>> >>
>>>>> Of course, the text module has only "implementation-independent
>>>>> tests that designed to be run from classpath". For modules that
>>>>> have got implementation-specific tests then I suppose we could use
>>>>> something like "org.apache.harmony.[module].tests.impl.[package
>>>>> under test]" or
>>>>> "org.apache.harmony.[module].tests.internal.[package under test]"
>>>>> etc. I've got no preference.
>>>>
>>>> -1
>>>>
>>>> Put them in the same package as the class being tested, please, if
>>>> there is no problem being run that way.
>>>
>>> Sorry to be so snippy. Been a long day.
>>
>> Sure. No worries.
>>
>>>
>>> Why would we want this convoluted package name? Why wouldn't we
>>> want to let them be in the same package as the class being tested
>>> since it's implementation testing?
>>
>> OK, point taken about the long package names that may result for
>> tests of org.apache.harmony.* stuff. If the class under test is an
>> internal type (i.e. in the org.apache.harmony.* package space) then I
>> am sure we can all agree on a sensible abbreviation so that the
>> org.apache.harmony part does not appear twice. So, for instance, if
>> we are testing public methods on
>> org.apache.harmony.security.foo.bar.MyType from the classpath then we
>> could have the test package be
>> org.apache.harmony.security.tests.foo.bar.
>>
>> I am concerned that we are all agree on what we mean by
>> "implementation"-specific testing and "internal"-specific testing. I
>> know it has been confusing me a lot.
>
> Yah - we should probably bottom out there...
>
>>
>> If we want to run API tests that are specific to our implementation
>> then there is no need to have the test types in the same package name
>> as the class under test and no need for those tests to be on the
>> bootclasspath. Orthogonal to that are tests that are actually testing
>> the internals of a type - these are what I take to be candidates for
>> being in the same package and on the bootclasspath at test time.
>
> So your suggestion is? I agree with you, but when reading the above,
> I was expecting some kind of "therefore, I think we should...". it
> could just be that I have no coffee this a.m.
>
> geir
Hi,
Actually it was me who was suffering the effects of caffeine deficiency.
All rectified now :-)
Right, here is a sort of "therefore" that you correctly noticed was
missing from my previous note. It's probably more a cry for help as I
try and grasp towards simplicity.
I really think that there are only two kinds of unit test in the context
of this discussion and I think that where in they live in the file
system is key.
First, those that are testing public/API methods (both Java API and our
implementation-specific org.apache.harmony.* API) and do not need to be
run on the bootclasspath. For tests of Java API types, our test classes
are in packages called o.a.h.<module>.tests.<package under test>. Those
tests that are dealing with API methods of implementation-specific types
(o.a.h.*) can be in test packages that are something like
o.a.h.<module>.tests.<appropriate package fragment minus any o.a.h
duplication. On the file system all of these API tests can live under
<module>/src/test/java and be built to a bin location such as bin/main
that can be added as a classpath element in our test scripts.
Next, we have "internal" tests that are intended to access non-public
properties of a type irrespective of whether it is a public API type or
an implementation-specific type. These tests are in the same package as
the type under test and are intended to run on the bootclasspath. They
can live under a separate source folder (e.g.
<module>/src/test/java-internal or any other of Paulex's suggestions)
and get built to a separate output location (e.g. bin/main-internal)
that gets added to the bootclasspath. I like using the "-internal"
suffix since it makes it clear that the code under those locations has
been written explicitly to reach into the internals of a type.
Of course the merits of having tests in the same package as the thing
under test is another story for another thread but it seems pretty
obvious that a lot of folks go for it so I'll respect that.
OK, my coffee cup is empty now.
Best regards,
George
>
>>
>> Best regards,
>> George
>>
>>>
>>> geir
>>>
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Geir Magnusson Jr <ge...@pobox.com>.
George Harley wrote:
> Geir Magnusson Jr wrote:
>>
>> Geir Magnusson Jr wrote:
>>>
>>> George Harley wrote:
>> >>
>>>> Of course, the text module has only "implementation-independent
>>>> tests that designed to be run from classpath". For modules that have
>>>> got implementation-specific tests then I suppose we could use
>>>> something like "org.apache.harmony.[module].tests.impl.[package
>>>> under test]" or "org.apache.harmony.[module].tests.internal.[package
>>>> under test]" etc. I've got no preference.
>>>
>>> -1
>>>
>>> Put them in the same package as the class being tested, please, if
>>> there is no problem being run that way.
>>
>> Sorry to be so snippy. Been a long day.
>
> Sure. No worries.
>
>>
>> Why would we want this convoluted package name? Why wouldn't we want
>> to let them be in the same package as the class being tested since
>> it's implementation testing?
>
> OK, point taken about the long package names that may result for tests
> of org.apache.harmony.* stuff. If the class under test is an internal
> type (i.e. in the org.apache.harmony.* package space) then I am sure we
> can all agree on a sensible abbreviation so that the org.apache.harmony
> part does not appear twice. So, for instance, if we are testing public
> methods on org.apache.harmony.security.foo.bar.MyType from the classpath
> then we could have the test package be
> org.apache.harmony.security.tests.foo.bar.
>
> I am concerned that we are all agree on what we mean by
> "implementation"-specific testing and "internal"-specific testing. I
> know it has been confusing me a lot.
Yah - we should probably bottom out there...
>
> If we want to run API tests that are specific to our implementation then
> there is no need to have the test types in the same package name as the
> class under test and no need for those tests to be on the bootclasspath.
> Orthogonal to that are tests that are actually testing the internals of
> a type - these are what I take to be candidates for being in the same
> package and on the bootclasspath at test time.
So your suggestion is? I agree with you, but when reading the above, I
was expecting some kind of "therefore, I think we should...". it could
just be that I have no coffee this a.m.
geir
>
> Best regards,
> George
>
>>
>> geir
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by George Harley <ge...@googlemail.com>.
Geir Magnusson Jr wrote:
>
> Geir Magnusson Jr wrote:
>>
>> George Harley wrote:
> >>
>>> Of course, the text module has only "implementation-independent
>>> tests that designed to be run from classpath". For modules that have
>>> got implementation-specific tests then I suppose we could use
>>> something like "org.apache.harmony.[module].tests.impl.[package
>>> under test]" or "org.apache.harmony.[module].tests.internal.[package
>>> under test]" etc. I've got no preference.
>>
>> -1
>>
>> Put them in the same package as the class being tested, please, if
>> there is no problem being run that way.
>
> Sorry to be so snippy. Been a long day.
Sure. No worries.
>
> Why would we want this convoluted package name? Why wouldn't we want
> to let them be in the same package as the class being tested since
> it's implementation testing?
OK, point taken about the long package names that may result for tests
of org.apache.harmony.* stuff. If the class under test is an internal
type (i.e. in the org.apache.harmony.* package space) then I am sure we
can all agree on a sensible abbreviation so that the org.apache.harmony
part does not appear twice. So, for instance, if we are testing public
methods on org.apache.harmony.security.foo.bar.MyType from the classpath
then we could have the test package be
org.apache.harmony.security.tests.foo.bar.
I am concerned that we are all agree on what we mean by
"implementation"-specific testing and "internal"-specific testing. I
know it has been confusing me a lot.
If we want to run API tests that are specific to our implementation then
there is no need to have the test types in the same package name as the
class under test and no need for those tests to be on the bootclasspath.
Orthogonal to that are tests that are actually testing the internals of
a type - these are what I take to be candidates for being in the same
package and on the bootclasspath at test time.
Best regards,
George
>
> geir
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Paulex Yang <pa...@gmail.com>.
Geir Magnusson Jr wrote:
>
>
> Paulex Yang wrote:
>> Mikhail Loenko wrote:
>>> If it's separated, I'm OK with it
>>>
>>> Can you suggest dir names?
>>>
>> Great!
>>
>> Either boot or injected(as Mark said) sounds good keyword to me. so
>> it can be one of below:
>> 1. src/test.injected/
>> 2. src/test/injected
>> 3. src/test.boot/
>> 4. src/test/boot
>>
>> Mmm...seems injected is more self-descriptive, I prefer option 1. So
>> the layout would be:
>> src/main
>> src/test
>> src/test.injected
>>
>> ideas?
>
> I suggest keeping "." chars out of pathnames....
no problem. I agree.
>
> Also, why "injected"? Aren't there more functional names like "spec"
> or "public" or "implementation" ?
IIRC we have proposed impl/internal as counterpart of "api" in this
thread, but didn't reach agreement yet (each got someone's objection for
some reasons). Really hard to make everyone happy:(.
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
--
Paulex Yang
China Software Development Lab
IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Geir Magnusson Jr <ge...@pobox.com>.
Paulex Yang wrote:
> Mikhail Loenko wrote:
>> If it's separated, I'm OK with it
>>
>> Can you suggest dir names?
>>
> Great!
>
> Either boot or injected(as Mark said) sounds good keyword to me. so it
> can be one of below:
> 1. src/test.injected/
> 2. src/test/injected
> 3. src/test.boot/
> 4. src/test/boot
>
> Mmm...seems injected is more self-descriptive, I prefer option 1. So
> the layout would be:
> src/main
> src/test
> src/test.injected
>
> ideas?
I suggest keeping "." chars out of pathnames....
Also, why "injected"? Aren't there more functional names like "spec" or
"public" or "implementation" ?
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Mikhail Loenko <ml...@gmail.com>.
Tim,
the reason for the layouts is:
we have only one consistent proposal that covers all the cases we now aware of.
The only problem I see with the current proposal is that we will have classes
with same package and names in different folders. Like:
tests/impl
o.a.h.security.test.java.security.ProviderTest
and
tests/api
o.a.h.security.test.java.security.ProviderTest
So we will always have to separate build targets runing these tests.
Probably it's
OK. Otherwise we have to add 'type' of the test to the package name:
tests/impl
o.a.h.security.test.impl.java.security.ProviderTest
or
tests/impl
o.a.h.security.impl-test.java.security.ProviderTest
Thanks,
Mikhail
2006/5/4, Tim Ellison <t....@gmail.com>:
> I was suggesting that you try to capture the reasons for the layout and
> the package naming of tests etc. in a descriptive doc.
>
> At this point it would be a 'proposed' doc -- the mail thread is getting
> too long to follow all the arguments so having a document we can work on
> together is likely more productive.
>
> We can use the wiki if you prefer...
>
> Regards,
> Tim
>
> Mikhail Loenko wrote:
> > Geir
> >
> > What kind of conventions do you mean?
> >
> > If it is package naming then I thought we have the same conventions
> > for all types of the tests: bootclasspath tests go to the same package as
> > class, classpath tests go to o.a.h.module.tests.package
> >
> > Am I missing something?
> >
> > Thanks,
> > Mikhail
> >
> > 2006/5/4, Geir Magnusson Jr <ge...@pobox.com>:
> >>
> >>
> >> Mikhail Loenko wrote:
> >> > 2006/5/3, Mikhail Loenko <ml...@gmail.com>:
> >> >>
> >> >> Ok, I'll put this on the website
> >> >
> >> > I've put it on the web-site (it has not appeared yet)
> >> >
> >> > Comments are welcome
> >>
> >> I think you only captures half of it. You also need to describe
> >> conventions for harmony-specific tests...
> >>
> >> I'm happy to help there if you want...
> >>
> >> geir
> >>
> >> ---------------------------------------------------------------------
> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
> --
>
> Tim Ellison (t.p.ellison@gmail.com)
> IBM Java technology centre, UK.
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Geir Magnusson Jr <ge...@pobox.com>.
Tim Ellison wrote:
> I was suggesting that you try to capture the reasons for the layout and
> the package naming of tests etc. in a descriptive doc.
>
> At this point it would be a 'proposed' doc -- the mail thread is getting
> too long to follow all the arguments so having a document we can work on
> together is likely more productive.
>
> We can use the wiki if you prefer...
I'm not interested in discussing anything on a wiki. It requires people
to be online, and there's no way to look back through history, the
"thread of changes", in an easy way to figure out how we got to wherever
we got to.
I think your suggestion to Mikhail to capture a summary on the website
was helpful, and we can then start a fresh thread for the next set of
aspects if you think this one is too long.
geir
>
> Regards,
> Tim
>
> Mikhail Loenko wrote:
>> Geir
>>
>> What kind of conventions do you mean?
>>
>> If it is package naming then I thought we have the same conventions
>> for all types of the tests: bootclasspath tests go to the same package as
>> class, classpath tests go to o.a.h.module.tests.package
>>
>> Am I missing something?
>>
>> Thanks,
>> Mikhail
>>
>> 2006/5/4, Geir Magnusson Jr <ge...@pobox.com>:
>>>
>>> Mikhail Loenko wrote:
>>>> 2006/5/3, Mikhail Loenko <ml...@gmail.com>:
>>>>> Ok, I'll put this on the website
>>>> I've put it on the web-site (it has not appeared yet)
>>>>
>>>> Comments are welcome
>>> I think you only captures half of it. You also need to describe
>>> conventions for harmony-specific tests...
>>>
>>> I'm happy to help there if you want...
>>>
>>> geir
>>>
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Tim Ellison <t....@gmail.com>.
I was suggesting that you try to capture the reasons for the layout and
the package naming of tests etc. in a descriptive doc.
At this point it would be a 'proposed' doc -- the mail thread is getting
too long to follow all the arguments so having a document we can work on
together is likely more productive.
We can use the wiki if you prefer...
Regards,
Tim
Mikhail Loenko wrote:
> Geir
>
> What kind of conventions do you mean?
>
> If it is package naming then I thought we have the same conventions
> for all types of the tests: bootclasspath tests go to the same package as
> class, classpath tests go to o.a.h.module.tests.package
>
> Am I missing something?
>
> Thanks,
> Mikhail
>
> 2006/5/4, Geir Magnusson Jr <ge...@pobox.com>:
>>
>>
>> Mikhail Loenko wrote:
>> > 2006/5/3, Mikhail Loenko <ml...@gmail.com>:
>> >>
>> >> Ok, I'll put this on the website
>> >
>> > I've put it on the web-site (it has not appeared yet)
>> >
>> > Comments are welcome
>>
>> I think you only captures half of it. You also need to describe
>> conventions for harmony-specific tests...
>>
>> I'm happy to help there if you want...
>>
>> geir
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
--
Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Mikhail Loenko <ml...@gmail.com>.
According to guidlines draft both are possible if you run tests
from bootclasspath:
"If the test is designed to be run from bootclasspath then its package
is the same
as the package of the class under test"
Thanks,
Mikhail
2006/5/4, Geir Magnusson Jr <ge...@pobox.com>:
> For example, you have
>
> org.apache.harmony.luni.internal.net.www.protocol - package under test
> org.apache.harmony.luni.tests.internal.net.www.protocol - package for
> the test
>
> where if it's an implementation test since it's an 'internal' package,
> wouldn't the the test most likely be
>
> o.a.h.l.i.n.w.p ?
>
> (no "tests")
>
> It should also be possible to have a testcase
>
> java.util.HashMapTest
>
> if we want to test some internal thing about j.u.HM?
>
> geir
>
>
> Mikhail Loenko wrote:
> > Geir
> >
> > What kind of conventions do you mean?
> >
> > If it is package naming then I thought we have the same conventions
> > for all types of the tests: bootclasspath tests go to the same package as
> > class, classpath tests go to o.a.h.module.tests.package
> >
> > Am I missing something?
> >
> > Thanks,
> > Mikhail
> >
> > 2006/5/4, Geir Magnusson Jr <ge...@pobox.com>:
> >>
> >>
> >> Mikhail Loenko wrote:
> >> > 2006/5/3, Mikhail Loenko <ml...@gmail.com>:
> >> >>
> >> >> Ok, I'll put this on the website
> >> >
> >> > I've put it on the web-site (it has not appeared yet)
> >> >
> >> > Comments are welcome
> >>
> >> I think you only captures half of it. You also need to describe
> >> conventions for harmony-specific tests...
> >>
> >> I'm happy to help there if you want...
> >>
> >> geir
> >>
> >> ---------------------------------------------------------------------
> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Geir Magnusson Jr <ge...@pobox.com>.
For example, you have
org.apache.harmony.luni.internal.net.www.protocol - package under test
org.apache.harmony.luni.tests.internal.net.www.protocol - package for
the test
where if it's an implementation test since it's an 'internal' package,
wouldn't the the test most likely be
o.a.h.l.i.n.w.p ?
(no "tests")
It should also be possible to have a testcase
java.util.HashMapTest
if we want to test some internal thing about j.u.HM?
geir
Mikhail Loenko wrote:
> Geir
>
> What kind of conventions do you mean?
>
> If it is package naming then I thought we have the same conventions
> for all types of the tests: bootclasspath tests go to the same package as
> class, classpath tests go to o.a.h.module.tests.package
>
> Am I missing something?
>
> Thanks,
> Mikhail
>
> 2006/5/4, Geir Magnusson Jr <ge...@pobox.com>:
>>
>>
>> Mikhail Loenko wrote:
>> > 2006/5/3, Mikhail Loenko <ml...@gmail.com>:
>> >>
>> >> Ok, I'll put this on the website
>> >
>> > I've put it on the web-site (it has not appeared yet)
>> >
>> > Comments are welcome
>>
>> I think you only captures half of it. You also need to describe
>> conventions for harmony-specific tests...
>>
>> I'm happy to help there if you want...
>>
>> geir
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Mikhail Loenko <ml...@gmail.com>.
Geir
What kind of conventions do you mean?
If it is package naming then I thought we have the same conventions
for all types of the tests: bootclasspath tests go to the same package as
class, classpath tests go to o.a.h.module.tests.package
Am I missing something?
Thanks,
Mikhail
2006/5/4, Geir Magnusson Jr <ge...@pobox.com>:
>
>
> Mikhail Loenko wrote:
> > 2006/5/3, Mikhail Loenko <ml...@gmail.com>:
> >>
> >> Ok, I'll put this on the website
> >
> > I've put it on the web-site (it has not appeared yet)
> >
> > Comments are welcome
>
> I think you only captures half of it. You also need to describe
> conventions for harmony-specific tests...
>
> I'm happy to help there if you want...
>
> geir
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Geir Magnusson Jr <ge...@pobox.com>.
Mikhail Loenko wrote:
> 2006/5/3, Mikhail Loenko <ml...@gmail.com>:
>>
>> Ok, I'll put this on the website
>
> I've put it on the web-site (it has not appeared yet)
>
> Comments are welcome
I think you only captures half of it. You also need to describe
conventions for harmony-specific tests...
I'm happy to help there if you want...
geir
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Mikhail Loenko <ml...@gmail.com>.
2006/5/3, Mikhail Loenko <ml...@gmail.com>:
>
> Ok, I'll put this on the website
I've put it on the web-site (it has not appeared yet)
Comments are welcome
Thanks,
Mikhail
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Mikhail Loenko <ml...@gmail.com>.
2006/5/3, Tim Ellison <t....@gmail.com>:
> Mikhail Loenko wrote:
> > Did everybody agree to:
>
> I think so -- I just need to clarify some terminology I think...
>
> > 1. Tests are separated on directory level by 'intention'
>
> I agree where the intention is broad (i.e. functional tests vs.
> performance tests vs. stress tests) but I'm not convinced that we need
> to separate to a precise 'intent', at least I've never been in a
> position where I've wished that my my serialization tests are separate
> from my cloning tests or whatever.
I did not mean that we separate "all tests against method 'clone'" from
"all tests against constructors" into different dirs :)
>
> The problem I see is that trying to represent too many different
> classifications in a test's directory/file name leads to complex naming
> rules and eventually ambiguity or duplication when tests cross those
> classification groups. So I'd like to keep it simple.
+1
>
> I do agree that we should separate tests that we expect to run on any
> compliant Java implementation from those that are Harmony specific.
>
> > src/tests/
> > |
> > + impl/
> > |
> > + compliant/
>
> Just a minor nit -- we may choose to call these 'api' tests just to be
> very clear that we are not defining or modifying the meaning of
> 'compliant' (which is the role of the JCK).
OK, I see
>
> > + etc [might be stress, exotic, or something else]/
>
> We would create new directories as we need them, right?
>
> > \ resources/ -- as they are
> >
> > Note: inside 'intention' directory they might be further separated e.g
> > by platform
>
> In fact, we should allow for some autonomy within a module too. There
> will be a different balance between the types of tests that are required
> for, say, LUNI and those for RMI that mean the structure may not suit
> every module equally well.
In general I'm ok with autonomy within a module. Though more specific
example about luni would be helpful.
>
> > 2. 'Bootclasspath' tests are in the same package as implementation
>
> Yes -- where do these appear in the layout?
Depending on what are these tests. Most likely they will be in 'impl'
as we likely will not have many 'bootclasspath' API tests
> > 3. 'classpath' tests against 'public packages' go to
> > org.apache.harmony.module.tests.<public package name>
> >
> > note: public package name is not necessary in the java.* or javax.*
> > example: org.ietf.jgss
>
> Yes.
>
> > 4. 'classpath' tests against org.apache.harmony.module.<rest of the name>
> > classes go to
> > org.apache.harmony.module.tests.<rest of the name>
>
> Yes.
>
>
> Thanks for working through this Mikhail -- I feel that we are converging
> on a good solution.
>
> Would you be able to put together a (proposed) document describing this
> on the classlib website?
Ok, I'll put this on the website
Thanks,
Mikhail
>
> Regards,
> Tim
>
> --
>
> Tim Ellison (t.p.ellison@gmail.com)
> IBM Java technology centre, UK.
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Tim Ellison <t....@gmail.com>.
Mikhail Loenko wrote:
> Did everybody agree to:
I think so -- I just need to clarify some terminology I think...
> 1. Tests are separated on directory level by 'intention'
I agree where the intention is broad (i.e. functional tests vs.
performance tests vs. stress tests) but I'm not convinced that we need
to separate to a precise 'intent', at least I've never been in a
position where I've wished that my my serialization tests are separate
from my cloning tests or whatever.
The problem I see is that trying to represent too many different
classifications in a test's directory/file name leads to complex naming
rules and eventually ambiguity or duplication when tests cross those
classification groups. So I'd like to keep it simple.
I do agree that we should separate tests that we expect to run on any
compliant Java implementation from those that are Harmony specific.
> src/tests/
> |
> + impl/
> |
> + compliant/
Just a minor nit -- we may choose to call these 'api' tests just to be
very clear that we are not defining or modifying the meaning of
'compliant' (which is the role of the JCK).
> + etc [might be stress, exotic, or something else]/
We would create new directories as we need them, right?
> \ resources/ -- as they are
>
> Note: inside 'intention' directory they might be further separated e.g
> by platform
In fact, we should allow for some autonomy within a module too. There
will be a different balance between the types of tests that are required
for, say, LUNI and those for RMI that mean the structure may not suit
every module equally well.
> 2. 'Bootclasspath' tests are in the same package as implementation
Yes -- where do these appear in the layout?
> 3. 'classpath' tests against 'public packages' go to
> org.apache.harmony.module.tests.<public package name>
>
> note: public package name is not necessary in the java.* or javax.*
> example: org.ietf.jgss
Yes.
> 4. 'classpath' tests against org.apache.harmony.module.<rest of the name>
> classes go to
> org.apache.harmony.module.tests.<rest of the name>
Yes.
Thanks for working through this Mikhail -- I feel that we are converging
on a good solution.
Would you be able to put together a (proposed) document describing this
on the classlib website?
Regards,
Tim
--
Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Mikhail Loenko <ml...@gmail.com>.
Did everybody agree to:
1. Tests are separated on directory level by 'intention'
src/tests/
|
+ impl/
|
+ compliant/
|
+ etc [might be stress, exotic, or something else]/
|
\ resources/ -- as they are
Note: inside 'intention' directory they might be further separated e.g
by platform
2. 'Bootclasspath' tests are in the same package as implementation
3. 'classpath' tests against 'public packages' go to
org.apache.harmony.module.tests.<public package name>
note: public package name is not necessary in the java.* or javax.*
example: org.ietf.jgss
4. 'classpath' tests against org.apache.harmony.module.<rest of the name>
classes go to
org.apache.harmony.module.tests.<rest of the name>
Thanks,
Mikhail
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Paulex Yang <pa...@gmail.com>.
Mikhail Loenko wrote:
> 2006/4/28, Paulex Yang <pa...@gmail.com>:
>
>> Mikhail Loenko wrote:
>>
>>> I suggest physically separate high-level suites:
>>>
>>> src/test/
>>> |
>>> +- impl/
>>> |
>>> +- compliant/ [names might be changed]
>>> |
>>> +- stress/
>>> |
>>> \- etc/
>>>
>>> Then on package-level separate classpath from bootclasspath stuff
>>>
>>> 1) We will run compliant to validate test suite
>>> 2) We will not run stress before commit
>>> 3) etc
>>>
>>>
>> Are the src/test/impl or so the src directory?
>>
>
> Sorry I did not understand the question
>
I think I can understand now, it's just directory layout, isn't it? I'm
fine with it.
>
> if so, I'm fine with it.
>
>> And you add stress category we haven't mentioned before, that's fine, too.
>>
>
> I've meant that we have other categories of the tests that will be somewhere
> around.
>
>
>> But I think I didn't catch up, where will you put the case 4 (the ones
>> of [classpath] [dependent] [java.*] )?
>>
>
> folder impl/
> package org.apache.harmony.module.tests.java.something
>
I guess Ant would know it needs to run on classpath by keyword "tests"?
That's fine.
> And how many folders it will be(I
>
>> just not sure what the etc means...;))?
>>
>
> I don't know. Couple month ago we were talking about testing in exotic
> configurations. People suggested separating some types of tests from general
> test run. These types of tests might reside in other 'etc' folders.
>
Agree.
>
> Thanks,
> Mikhail
>
>
>
>
>>> What do you think?
>>>
>>> Thanks,
>>> Mikhail
>>>
>>> 2006/4/28, Paulex Yang <pa...@gmail.com>:
>>>
>>>
>>>> Mikhail Loenko wrote:
>>>>
>>>>
>>>>> Hi Paulex
>>>>>
>>>>> 2006/4/28, Paulex Yang <pa...@gmail.com>:
>>>>>
>>>>>
>>>>>
>>>>>> Mikhail Loenko wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Then how to run only independent tests?
>>>>>>>
>>>>>>> Such a run would make sense to validate the tests or
>>>>>>> if we switch to different RI version
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> It should be easy for ant script to identify independent tests by
>>>>>> package name convention, e.g, we have a keyword "API" for the
>>>>>> independent tests (1), and 2 can be easily excluded.
>>>>>>
>>>>>> For the intended different behavior (4), test can be added to ant's
>>>>>> exclude list (I don't expect there are too many such cases)
>>>>>>
>>>>>>
>>>>>>
>>>>> Did you count such tests? As I remember we have a lof of them in security.
>>>>>
>>>>>
>>>>>
>>>>>
>>>> Lots of ? Ooops...Seems I didn't understand the situation clearly (I
>>>> knew I must miss sth. ;-) )
>>>>
>>>>
>>>>> If these tests are in the same package and in the same directory
>>>>> then we will have something like
>>>>>
>>>>> org.apache.harmony.security.tests.api.java.security.CodeSignerTest1 - impl
>>>>> specific test methods
>>>>>
>>>>> org.apache.harmony.security.tests.api.java.security.CodeSignerTest2 - impl
>>>>> independent test methods
>>>>>
>>>>>
>>>>>
>>>> If there are many similar cases, I agree that it is
>>>> unacceptable(although I think the test case can be named more
>>>> meaningfully than xxxTest1/xxxTest2), because I prefer 1 test for 1
>>>> class in general . It will be perfect if we can introduce some
>>>> mechanism to control test execution in method level, but I have no good
>>>> idea on this so far.
>>>>
>>>> So do you have any better ideas on this issue?
>>>>
>>>>
>>>>> How do you like it?
>>>>>
>>>>> Thanks,
>>>>> Mikhail
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>> Thanks,
>>>>>>> Mikhail
>>>>>>>
>>>>>>> 2006/4/28, Paulex Yang <pa...@gmail.com>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Mikhail Loenko wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> And how would you split 1,2,4,5 between these two folders?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> I thought it is obvious.
>>>>>>>> test 1,2,4 goes to src/test
>>>>>>>> test 5 goes to src/impltest (internl/injected/boot anything is fine)
>>>>>>>> and all implementation goes to src/main
>>>>>>>>
>>>>>>>> and they can be compiled to corresponding bin dir, i.e.
>>>>>>>> bin/test
>>>>>>>> bin/impltest
>>>>>>>> bin/main
>>>>>>>>
>>>>>>>> bin/main and bin/impltest goes to bootclasspath
>>>>>>>> bin/test goes to classpath
>>>>>>>>
>>>>>>>> Pls. correct me if I made mistake or missed sth.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Mikhail
>>>>>>>>>
>>>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Mikhail Loenko wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> If it's separated, I'm OK with it
>>>>>>>>>>>
>>>>>>>>>>> Can you suggest dir names?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Great!
>>>>>>>>>>
>>>>>>>>>> Either boot or injected(as Mark said) sounds good keyword to me. so it
>>>>>>>>>> can be one of below:
>>>>>>>>>> 1. src/test.injected/
>>>>>>>>>> 2. src/test/injected
>>>>>>>>>> 3. src/test.boot/
>>>>>>>>>> 4. src/test/boot
>>>>>>>>>>
>>>>>>>>>> Mmm...seems injected is more self-descriptive, I prefer option 1. So
>>>>>>>>>> the layout would be:
>>>>>>>>>> src/main
>>>>>>>>>> src/test
>>>>>>>>>> src/test.injected
>>>>>>>>>>
>>>>>>>>>> ideas?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Mikhail
>>>>>>>>>>>
>>>>>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Mikhail Loenko wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> The problem is the tests will be mixed like that:
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.harmony.security.asn1 -- tests in bootclasspath
>>>>>>>>>>>>> org.apache.harmony.security.cert -- tests in bootclasspath
>>>>>>>>>>>>> org.apache.harmony.security.tests -- tests in classpath
>>>>>>>>>>>>> org.apache.harmony.security.x509 -- tests in bootclasspath
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> Good catch! :)
>>>>>>>>>>>>
>>>>>>>>>>>> It's not so hard for script to separate, if the package name doesn't
>>>>>>>>>>>> contains "test", it belongs to bootclasspath
>>>>>>>>>>>>
>>>>>>>>>>>> And if you concern about human reading, we can separate src directory
>>>>>>>>>>>> for them, i.e.
>>>>>>>>>>>> src/main/java
>>>>>>>>>>>> src/test/java
>>>>>>>>>>>> src/boottest/java or src/test/boot/java
>>>>>>>>>>>>
>>>>>>>>>>>> comments?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Mikhail
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Mikhail Loenko wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Paulex,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> we have at least 8 categories of tests:
>>>>>>>>>>>>>>> running from classpath or bootclasspath
>>>>>>>>>>>>>>> implementation specific or independent
>>>>>>>>>>>>>>> testing org.apache.harmony.* or java.*
>>>>>>>>>>>>>>> Could you please list how all the tests will be named
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Mikhail,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> OK, let me try. I'm not sure if all 8 categories are necessary,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1. [classpath], [independent], [java.*]
>>>>>>>>>>>>>> o.a.h.<module name>.tests.api.<package name>, e.g.,
>>>>>>>>>>>>>> o.a.h.luni.tests.api.java.util is test package for java.util
>>>>>>>>>>>>>> I consider the module name is optional and can be removed, so it can be
>>>>>>>>>>>>>> o.a.h.tests.api.java.util for java.util
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2. [classpath], [dependent], [o.a.h.*]
>>>>>>>>>>>>>> o.a.h.<module name>.tests.<package name without o.a.h...prefix>,
>>>>>>>>>>>>>> e.g., o.a.h.luni.tests.util is test package for o.a.h.luni.util
>>>>>>>>>>>>>> Again, the module name is optional, but if module name is not
>>>>>>>>>>>>>> prefix, it should be part of package name, so it is:
>>>>>>>>>>>>>> o.a.h.tests.luni.util for o.a.h.luni.util
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 3. [classpath], [independent], [o.a.h.*]
>>>>>>>>>>>>>> I don't expect these classes/tests exist.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 4. [classpath], [dependent], [java.*]
>>>>>>>>>>>>>> if my understanding is correct, this category means the tests for
>>>>>>>>>>>>>> intended different behavior between Harmony and RI, I consider these
>>>>>>>>>>>>>> tests should be put in same package with API tests, and separate them by
>>>>>>>>>>>>>> running script instead of different package, because RI may fix bug in
>>>>>>>>>>>>>> the revision later(e.g, Sun JDK 1.5.1 maybe?), and IIRC in another
>>>>>>>>>>>>>> thread we have agreed that we always follow latest version of RI, so it
>>>>>>>>>>>>>> will be much easier to change the different behavior tests to API tests
>>>>>>>>>>>>>> if we use script to separate them, and vice versa.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 5. [bootclasspath],
>>>>>>>>>>>>>> which covers 4 categories you mentioned, the test cases are in same
>>>>>>>>>>>>>> package with implementions
>>>>>>>>>>>>>> e.g.
>>>>>>>>>>>>>> java.util is test package for java.util
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> o.a.h.luni.util is test packag for o.a.h.luni.util
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As a summary, we'll have four kind of test packages:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> a. o.a.h.<module>.tests.api.java.*** for case 1,4 above
>>>>>>>>>>>>>> b. o.a.h.<module>.tests.*** for case 2 above
>>>>>>>>>>>>>> c. java.***, for case 5 above
>>>>>>>>>>>>>> d. o.a.h.***, for case 5 above
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> comments?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Mikhail
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Mikhail Loenko wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Geir,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The problem is that no one yet suggested a consistent solution
>>>>>>>>>>>>>>>>> that would fit for all the tests and would not get into packages like
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> org.apache.harmony.security.tests.org.apache.harmony.security.util
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Everybody seems to agree that SOME implementation specific tests are
>>>>>>>>>>>>>>>>> in the same package as implementation.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> But if we have ALL the tests in the same package as implementation,
>>>>>>>>>>>>>>>>> then we can't run e.g. tests against java.beans.Beans from classpath, so
>>>>>>>>>>>>>>>>> we can't run regression test for HARMONY-358
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The question is where we put 'classpath' test. If we put them into
>>>>>>>>>>>>>>>>> o.a.h.module.tests.[package under test] we result in that ugly package
>>>>>>>>>>>>>>>>> names like
>>>>>>>>>>>>>>>>> o.a.h.security.tests.o.a.h.security.util
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Mikhail,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Why my prior proposal that o.a.h.security.tests.impl.util is unacceptable?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> And I have no objection if "impl" considered redundant, too.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Mikhail
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2006/4/27, Geir Magnusson Jr <ge...@pobox.com>:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Geir Magnusson Jr wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> George Harley wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Of course, the text module has only "implementation-independent tests
>>>>>>>>>>>>>>>>>>>> that designed to be run from classpath". For modules that have got
>>>>>>>>>>>>>>>>>>>> implementation-specific tests then I suppose we could use something
>>>>>>>>>>>>>>>>>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
>>>>>>>>>>>>>>>>>>>> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
>>>>>>>>>>>>>>>>>>>> I've got no preference.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -1
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Put them in the same package as the class being tested, please, if there
>>>>>>>>>>>>>>>>>>> is no problem being run that way.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Sorry to be so snippy. Been a long day.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Why would we want this convoluted package name? Why wouldn't we want to
>>>>>>>>>>>>>>>>>> let them be in the same package as the class being tested since it's
>>>>>>>>>>>>>>>>>> implementation testing?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> geir
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Paulex Yang
>>>>>>>>>>>>>>>> China Software Development Lab
>>>>>>>>>>>>>>>> IBM
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Paulex Yang
>>>>>>>>>>>>>> China Software Development Lab
>>>>>>>>>>>>>> IBM
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Paulex Yang
>>>>>>>>>>>> China Software Development Lab
>>>>>>>>>>>> IBM
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Paulex Yang
>>>>>>>>>> China Software Development Lab
>>>>>>>>>> IBM
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Paulex Yang
>>>>>>>> China Software Development Lab
>>>>>>>> IBM
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Paulex Yang
>>>>>> China Software Development Lab
>>>>>> IBM
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> Paulex Yang
>>>> China Software Development Lab
>>>> IBM
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>>>
>>>
>> --
>> Paulex Yang
>> China Software Development Lab
>> IBM
>>
>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
>
--
Paulex Yang
China Software Development Lab
IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Mikhail Loenko <ml...@gmail.com>.
2006/4/28, Paulex Yang <pa...@gmail.com>:
> Mikhail Loenko wrote:
> > I suggest physically separate high-level suites:
> >
> > src/test/
> > |
> > +- impl/
> > |
> > +- compliant/ [names might be changed]
> > |
> > +- stress/
> > |
> > \- etc/
> >
> > Then on package-level separate classpath from bootclasspath stuff
> >
> > 1) We will run compliant to validate test suite
> > 2) We will not run stress before commit
> > 3) etc
> >
> Are the src/test/impl or so the src directory?
Sorry I did not understand the question
if so, I'm fine with it.
>
> And you add stress category we haven't mentioned before, that's fine, too.
I've meant that we have other categories of the tests that will be somewhere
around.
>
> But I think I didn't catch up, where will you put the case 4 (the ones
> of [classpath] [dependent] [java.*] )?
folder impl/
package org.apache.harmony.module.tests.java.something
And how many folders it will be(I
> just not sure what the etc means...;))?
I don't know. Couple month ago we were talking about testing in exotic
configurations. People suggested separating some types of tests from general
test run. These types of tests might reside in other 'etc' folders.
Thanks,
Mikhail
> > What do you think?
> >
> > Thanks,
> > Mikhail
> >
> > 2006/4/28, Paulex Yang <pa...@gmail.com>:
> >
> >> Mikhail Loenko wrote:
> >>
> >>> Hi Paulex
> >>>
> >>> 2006/4/28, Paulex Yang <pa...@gmail.com>:
> >>>
> >>>
> >>>> Mikhail Loenko wrote:
> >>>>
> >>>>
> >>>>> Then how to run only independent tests?
> >>>>>
> >>>>> Such a run would make sense to validate the tests or
> >>>>> if we switch to different RI version
> >>>>>
> >>>>>
> >>>>>
> >>>> It should be easy for ant script to identify independent tests by
> >>>> package name convention, e.g, we have a keyword "API" for the
> >>>> independent tests (1), and 2 can be easily excluded.
> >>>>
> >>>> For the intended different behavior (4), test can be added to ant's
> >>>> exclude list (I don't expect there are too many such cases)
> >>>>
> >>>>
> >>> Did you count such tests? As I remember we have a lof of them in security.
> >>>
> >>>
> >>>
> >> Lots of ? Ooops...Seems I didn't understand the situation clearly (I
> >> knew I must miss sth. ;-) )
> >>
> >>> If these tests are in the same package and in the same directory
> >>> then we will have something like
> >>>
> >>> org.apache.harmony.security.tests.api.java.security.CodeSignerTest1 - impl
> >>> specific test methods
> >>>
> >>> org.apache.harmony.security.tests.api.java.security.CodeSignerTest2 - impl
> >>> independent test methods
> >>>
> >>>
> >> If there are many similar cases, I agree that it is
> >> unacceptable(although I think the test case can be named more
> >> meaningfully than xxxTest1/xxxTest2), because I prefer 1 test for 1
> >> class in general . It will be perfect if we can introduce some
> >> mechanism to control test execution in method level, but I have no good
> >> idea on this so far.
> >>
> >> So do you have any better ideas on this issue?
> >>
> >>> How do you like it?
> >>>
> >>> Thanks,
> >>> Mikhail
> >>>
> >>>
> >>>
> >>>>> Thanks,
> >>>>> Mikhail
> >>>>>
> >>>>> 2006/4/28, Paulex Yang <pa...@gmail.com>:
> >>>>>
> >>>>>
> >>>>>
> >>>>>> Mikhail Loenko wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> And how would you split 1,2,4,5 between these two folders?
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> I thought it is obvious.
> >>>>>> test 1,2,4 goes to src/test
> >>>>>> test 5 goes to src/impltest (internl/injected/boot anything is fine)
> >>>>>> and all implementation goes to src/main
> >>>>>>
> >>>>>> and they can be compiled to corresponding bin dir, i.e.
> >>>>>> bin/test
> >>>>>> bin/impltest
> >>>>>> bin/main
> >>>>>>
> >>>>>> bin/main and bin/impltest goes to bootclasspath
> >>>>>> bin/test goes to classpath
> >>>>>>
> >>>>>> Pls. correct me if I made mistake or missed sth.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Thanks,
> >>>>>>> Mikhail
> >>>>>>>
> >>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> Mikhail Loenko wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> If it's separated, I'm OK with it
> >>>>>>>>>
> >>>>>>>>> Can you suggest dir names?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> Great!
> >>>>>>>>
> >>>>>>>> Either boot or injected(as Mark said) sounds good keyword to me. so it
> >>>>>>>> can be one of below:
> >>>>>>>> 1. src/test.injected/
> >>>>>>>> 2. src/test/injected
> >>>>>>>> 3. src/test.boot/
> >>>>>>>> 4. src/test/boot
> >>>>>>>>
> >>>>>>>> Mmm...seems injected is more self-descriptive, I prefer option 1. So
> >>>>>>>> the layout would be:
> >>>>>>>> src/main
> >>>>>>>> src/test
> >>>>>>>> src/test.injected
> >>>>>>>>
> >>>>>>>> ideas?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Mikhail
> >>>>>>>>>
> >>>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> Mikhail Loenko wrote:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> The problem is the tests will be mixed like that:
> >>>>>>>>>>>
> >>>>>>>>>>> org.apache.harmony.security.asn1 -- tests in bootclasspath
> >>>>>>>>>>> org.apache.harmony.security.cert -- tests in bootclasspath
> >>>>>>>>>>> org.apache.harmony.security.tests -- tests in classpath
> >>>>>>>>>>> org.apache.harmony.security.x509 -- tests in bootclasspath
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> Good catch! :)
> >>>>>>>>>>
> >>>>>>>>>> It's not so hard for script to separate, if the package name doesn't
> >>>>>>>>>> contains "test", it belongs to bootclasspath
> >>>>>>>>>>
> >>>>>>>>>> And if you concern about human reading, we can separate src directory
> >>>>>>>>>> for them, i.e.
> >>>>>>>>>> src/main/java
> >>>>>>>>>> src/test/java
> >>>>>>>>>> src/boottest/java or src/test/boot/java
> >>>>>>>>>>
> >>>>>>>>>> comments?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> Mikhail
> >>>>>>>>>>>
> >>>>>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> Mikhail Loenko wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Paulex,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> we have at least 8 categories of tests:
> >>>>>>>>>>>>> running from classpath or bootclasspath
> >>>>>>>>>>>>> implementation specific or independent
> >>>>>>>>>>>>> testing org.apache.harmony.* or java.*
> >>>>>>>>>>>>> Could you please list how all the tests will be named
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>> Mikhail,
> >>>>>>>>>>>>
> >>>>>>>>>>>> OK, let me try. I'm not sure if all 8 categories are necessary,
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1. [classpath], [independent], [java.*]
> >>>>>>>>>>>> o.a.h.<module name>.tests.api.<package name>, e.g.,
> >>>>>>>>>>>> o.a.h.luni.tests.api.java.util is test package for java.util
> >>>>>>>>>>>> I consider the module name is optional and can be removed, so it can be
> >>>>>>>>>>>> o.a.h.tests.api.java.util for java.util
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2. [classpath], [dependent], [o.a.h.*]
> >>>>>>>>>>>> o.a.h.<module name>.tests.<package name without o.a.h...prefix>,
> >>>>>>>>>>>> e.g., o.a.h.luni.tests.util is test package for o.a.h.luni.util
> >>>>>>>>>>>> Again, the module name is optional, but if module name is not
> >>>>>>>>>>>> prefix, it should be part of package name, so it is:
> >>>>>>>>>>>> o.a.h.tests.luni.util for o.a.h.luni.util
> >>>>>>>>>>>>
> >>>>>>>>>>>> 3. [classpath], [independent], [o.a.h.*]
> >>>>>>>>>>>> I don't expect these classes/tests exist.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 4. [classpath], [dependent], [java.*]
> >>>>>>>>>>>> if my understanding is correct, this category means the tests for
> >>>>>>>>>>>> intended different behavior between Harmony and RI, I consider these
> >>>>>>>>>>>> tests should be put in same package with API tests, and separate them by
> >>>>>>>>>>>> running script instead of different package, because RI may fix bug in
> >>>>>>>>>>>> the revision later(e.g, Sun JDK 1.5.1 maybe?), and IIRC in another
> >>>>>>>>>>>> thread we have agreed that we always follow latest version of RI, so it
> >>>>>>>>>>>> will be much easier to change the different behavior tests to API tests
> >>>>>>>>>>>> if we use script to separate them, and vice versa.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 5. [bootclasspath],
> >>>>>>>>>>>> which covers 4 categories you mentioned, the test cases are in same
> >>>>>>>>>>>> package with implementions
> >>>>>>>>>>>> e.g.
> >>>>>>>>>>>> java.util is test package for java.util
> >>>>>>>>>>>> and
> >>>>>>>>>>>> o.a.h.luni.util is test packag for o.a.h.luni.util
> >>>>>>>>>>>>
> >>>>>>>>>>>> As a summary, we'll have four kind of test packages:
> >>>>>>>>>>>>
> >>>>>>>>>>>> a. o.a.h.<module>.tests.api.java.*** for case 1,4 above
> >>>>>>>>>>>> b. o.a.h.<module>.tests.*** for case 2 above
> >>>>>>>>>>>> c. java.***, for case 5 above
> >>>>>>>>>>>> d. o.a.h.***, for case 5 above
> >>>>>>>>>>>>
> >>>>>>>>>>>> comments?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> Mikhail
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Mikhail Loenko wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Geir,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The problem is that no one yet suggested a consistent solution
> >>>>>>>>>>>>>>> that would fit for all the tests and would not get into packages like
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> org.apache.harmony.security.tests.org.apache.harmony.security.util
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Everybody seems to agree that SOME implementation specific tests are
> >>>>>>>>>>>>>>> in the same package as implementation.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> But if we have ALL the tests in the same package as implementation,
> >>>>>>>>>>>>>>> then we can't run e.g. tests against java.beans.Beans from classpath, so
> >>>>>>>>>>>>>>> we can't run regression test for HARMONY-358
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The question is where we put 'classpath' test. If we put them into
> >>>>>>>>>>>>>>> o.a.h.module.tests.[package under test] we result in that ugly package
> >>>>>>>>>>>>>>> names like
> >>>>>>>>>>>>>>> o.a.h.security.tests.o.a.h.security.util
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Mikhail,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Why my prior proposal that o.a.h.security.tests.impl.util is unacceptable?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> And I have no objection if "impl" considered redundant, too.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> Mikhail
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 2006/4/27, Geir Magnusson Jr <ge...@pobox.com>:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Geir Magnusson Jr wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> George Harley wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> >>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Of course, the text module has only "implementation-independent tests
> >>>>>>>>>>>>>>>>>> that designed to be run from classpath". For modules that have got
> >>>>>>>>>>>>>>>>>> implementation-specific tests then I suppose we could use something
> >>>>>>>>>>>>>>>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
> >>>>>>>>>>>>>>>>>> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
> >>>>>>>>>>>>>>>>>> I've got no preference.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> -1
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Put them in the same package as the class being tested, please, if there
> >>>>>>>>>>>>>>>>> is no problem being run that way.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Sorry to be so snippy. Been a long day.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Why would we want this convoluted package name? Why wouldn't we want to
> >>>>>>>>>>>>>>>> let them be in the same package as the class being tested since it's
> >>>>>>>>>>>>>>>> implementation testing?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> geir
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>> Paulex Yang
> >>>>>>>>>>>>>> China Software Development Lab
> >>>>>>>>>>>>>> IBM
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> Paulex Yang
> >>>>>>>>>>>> China Software Development Lab
> >>>>>>>>>>>> IBM
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Paulex Yang
> >>>>>>>>>> China Software Development Lab
> >>>>>>>>>> IBM
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> --
> >>>>>>>> Paulex Yang
> >>>>>>>> China Software Development Lab
> >>>>>>>> IBM
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ---------------------------------------------------------------------
> >>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>> ---------------------------------------------------------------------
> >>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> --
> >>>>>> Paulex Yang
> >>>>>> China Software Development Lab
> >>>>>> IBM
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ---------------------------------------------------------------------
> >>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>> --
> >>>> Paulex Yang
> >>>> China Software Development Lab
> >>>> IBM
> >>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>
> >>>>
> >>>>
> >>>>
> >>> ---------------------------------------------------------------------
> >>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>
> >>>
> >>>
> >>>
> >> --
> >> Paulex Yang
> >> China Software Development Lab
> >> IBM
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
> >
>
>
> --
> Paulex Yang
> China Software Development Lab
> IBM
>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Paulex Yang <pa...@gmail.com>.
Mikhail Loenko wrote:
> I suggest physically separate high-level suites:
>
> src/test/
> |
> +- impl/
> |
> +- compliant/ [names might be changed]
> |
> +- stress/
> |
> \- etc/
>
> Then on package-level separate classpath from bootclasspath stuff
>
> 1) We will run compliant to validate test suite
> 2) We will not run stress before commit
> 3) etc
>
Are the src/test/impl or so the src directory? if so, I'm fine with it.
And you add stress category we haven't mentioned before, that's fine, too.
But I think I didn't catch up, where will you put the case 4 (the ones
of [classpath] [dependent] [java.*] )? And how many folders it will be(I
just not sure what the etc means...;))?
> What do you think?
>
> Thanks,
> Mikhail
>
> 2006/4/28, Paulex Yang <pa...@gmail.com>:
>
>> Mikhail Loenko wrote:
>>
>>> Hi Paulex
>>>
>>> 2006/4/28, Paulex Yang <pa...@gmail.com>:
>>>
>>>
>>>> Mikhail Loenko wrote:
>>>>
>>>>
>>>>> Then how to run only independent tests?
>>>>>
>>>>> Such a run would make sense to validate the tests or
>>>>> if we switch to different RI version
>>>>>
>>>>>
>>>>>
>>>> It should be easy for ant script to identify independent tests by
>>>> package name convention, e.g, we have a keyword "API" for the
>>>> independent tests (1), and 2 can be easily excluded.
>>>>
>>>> For the intended different behavior (4), test can be added to ant's
>>>> exclude list (I don't expect there are too many such cases)
>>>>
>>>>
>>> Did you count such tests? As I remember we have a lof of them in security.
>>>
>>>
>>>
>> Lots of ? Ooops...Seems I didn't understand the situation clearly (I
>> knew I must miss sth. ;-) )
>>
>>> If these tests are in the same package and in the same directory
>>> then we will have something like
>>>
>>> org.apache.harmony.security.tests.api.java.security.CodeSignerTest1 - impl
>>> specific test methods
>>>
>>> org.apache.harmony.security.tests.api.java.security.CodeSignerTest2 - impl
>>> independent test methods
>>>
>>>
>> If there are many similar cases, I agree that it is
>> unacceptable(although I think the test case can be named more
>> meaningfully than xxxTest1/xxxTest2), because I prefer 1 test for 1
>> class in general . It will be perfect if we can introduce some
>> mechanism to control test execution in method level, but I have no good
>> idea on this so far.
>>
>> So do you have any better ideas on this issue?
>>
>>> How do you like it?
>>>
>>> Thanks,
>>> Mikhail
>>>
>>>
>>>
>>>>> Thanks,
>>>>> Mikhail
>>>>>
>>>>> 2006/4/28, Paulex Yang <pa...@gmail.com>:
>>>>>
>>>>>
>>>>>
>>>>>> Mikhail Loenko wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> And how would you split 1,2,4,5 between these two folders?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> I thought it is obvious.
>>>>>> test 1,2,4 goes to src/test
>>>>>> test 5 goes to src/impltest (internl/injected/boot anything is fine)
>>>>>> and all implementation goes to src/main
>>>>>>
>>>>>> and they can be compiled to corresponding bin dir, i.e.
>>>>>> bin/test
>>>>>> bin/impltest
>>>>>> bin/main
>>>>>>
>>>>>> bin/main and bin/impltest goes to bootclasspath
>>>>>> bin/test goes to classpath
>>>>>>
>>>>>> Pls. correct me if I made mistake or missed sth.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Thanks,
>>>>>>> Mikhail
>>>>>>>
>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Mikhail Loenko wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> If it's separated, I'm OK with it
>>>>>>>>>
>>>>>>>>> Can you suggest dir names?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Great!
>>>>>>>>
>>>>>>>> Either boot or injected(as Mark said) sounds good keyword to me. so it
>>>>>>>> can be one of below:
>>>>>>>> 1. src/test.injected/
>>>>>>>> 2. src/test/injected
>>>>>>>> 3. src/test.boot/
>>>>>>>> 4. src/test/boot
>>>>>>>>
>>>>>>>> Mmm...seems injected is more self-descriptive, I prefer option 1. So
>>>>>>>> the layout would be:
>>>>>>>> src/main
>>>>>>>> src/test
>>>>>>>> src/test.injected
>>>>>>>>
>>>>>>>> ideas?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Mikhail
>>>>>>>>>
>>>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Mikhail Loenko wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> The problem is the tests will be mixed like that:
>>>>>>>>>>>
>>>>>>>>>>> org.apache.harmony.security.asn1 -- tests in bootclasspath
>>>>>>>>>>> org.apache.harmony.security.cert -- tests in bootclasspath
>>>>>>>>>>> org.apache.harmony.security.tests -- tests in classpath
>>>>>>>>>>> org.apache.harmony.security.x509 -- tests in bootclasspath
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Good catch! :)
>>>>>>>>>>
>>>>>>>>>> It's not so hard for script to separate, if the package name doesn't
>>>>>>>>>> contains "test", it belongs to bootclasspath
>>>>>>>>>>
>>>>>>>>>> And if you concern about human reading, we can separate src directory
>>>>>>>>>> for them, i.e.
>>>>>>>>>> src/main/java
>>>>>>>>>> src/test/java
>>>>>>>>>> src/boottest/java or src/test/boot/java
>>>>>>>>>>
>>>>>>>>>> comments?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Mikhail
>>>>>>>>>>>
>>>>>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Mikhail Loenko wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Paulex,
>>>>>>>>>>>>>
>>>>>>>>>>>>> we have at least 8 categories of tests:
>>>>>>>>>>>>> running from classpath or bootclasspath
>>>>>>>>>>>>> implementation specific or independent
>>>>>>>>>>>>> testing org.apache.harmony.* or java.*
>>>>>>>>>>>>> Could you please list how all the tests will be named
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> Mikhail,
>>>>>>>>>>>>
>>>>>>>>>>>> OK, let me try. I'm not sure if all 8 categories are necessary,
>>>>>>>>>>>>
>>>>>>>>>>>> 1. [classpath], [independent], [java.*]
>>>>>>>>>>>> o.a.h.<module name>.tests.api.<package name>, e.g.,
>>>>>>>>>>>> o.a.h.luni.tests.api.java.util is test package for java.util
>>>>>>>>>>>> I consider the module name is optional and can be removed, so it can be
>>>>>>>>>>>> o.a.h.tests.api.java.util for java.util
>>>>>>>>>>>>
>>>>>>>>>>>> 2. [classpath], [dependent], [o.a.h.*]
>>>>>>>>>>>> o.a.h.<module name>.tests.<package name without o.a.h...prefix>,
>>>>>>>>>>>> e.g., o.a.h.luni.tests.util is test package for o.a.h.luni.util
>>>>>>>>>>>> Again, the module name is optional, but if module name is not
>>>>>>>>>>>> prefix, it should be part of package name, so it is:
>>>>>>>>>>>> o.a.h.tests.luni.util for o.a.h.luni.util
>>>>>>>>>>>>
>>>>>>>>>>>> 3. [classpath], [independent], [o.a.h.*]
>>>>>>>>>>>> I don't expect these classes/tests exist.
>>>>>>>>>>>>
>>>>>>>>>>>> 4. [classpath], [dependent], [java.*]
>>>>>>>>>>>> if my understanding is correct, this category means the tests for
>>>>>>>>>>>> intended different behavior between Harmony and RI, I consider these
>>>>>>>>>>>> tests should be put in same package with API tests, and separate them by
>>>>>>>>>>>> running script instead of different package, because RI may fix bug in
>>>>>>>>>>>> the revision later(e.g, Sun JDK 1.5.1 maybe?), and IIRC in another
>>>>>>>>>>>> thread we have agreed that we always follow latest version of RI, so it
>>>>>>>>>>>> will be much easier to change the different behavior tests to API tests
>>>>>>>>>>>> if we use script to separate them, and vice versa.
>>>>>>>>>>>>
>>>>>>>>>>>> 5. [bootclasspath],
>>>>>>>>>>>> which covers 4 categories you mentioned, the test cases are in same
>>>>>>>>>>>> package with implementions
>>>>>>>>>>>> e.g.
>>>>>>>>>>>> java.util is test package for java.util
>>>>>>>>>>>> and
>>>>>>>>>>>> o.a.h.luni.util is test packag for o.a.h.luni.util
>>>>>>>>>>>>
>>>>>>>>>>>> As a summary, we'll have four kind of test packages:
>>>>>>>>>>>>
>>>>>>>>>>>> a. o.a.h.<module>.tests.api.java.*** for case 1,4 above
>>>>>>>>>>>> b. o.a.h.<module>.tests.*** for case 2 above
>>>>>>>>>>>> c. java.***, for case 5 above
>>>>>>>>>>>> d. o.a.h.***, for case 5 above
>>>>>>>>>>>>
>>>>>>>>>>>> comments?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Mikhail
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Mikhail Loenko wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Geir,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The problem is that no one yet suggested a consistent solution
>>>>>>>>>>>>>>> that would fit for all the tests and would not get into packages like
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.harmony.security.tests.org.apache.harmony.security.util
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Everybody seems to agree that SOME implementation specific tests are
>>>>>>>>>>>>>>> in the same package as implementation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> But if we have ALL the tests in the same package as implementation,
>>>>>>>>>>>>>>> then we can't run e.g. tests against java.beans.Beans from classpath, so
>>>>>>>>>>>>>>> we can't run regression test for HARMONY-358
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The question is where we put 'classpath' test. If we put them into
>>>>>>>>>>>>>>> o.a.h.module.tests.[package under test] we result in that ugly package
>>>>>>>>>>>>>>> names like
>>>>>>>>>>>>>>> o.a.h.security.tests.o.a.h.security.util
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Mikhail,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Why my prior proposal that o.a.h.security.tests.impl.util is unacceptable?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And I have no objection if "impl" considered redundant, too.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Mikhail
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2006/4/27, Geir Magnusson Jr <ge...@pobox.com>:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Geir Magnusson Jr wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> George Harley wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Of course, the text module has only "implementation-independent tests
>>>>>>>>>>>>>>>>>> that designed to be run from classpath". For modules that have got
>>>>>>>>>>>>>>>>>> implementation-specific tests then I suppose we could use something
>>>>>>>>>>>>>>>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
>>>>>>>>>>>>>>>>>> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
>>>>>>>>>>>>>>>>>> I've got no preference.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -1
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Put them in the same package as the class being tested, please, if there
>>>>>>>>>>>>>>>>> is no problem being run that way.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sorry to be so snippy. Been a long day.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Why would we want this convoluted package name? Why wouldn't we want to
>>>>>>>>>>>>>>>> let them be in the same package as the class being tested since it's
>>>>>>>>>>>>>>>> implementation testing?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> geir
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Paulex Yang
>>>>>>>>>>>>>> China Software Development Lab
>>>>>>>>>>>>>> IBM
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Paulex Yang
>>>>>>>>>>>> China Software Development Lab
>>>>>>>>>>>> IBM
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Paulex Yang
>>>>>>>>>> China Software Development Lab
>>>>>>>>>> IBM
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Paulex Yang
>>>>>>>> China Software Development Lab
>>>>>>>> IBM
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Paulex Yang
>>>>>> China Software Development Lab
>>>>>> IBM
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> Paulex Yang
>>>> China Software Development Lab
>>>> IBM
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>>>
>>>
>> --
>> Paulex Yang
>> China Software Development Lab
>> IBM
>>
>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
>
--
Paulex Yang
China Software Development Lab
IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Geir Magnusson Jr <ge...@pobox.com>.
I think capturing as much information in directory layout as possible is
a good thing - scripts can always combine stuff into jars or whatever as
needed. It's far easier to scramble an egg compared to unscrambling it...
geir
Mikhail Loenko wrote:
> I suggest physically separate high-level suites:
>
> src/test/
> |
> +- impl/
> |
> +- compliant/ [names might be changed]
> |
> +- stress/
> |
> \- etc/
>
> Then on package-level separate classpath from bootclasspath stuff
>
> 1) We will run compliant to validate test suite
> 2) We will not run stress before commit
> 3) etc
>
> What do you think?
>
> Thanks,
> Mikhail
>
> 2006/4/28, Paulex Yang <pa...@gmail.com>:
>> Mikhail Loenko wrote:
>>> Hi Paulex
>>>
>>> 2006/4/28, Paulex Yang <pa...@gmail.com>:
>>>
>>>> Mikhail Loenko wrote:
>>>>
>>>>> Then how to run only independent tests?
>>>>>
>>>>> Such a run would make sense to validate the tests or
>>>>> if we switch to different RI version
>>>>>
>>>>>
>>>> It should be easy for ant script to identify independent tests by
>>>> package name convention, e.g, we have a keyword "API" for the
>>>> independent tests (1), and 2 can be easily excluded.
>>>>
>>>> For the intended different behavior (4), test can be added to ant's
>>>> exclude list (I don't expect there are too many such cases)
>>>>
>>> Did you count such tests? As I remember we have a lof of them in security.
>>>
>>>
>> Lots of ? Ooops...Seems I didn't understand the situation clearly (I
>> knew I must miss sth. ;-) )
>>> If these tests are in the same package and in the same directory
>>> then we will have something like
>>>
>>> org.apache.harmony.security.tests.api.java.security.CodeSignerTest1 - impl
>>> specific test methods
>>>
>>> org.apache.harmony.security.tests.api.java.security.CodeSignerTest2 - impl
>>> independent test methods
>>>
>> If there are many similar cases, I agree that it is
>> unacceptable(although I think the test case can be named more
>> meaningfully than xxxTest1/xxxTest2), because I prefer 1 test for 1
>> class in general . It will be perfect if we can introduce some
>> mechanism to control test execution in method level, but I have no good
>> idea on this so far.
>>
>> So do you have any better ideas on this issue?
>>> How do you like it?
>>>
>>> Thanks,
>>> Mikhail
>>>
>>>
>>>>> Thanks,
>>>>> Mikhail
>>>>>
>>>>> 2006/4/28, Paulex Yang <pa...@gmail.com>:
>>>>>
>>>>>
>>>>>> Mikhail Loenko wrote:
>>>>>>
>>>>>>
>>>>>>> And how would you split 1,2,4,5 between these two folders?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> I thought it is obvious.
>>>>>> test 1,2,4 goes to src/test
>>>>>> test 5 goes to src/impltest (internl/injected/boot anything is fine)
>>>>>> and all implementation goes to src/main
>>>>>>
>>>>>> and they can be compiled to corresponding bin dir, i.e.
>>>>>> bin/test
>>>>>> bin/impltest
>>>>>> bin/main
>>>>>>
>>>>>> bin/main and bin/impltest goes to bootclasspath
>>>>>> bin/test goes to classpath
>>>>>>
>>>>>> Pls. correct me if I made mistake or missed sth.
>>>>>>
>>>>>>
>>>>>>> Thanks,
>>>>>>> Mikhail
>>>>>>>
>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Mikhail Loenko wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> If it's separated, I'm OK with it
>>>>>>>>>
>>>>>>>>> Can you suggest dir names?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Great!
>>>>>>>>
>>>>>>>> Either boot or injected(as Mark said) sounds good keyword to me. so it
>>>>>>>> can be one of below:
>>>>>>>> 1. src/test.injected/
>>>>>>>> 2. src/test/injected
>>>>>>>> 3. src/test.boot/
>>>>>>>> 4. src/test/boot
>>>>>>>>
>>>>>>>> Mmm...seems injected is more self-descriptive, I prefer option 1. So
>>>>>>>> the layout would be:
>>>>>>>> src/main
>>>>>>>> src/test
>>>>>>>> src/test.injected
>>>>>>>>
>>>>>>>> ideas?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Mikhail
>>>>>>>>>
>>>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Mikhail Loenko wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> The problem is the tests will be mixed like that:
>>>>>>>>>>>
>>>>>>>>>>> org.apache.harmony.security.asn1 -- tests in bootclasspath
>>>>>>>>>>> org.apache.harmony.security.cert -- tests in bootclasspath
>>>>>>>>>>> org.apache.harmony.security.tests -- tests in classpath
>>>>>>>>>>> org.apache.harmony.security.x509 -- tests in bootclasspath
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Good catch! :)
>>>>>>>>>>
>>>>>>>>>> It's not so hard for script to separate, if the package name doesn't
>>>>>>>>>> contains "test", it belongs to bootclasspath
>>>>>>>>>>
>>>>>>>>>> And if you concern about human reading, we can separate src directory
>>>>>>>>>> for them, i.e.
>>>>>>>>>> src/main/java
>>>>>>>>>> src/test/java
>>>>>>>>>> src/boottest/java or src/test/boot/java
>>>>>>>>>>
>>>>>>>>>> comments?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Mikhail
>>>>>>>>>>>
>>>>>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Mikhail Loenko wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Paulex,
>>>>>>>>>>>>>
>>>>>>>>>>>>> we have at least 8 categories of tests:
>>>>>>>>>>>>> running from classpath or bootclasspath
>>>>>>>>>>>>> implementation specific or independent
>>>>>>>>>>>>> testing org.apache.harmony.* or java.*
>>>>>>>>>>>>> Could you please list how all the tests will be named
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> Mikhail,
>>>>>>>>>>>>
>>>>>>>>>>>> OK, let me try. I'm not sure if all 8 categories are necessary,
>>>>>>>>>>>>
>>>>>>>>>>>> 1. [classpath], [independent], [java.*]
>>>>>>>>>>>> o.a.h.<module name>.tests.api.<package name>, e.g.,
>>>>>>>>>>>> o.a.h.luni.tests.api.java.util is test package for java.util
>>>>>>>>>>>> I consider the module name is optional and can be removed, so it can be
>>>>>>>>>>>> o.a.h.tests.api.java.util for java.util
>>>>>>>>>>>>
>>>>>>>>>>>> 2. [classpath], [dependent], [o.a.h.*]
>>>>>>>>>>>> o.a.h.<module name>.tests.<package name without o.a.h...prefix>,
>>>>>>>>>>>> e.g., o.a.h.luni.tests.util is test package for o.a.h.luni.util
>>>>>>>>>>>> Again, the module name is optional, but if module name is not
>>>>>>>>>>>> prefix, it should be part of package name, so it is:
>>>>>>>>>>>> o.a.h.tests.luni.util for o.a.h.luni.util
>>>>>>>>>>>>
>>>>>>>>>>>> 3. [classpath], [independent], [o.a.h.*]
>>>>>>>>>>>> I don't expect these classes/tests exist.
>>>>>>>>>>>>
>>>>>>>>>>>> 4. [classpath], [dependent], [java.*]
>>>>>>>>>>>> if my understanding is correct, this category means the tests for
>>>>>>>>>>>> intended different behavior between Harmony and RI, I consider these
>>>>>>>>>>>> tests should be put in same package with API tests, and separate them by
>>>>>>>>>>>> running script instead of different package, because RI may fix bug in
>>>>>>>>>>>> the revision later(e.g, Sun JDK 1.5.1 maybe?), and IIRC in another
>>>>>>>>>>>> thread we have agreed that we always follow latest version of RI, so it
>>>>>>>>>>>> will be much easier to change the different behavior tests to API tests
>>>>>>>>>>>> if we use script to separate them, and vice versa.
>>>>>>>>>>>>
>>>>>>>>>>>> 5. [bootclasspath],
>>>>>>>>>>>> which covers 4 categories you mentioned, the test cases are in same
>>>>>>>>>>>> package with implementions
>>>>>>>>>>>> e.g.
>>>>>>>>>>>> java.util is test package for java.util
>>>>>>>>>>>> and
>>>>>>>>>>>> o.a.h.luni.util is test packag for o.a.h.luni.util
>>>>>>>>>>>>
>>>>>>>>>>>> As a summary, we'll have four kind of test packages:
>>>>>>>>>>>>
>>>>>>>>>>>> a. o.a.h.<module>.tests.api.java.*** for case 1,4 above
>>>>>>>>>>>> b. o.a.h.<module>.tests.*** for case 2 above
>>>>>>>>>>>> c. java.***, for case 5 above
>>>>>>>>>>>> d. o.a.h.***, for case 5 above
>>>>>>>>>>>>
>>>>>>>>>>>> comments?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Mikhail
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Mikhail Loenko wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Geir,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The problem is that no one yet suggested a consistent solution
>>>>>>>>>>>>>>> that would fit for all the tests and would not get into packages like
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.harmony.security.tests.org.apache.harmony.security.util
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Everybody seems to agree that SOME implementation specific tests are
>>>>>>>>>>>>>>> in the same package as implementation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> But if we have ALL the tests in the same package as implementation,
>>>>>>>>>>>>>>> then we can't run e.g. tests against java.beans.Beans from classpath, so
>>>>>>>>>>>>>>> we can't run regression test for HARMONY-358
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The question is where we put 'classpath' test. If we put them into
>>>>>>>>>>>>>>> o.a.h.module.tests.[package under test] we result in that ugly package
>>>>>>>>>>>>>>> names like
>>>>>>>>>>>>>>> o.a.h.security.tests.o.a.h.security.util
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Mikhail,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Why my prior proposal that o.a.h.security.tests.impl.util is unacceptable?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And I have no objection if "impl" considered redundant, too.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Mikhail
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2006/4/27, Geir Magnusson Jr <ge...@pobox.com>:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Geir Magnusson Jr wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> George Harley wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Of course, the text module has only "implementation-independent tests
>>>>>>>>>>>>>>>>>> that designed to be run from classpath". For modules that have got
>>>>>>>>>>>>>>>>>> implementation-specific tests then I suppose we could use something
>>>>>>>>>>>>>>>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
>>>>>>>>>>>>>>>>>> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
>>>>>>>>>>>>>>>>>> I've got no preference.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -1
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Put them in the same package as the class being tested, please, if there
>>>>>>>>>>>>>>>>> is no problem being run that way.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sorry to be so snippy. Been a long day.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Why would we want this convoluted package name? Why wouldn't we want to
>>>>>>>>>>>>>>>> let them be in the same package as the class being tested since it's
>>>>>>>>>>>>>>>> implementation testing?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> geir
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Paulex Yang
>>>>>>>>>>>>>> China Software Development Lab
>>>>>>>>>>>>>> IBM
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Paulex Yang
>>>>>>>>>>>> China Software Development Lab
>>>>>>>>>>>> IBM
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Paulex Yang
>>>>>>>>>> China Software Development Lab
>>>>>>>>>> IBM
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Paulex Yang
>>>>>>>> China Software Development Lab
>>>>>>>> IBM
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Paulex Yang
>>>>>> China Software Development Lab
>>>>>> IBM
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> Paulex Yang
>>>> China Software Development Lab
>>>> IBM
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>>>
>>
>> --
>> Paulex Yang
>> China Software Development Lab
>> IBM
>>
>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Mikhail Loenko <ml...@gmail.com>.
I suggest physically separate high-level suites:
src/test/
|
+- impl/
|
+- compliant/ [names might be changed]
|
+- stress/
|
\- etc/
Then on package-level separate classpath from bootclasspath stuff
1) We will run compliant to validate test suite
2) We will not run stress before commit
3) etc
What do you think?
Thanks,
Mikhail
2006/4/28, Paulex Yang <pa...@gmail.com>:
> Mikhail Loenko wrote:
> > Hi Paulex
> >
> > 2006/4/28, Paulex Yang <pa...@gmail.com>:
> >
> >> Mikhail Loenko wrote:
> >>
> >>> Then how to run only independent tests?
> >>>
> >>> Such a run would make sense to validate the tests or
> >>> if we switch to different RI version
> >>>
> >>>
> >> It should be easy for ant script to identify independent tests by
> >> package name convention, e.g, we have a keyword "API" for the
> >> independent tests (1), and 2 can be easily excluded.
> >>
> >> For the intended different behavior (4), test can be added to ant's
> >> exclude list (I don't expect there are too many such cases)
> >>
> >
> > Did you count such tests? As I remember we have a lof of them in security.
> >
> >
> Lots of ? Ooops...Seems I didn't understand the situation clearly (I
> knew I must miss sth. ;-) )
> > If these tests are in the same package and in the same directory
> > then we will have something like
> >
> > org.apache.harmony.security.tests.api.java.security.CodeSignerTest1 - impl
> > specific test methods
> >
> > org.apache.harmony.security.tests.api.java.security.CodeSignerTest2 - impl
> > independent test methods
> >
> If there are many similar cases, I agree that it is
> unacceptable(although I think the test case can be named more
> meaningfully than xxxTest1/xxxTest2), because I prefer 1 test for 1
> class in general . It will be perfect if we can introduce some
> mechanism to control test execution in method level, but I have no good
> idea on this so far.
>
> So do you have any better ideas on this issue?
> > How do you like it?
> >
> > Thanks,
> > Mikhail
> >
> >
> >>> Thanks,
> >>> Mikhail
> >>>
> >>> 2006/4/28, Paulex Yang <pa...@gmail.com>:
> >>>
> >>>
> >>>> Mikhail Loenko wrote:
> >>>>
> >>>>
> >>>>> And how would you split 1,2,4,5 between these two folders?
> >>>>>
> >>>>>
> >>>>>
> >>>> I thought it is obvious.
> >>>> test 1,2,4 goes to src/test
> >>>> test 5 goes to src/impltest (internl/injected/boot anything is fine)
> >>>> and all implementation goes to src/main
> >>>>
> >>>> and they can be compiled to corresponding bin dir, i.e.
> >>>> bin/test
> >>>> bin/impltest
> >>>> bin/main
> >>>>
> >>>> bin/main and bin/impltest goes to bootclasspath
> >>>> bin/test goes to classpath
> >>>>
> >>>> Pls. correct me if I made mistake or missed sth.
> >>>>
> >>>>
> >>>>> Thanks,
> >>>>> Mikhail
> >>>>>
> >>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
> >>>>>
> >>>>>
> >>>>>
> >>>>>> Mikhail Loenko wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> If it's separated, I'm OK with it
> >>>>>>>
> >>>>>>> Can you suggest dir names?
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> Great!
> >>>>>>
> >>>>>> Either boot or injected(as Mark said) sounds good keyword to me. so it
> >>>>>> can be one of below:
> >>>>>> 1. src/test.injected/
> >>>>>> 2. src/test/injected
> >>>>>> 3. src/test.boot/
> >>>>>> 4. src/test/boot
> >>>>>>
> >>>>>> Mmm...seems injected is more self-descriptive, I prefer option 1. So
> >>>>>> the layout would be:
> >>>>>> src/main
> >>>>>> src/test
> >>>>>> src/test.injected
> >>>>>>
> >>>>>> ideas?
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Thanks,
> >>>>>>> Mikhail
> >>>>>>>
> >>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> Mikhail Loenko wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> The problem is the tests will be mixed like that:
> >>>>>>>>>
> >>>>>>>>> org.apache.harmony.security.asn1 -- tests in bootclasspath
> >>>>>>>>> org.apache.harmony.security.cert -- tests in bootclasspath
> >>>>>>>>> org.apache.harmony.security.tests -- tests in classpath
> >>>>>>>>> org.apache.harmony.security.x509 -- tests in bootclasspath
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> Good catch! :)
> >>>>>>>>
> >>>>>>>> It's not so hard for script to separate, if the package name doesn't
> >>>>>>>> contains "test", it belongs to bootclasspath
> >>>>>>>>
> >>>>>>>> And if you concern about human reading, we can separate src directory
> >>>>>>>> for them, i.e.
> >>>>>>>> src/main/java
> >>>>>>>> src/test/java
> >>>>>>>> src/boottest/java or src/test/boot/java
> >>>>>>>>
> >>>>>>>> comments?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Mikhail
> >>>>>>>>>
> >>>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> Mikhail Loenko wrote:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Paulex,
> >>>>>>>>>>>
> >>>>>>>>>>> we have at least 8 categories of tests:
> >>>>>>>>>>> running from classpath or bootclasspath
> >>>>>>>>>>> implementation specific or independent
> >>>>>>>>>>> testing org.apache.harmony.* or java.*
> >>>>>>>>>>> Could you please list how all the tests will be named
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> Mikhail,
> >>>>>>>>>>
> >>>>>>>>>> OK, let me try. I'm not sure if all 8 categories are necessary,
> >>>>>>>>>>
> >>>>>>>>>> 1. [classpath], [independent], [java.*]
> >>>>>>>>>> o.a.h.<module name>.tests.api.<package name>, e.g.,
> >>>>>>>>>> o.a.h.luni.tests.api.java.util is test package for java.util
> >>>>>>>>>> I consider the module name is optional and can be removed, so it can be
> >>>>>>>>>> o.a.h.tests.api.java.util for java.util
> >>>>>>>>>>
> >>>>>>>>>> 2. [classpath], [dependent], [o.a.h.*]
> >>>>>>>>>> o.a.h.<module name>.tests.<package name without o.a.h...prefix>,
> >>>>>>>>>> e.g., o.a.h.luni.tests.util is test package for o.a.h.luni.util
> >>>>>>>>>> Again, the module name is optional, but if module name is not
> >>>>>>>>>> prefix, it should be part of package name, so it is:
> >>>>>>>>>> o.a.h.tests.luni.util for o.a.h.luni.util
> >>>>>>>>>>
> >>>>>>>>>> 3. [classpath], [independent], [o.a.h.*]
> >>>>>>>>>> I don't expect these classes/tests exist.
> >>>>>>>>>>
> >>>>>>>>>> 4. [classpath], [dependent], [java.*]
> >>>>>>>>>> if my understanding is correct, this category means the tests for
> >>>>>>>>>> intended different behavior between Harmony and RI, I consider these
> >>>>>>>>>> tests should be put in same package with API tests, and separate them by
> >>>>>>>>>> running script instead of different package, because RI may fix bug in
> >>>>>>>>>> the revision later(e.g, Sun JDK 1.5.1 maybe?), and IIRC in another
> >>>>>>>>>> thread we have agreed that we always follow latest version of RI, so it
> >>>>>>>>>> will be much easier to change the different behavior tests to API tests
> >>>>>>>>>> if we use script to separate them, and vice versa.
> >>>>>>>>>>
> >>>>>>>>>> 5. [bootclasspath],
> >>>>>>>>>> which covers 4 categories you mentioned, the test cases are in same
> >>>>>>>>>> package with implementions
> >>>>>>>>>> e.g.
> >>>>>>>>>> java.util is test package for java.util
> >>>>>>>>>> and
> >>>>>>>>>> o.a.h.luni.util is test packag for o.a.h.luni.util
> >>>>>>>>>>
> >>>>>>>>>> As a summary, we'll have four kind of test packages:
> >>>>>>>>>>
> >>>>>>>>>> a. o.a.h.<module>.tests.api.java.*** for case 1,4 above
> >>>>>>>>>> b. o.a.h.<module>.tests.*** for case 2 above
> >>>>>>>>>> c. java.***, for case 5 above
> >>>>>>>>>> d. o.a.h.***, for case 5 above
> >>>>>>>>>>
> >>>>>>>>>> comments?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> Mikhail
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> Mikhail Loenko wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Geir,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The problem is that no one yet suggested a consistent solution
> >>>>>>>>>>>>> that would fit for all the tests and would not get into packages like
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> org.apache.harmony.security.tests.org.apache.harmony.security.util
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Everybody seems to agree that SOME implementation specific tests are
> >>>>>>>>>>>>> in the same package as implementation.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> But if we have ALL the tests in the same package as implementation,
> >>>>>>>>>>>>> then we can't run e.g. tests against java.beans.Beans from classpath, so
> >>>>>>>>>>>>> we can't run regression test for HARMONY-358
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The question is where we put 'classpath' test. If we put them into
> >>>>>>>>>>>>> o.a.h.module.tests.[package under test] we result in that ugly package
> >>>>>>>>>>>>> names like
> >>>>>>>>>>>>> o.a.h.security.tests.o.a.h.security.util
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>> Mikhail,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Why my prior proposal that o.a.h.security.tests.impl.util is unacceptable?
> >>>>>>>>>>>>
> >>>>>>>>>>>> And I have no objection if "impl" considered redundant, too.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> Mikhail
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 2006/4/27, Geir Magnusson Jr <ge...@pobox.com>:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Geir Magnusson Jr wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> George Harley wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> >>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Of course, the text module has only "implementation-independent tests
> >>>>>>>>>>>>>>>> that designed to be run from classpath". For modules that have got
> >>>>>>>>>>>>>>>> implementation-specific tests then I suppose we could use something
> >>>>>>>>>>>>>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
> >>>>>>>>>>>>>>>> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
> >>>>>>>>>>>>>>>> I've got no preference.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -1
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Put them in the same package as the class being tested, please, if there
> >>>>>>>>>>>>>>> is no problem being run that way.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Sorry to be so snippy. Been a long day.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Why would we want this convoluted package name? Why wouldn't we want to
> >>>>>>>>>>>>>> let them be in the same package as the class being tested since it's
> >>>>>>>>>>>>>> implementation testing?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> geir
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> Paulex Yang
> >>>>>>>>>>>> China Software Development Lab
> >>>>>>>>>>>> IBM
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Paulex Yang
> >>>>>>>>>> China Software Development Lab
> >>>>>>>>>> IBM
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> --
> >>>>>>>> Paulex Yang
> >>>>>>>> China Software Development Lab
> >>>>>>>> IBM
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ---------------------------------------------------------------------
> >>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>> ---------------------------------------------------------------------
> >>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> --
> >>>>>> Paulex Yang
> >>>>>> China Software Development Lab
> >>>>>> IBM
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ---------------------------------------------------------------------
> >>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>> --
> >>>> Paulex Yang
> >>>> China Software Development Lab
> >>>> IBM
> >>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>
> >>>>
> >>>>
> >>>>
> >>> ---------------------------------------------------------------------
> >>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>
> >>>
> >>>
> >>>
> >> --
> >> Paulex Yang
> >> China Software Development Lab
> >> IBM
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
> >
>
>
> --
> Paulex Yang
> China Software Development Lab
> IBM
>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Paulex Yang <pa...@gmail.com>.
Mikhail Loenko wrote:
> Hi Paulex
>
> 2006/4/28, Paulex Yang <pa...@gmail.com>:
>
>> Mikhail Loenko wrote:
>>
>>> Then how to run only independent tests?
>>>
>>> Such a run would make sense to validate the tests or
>>> if we switch to different RI version
>>>
>>>
>> It should be easy for ant script to identify independent tests by
>> package name convention, e.g, we have a keyword "API" for the
>> independent tests (1), and 2 can be easily excluded.
>>
>> For the intended different behavior (4), test can be added to ant's
>> exclude list (I don't expect there are too many such cases)
>>
>
> Did you count such tests? As I remember we have a lof of them in security.
>
>
Lots of ? Ooops...Seems I didn't understand the situation clearly (I
knew I must miss sth. ;-) )
> If these tests are in the same package and in the same directory
> then we will have something like
>
> org.apache.harmony.security.tests.api.java.security.CodeSignerTest1 - impl
> specific test methods
>
> org.apache.harmony.security.tests.api.java.security.CodeSignerTest2 - impl
> independent test methods
>
If there are many similar cases, I agree that it is
unacceptable(although I think the test case can be named more
meaningfully than xxxTest1/xxxTest2), because I prefer 1 test for 1
class in general . It will be perfect if we can introduce some
mechanism to control test execution in method level, but I have no good
idea on this so far.
So do you have any better ideas on this issue?
> How do you like it?
>
> Thanks,
> Mikhail
>
>
>>> Thanks,
>>> Mikhail
>>>
>>> 2006/4/28, Paulex Yang <pa...@gmail.com>:
>>>
>>>
>>>> Mikhail Loenko wrote:
>>>>
>>>>
>>>>> And how would you split 1,2,4,5 between these two folders?
>>>>>
>>>>>
>>>>>
>>>> I thought it is obvious.
>>>> test 1,2,4 goes to src/test
>>>> test 5 goes to src/impltest (internl/injected/boot anything is fine)
>>>> and all implementation goes to src/main
>>>>
>>>> and they can be compiled to corresponding bin dir, i.e.
>>>> bin/test
>>>> bin/impltest
>>>> bin/main
>>>>
>>>> bin/main and bin/impltest goes to bootclasspath
>>>> bin/test goes to classpath
>>>>
>>>> Pls. correct me if I made mistake or missed sth.
>>>>
>>>>
>>>>> Thanks,
>>>>> Mikhail
>>>>>
>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>>>
>>>>>
>>>>>
>>>>>> Mikhail Loenko wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> If it's separated, I'm OK with it
>>>>>>>
>>>>>>> Can you suggest dir names?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Great!
>>>>>>
>>>>>> Either boot or injected(as Mark said) sounds good keyword to me. so it
>>>>>> can be one of below:
>>>>>> 1. src/test.injected/
>>>>>> 2. src/test/injected
>>>>>> 3. src/test.boot/
>>>>>> 4. src/test/boot
>>>>>>
>>>>>> Mmm...seems injected is more self-descriptive, I prefer option 1. So
>>>>>> the layout would be:
>>>>>> src/main
>>>>>> src/test
>>>>>> src/test.injected
>>>>>>
>>>>>> ideas?
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Thanks,
>>>>>>> Mikhail
>>>>>>>
>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Mikhail Loenko wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> The problem is the tests will be mixed like that:
>>>>>>>>>
>>>>>>>>> org.apache.harmony.security.asn1 -- tests in bootclasspath
>>>>>>>>> org.apache.harmony.security.cert -- tests in bootclasspath
>>>>>>>>> org.apache.harmony.security.tests -- tests in classpath
>>>>>>>>> org.apache.harmony.security.x509 -- tests in bootclasspath
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Good catch! :)
>>>>>>>>
>>>>>>>> It's not so hard for script to separate, if the package name doesn't
>>>>>>>> contains "test", it belongs to bootclasspath
>>>>>>>>
>>>>>>>> And if you concern about human reading, we can separate src directory
>>>>>>>> for them, i.e.
>>>>>>>> src/main/java
>>>>>>>> src/test/java
>>>>>>>> src/boottest/java or src/test/boot/java
>>>>>>>>
>>>>>>>> comments?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Mikhail
>>>>>>>>>
>>>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Mikhail Loenko wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Paulex,
>>>>>>>>>>>
>>>>>>>>>>> we have at least 8 categories of tests:
>>>>>>>>>>> running from classpath or bootclasspath
>>>>>>>>>>> implementation specific or independent
>>>>>>>>>>> testing org.apache.harmony.* or java.*
>>>>>>>>>>> Could you please list how all the tests will be named
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Mikhail,
>>>>>>>>>>
>>>>>>>>>> OK, let me try. I'm not sure if all 8 categories are necessary,
>>>>>>>>>>
>>>>>>>>>> 1. [classpath], [independent], [java.*]
>>>>>>>>>> o.a.h.<module name>.tests.api.<package name>, e.g.,
>>>>>>>>>> o.a.h.luni.tests.api.java.util is test package for java.util
>>>>>>>>>> I consider the module name is optional and can be removed, so it can be
>>>>>>>>>> o.a.h.tests.api.java.util for java.util
>>>>>>>>>>
>>>>>>>>>> 2. [classpath], [dependent], [o.a.h.*]
>>>>>>>>>> o.a.h.<module name>.tests.<package name without o.a.h...prefix>,
>>>>>>>>>> e.g., o.a.h.luni.tests.util is test package for o.a.h.luni.util
>>>>>>>>>> Again, the module name is optional, but if module name is not
>>>>>>>>>> prefix, it should be part of package name, so it is:
>>>>>>>>>> o.a.h.tests.luni.util for o.a.h.luni.util
>>>>>>>>>>
>>>>>>>>>> 3. [classpath], [independent], [o.a.h.*]
>>>>>>>>>> I don't expect these classes/tests exist.
>>>>>>>>>>
>>>>>>>>>> 4. [classpath], [dependent], [java.*]
>>>>>>>>>> if my understanding is correct, this category means the tests for
>>>>>>>>>> intended different behavior between Harmony and RI, I consider these
>>>>>>>>>> tests should be put in same package with API tests, and separate them by
>>>>>>>>>> running script instead of different package, because RI may fix bug in
>>>>>>>>>> the revision later(e.g, Sun JDK 1.5.1 maybe?), and IIRC in another
>>>>>>>>>> thread we have agreed that we always follow latest version of RI, so it
>>>>>>>>>> will be much easier to change the different behavior tests to API tests
>>>>>>>>>> if we use script to separate them, and vice versa.
>>>>>>>>>>
>>>>>>>>>> 5. [bootclasspath],
>>>>>>>>>> which covers 4 categories you mentioned, the test cases are in same
>>>>>>>>>> package with implementions
>>>>>>>>>> e.g.
>>>>>>>>>> java.util is test package for java.util
>>>>>>>>>> and
>>>>>>>>>> o.a.h.luni.util is test packag for o.a.h.luni.util
>>>>>>>>>>
>>>>>>>>>> As a summary, we'll have four kind of test packages:
>>>>>>>>>>
>>>>>>>>>> a. o.a.h.<module>.tests.api.java.*** for case 1,4 above
>>>>>>>>>> b. o.a.h.<module>.tests.*** for case 2 above
>>>>>>>>>> c. java.***, for case 5 above
>>>>>>>>>> d. o.a.h.***, for case 5 above
>>>>>>>>>>
>>>>>>>>>> comments?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Mikhail
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Mikhail Loenko wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Geir,
>>>>>>>>>>>>>
>>>>>>>>>>>>> The problem is that no one yet suggested a consistent solution
>>>>>>>>>>>>> that would fit for all the tests and would not get into packages like
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.harmony.security.tests.org.apache.harmony.security.util
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Everybody seems to agree that SOME implementation specific tests are
>>>>>>>>>>>>> in the same package as implementation.
>>>>>>>>>>>>>
>>>>>>>>>>>>> But if we have ALL the tests in the same package as implementation,
>>>>>>>>>>>>> then we can't run e.g. tests against java.beans.Beans from classpath, so
>>>>>>>>>>>>> we can't run regression test for HARMONY-358
>>>>>>>>>>>>>
>>>>>>>>>>>>> The question is where we put 'classpath' test. If we put them into
>>>>>>>>>>>>> o.a.h.module.tests.[package under test] we result in that ugly package
>>>>>>>>>>>>> names like
>>>>>>>>>>>>> o.a.h.security.tests.o.a.h.security.util
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> Mikhail,
>>>>>>>>>>>>
>>>>>>>>>>>> Why my prior proposal that o.a.h.security.tests.impl.util is unacceptable?
>>>>>>>>>>>>
>>>>>>>>>>>> And I have no objection if "impl" considered redundant, too.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Mikhail
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2006/4/27, Geir Magnusson Jr <ge...@pobox.com>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Geir Magnusson Jr wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> George Harley wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Of course, the text module has only "implementation-independent tests
>>>>>>>>>>>>>>>> that designed to be run from classpath". For modules that have got
>>>>>>>>>>>>>>>> implementation-specific tests then I suppose we could use something
>>>>>>>>>>>>>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
>>>>>>>>>>>>>>>> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
>>>>>>>>>>>>>>>> I've got no preference.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -1
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Put them in the same package as the class being tested, please, if there
>>>>>>>>>>>>>>> is no problem being run that way.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sorry to be so snippy. Been a long day.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Why would we want this convoluted package name? Why wouldn't we want to
>>>>>>>>>>>>>> let them be in the same package as the class being tested since it's
>>>>>>>>>>>>>> implementation testing?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> geir
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Paulex Yang
>>>>>>>>>>>> China Software Development Lab
>>>>>>>>>>>> IBM
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Paulex Yang
>>>>>>>>>> China Software Development Lab
>>>>>>>>>> IBM
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Paulex Yang
>>>>>>>> China Software Development Lab
>>>>>>>> IBM
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Paulex Yang
>>>>>> China Software Development Lab
>>>>>> IBM
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> Paulex Yang
>>>> China Software Development Lab
>>>> IBM
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>>>
>>>
>> --
>> Paulex Yang
>> China Software Development Lab
>> IBM
>>
>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
>
--
Paulex Yang
China Software Development Lab
IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Mikhail Loenko <ml...@gmail.com>.
Hi Paulex
2006/4/28, Paulex Yang <pa...@gmail.com>:
> Mikhail Loenko wrote:
> > Then how to run only independent tests?
> >
> > Such a run would make sense to validate the tests or
> > if we switch to different RI version
> >
> It should be easy for ant script to identify independent tests by
> package name convention, e.g, we have a keyword "API" for the
> independent tests (1), and 2 can be easily excluded.
>
> For the intended different behavior (4), test can be added to ant's
> exclude list (I don't expect there are too many such cases)
Did you count such tests? As I remember we have a lof of them in security.
If these tests are in the same package and in the same directory
then we will have something like
org.apache.harmony.security.tests.api.java.security.CodeSignerTest1 - impl
specific test methods
org.apache.harmony.security.tests.api.java.security.CodeSignerTest2 - impl
independent test methods
How do you like it?
Thanks,
Mikhail
>
> > Thanks,
> > Mikhail
> >
> > 2006/4/28, Paulex Yang <pa...@gmail.com>:
> >
> >> Mikhail Loenko wrote:
> >>
> >>> And how would you split 1,2,4,5 between these two folders?
> >>>
> >>>
> >> I thought it is obvious.
> >> test 1,2,4 goes to src/test
> >> test 5 goes to src/impltest (internl/injected/boot anything is fine)
> >> and all implementation goes to src/main
> >>
> >> and they can be compiled to corresponding bin dir, i.e.
> >> bin/test
> >> bin/impltest
> >> bin/main
> >>
> >> bin/main and bin/impltest goes to bootclasspath
> >> bin/test goes to classpath
> >>
> >> Pls. correct me if I made mistake or missed sth.
> >>
> >>> Thanks,
> >>> Mikhail
> >>>
> >>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
> >>>
> >>>
> >>>> Mikhail Loenko wrote:
> >>>>
> >>>>
> >>>>> If it's separated, I'm OK with it
> >>>>>
> >>>>> Can you suggest dir names?
> >>>>>
> >>>>>
> >>>>>
> >>>> Great!
> >>>>
> >>>> Either boot or injected(as Mark said) sounds good keyword to me. so it
> >>>> can be one of below:
> >>>> 1. src/test.injected/
> >>>> 2. src/test/injected
> >>>> 3. src/test.boot/
> >>>> 4. src/test/boot
> >>>>
> >>>> Mmm...seems injected is more self-descriptive, I prefer option 1. So
> >>>> the layout would be:
> >>>> src/main
> >>>> src/test
> >>>> src/test.injected
> >>>>
> >>>> ideas?
> >>>>
> >>>>
> >>>>> Thanks,
> >>>>> Mikhail
> >>>>>
> >>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
> >>>>>
> >>>>>
> >>>>>
> >>>>>> Mikhail Loenko wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> The problem is the tests will be mixed like that:
> >>>>>>>
> >>>>>>> org.apache.harmony.security.asn1 -- tests in bootclasspath
> >>>>>>> org.apache.harmony.security.cert -- tests in bootclasspath
> >>>>>>> org.apache.harmony.security.tests -- tests in classpath
> >>>>>>> org.apache.harmony.security.x509 -- tests in bootclasspath
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> Good catch! :)
> >>>>>>
> >>>>>> It's not so hard for script to separate, if the package name doesn't
> >>>>>> contains "test", it belongs to bootclasspath
> >>>>>>
> >>>>>> And if you concern about human reading, we can separate src directory
> >>>>>> for them, i.e.
> >>>>>> src/main/java
> >>>>>> src/test/java
> >>>>>> src/boottest/java or src/test/boot/java
> >>>>>>
> >>>>>> comments?
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Thanks,
> >>>>>>> Mikhail
> >>>>>>>
> >>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> Mikhail Loenko wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Paulex,
> >>>>>>>>>
> >>>>>>>>> we have at least 8 categories of tests:
> >>>>>>>>> running from classpath or bootclasspath
> >>>>>>>>> implementation specific or independent
> >>>>>>>>> testing org.apache.harmony.* or java.*
> >>>>>>>>> Could you please list how all the tests will be named
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> Mikhail,
> >>>>>>>>
> >>>>>>>> OK, let me try. I'm not sure if all 8 categories are necessary,
> >>>>>>>>
> >>>>>>>> 1. [classpath], [independent], [java.*]
> >>>>>>>> o.a.h.<module name>.tests.api.<package name>, e.g.,
> >>>>>>>> o.a.h.luni.tests.api.java.util is test package for java.util
> >>>>>>>> I consider the module name is optional and can be removed, so it can be
> >>>>>>>> o.a.h.tests.api.java.util for java.util
> >>>>>>>>
> >>>>>>>> 2. [classpath], [dependent], [o.a.h.*]
> >>>>>>>> o.a.h.<module name>.tests.<package name without o.a.h...prefix>,
> >>>>>>>> e.g., o.a.h.luni.tests.util is test package for o.a.h.luni.util
> >>>>>>>> Again, the module name is optional, but if module name is not
> >>>>>>>> prefix, it should be part of package name, so it is:
> >>>>>>>> o.a.h.tests.luni.util for o.a.h.luni.util
> >>>>>>>>
> >>>>>>>> 3. [classpath], [independent], [o.a.h.*]
> >>>>>>>> I don't expect these classes/tests exist.
> >>>>>>>>
> >>>>>>>> 4. [classpath], [dependent], [java.*]
> >>>>>>>> if my understanding is correct, this category means the tests for
> >>>>>>>> intended different behavior between Harmony and RI, I consider these
> >>>>>>>> tests should be put in same package with API tests, and separate them by
> >>>>>>>> running script instead of different package, because RI may fix bug in
> >>>>>>>> the revision later(e.g, Sun JDK 1.5.1 maybe?), and IIRC in another
> >>>>>>>> thread we have agreed that we always follow latest version of RI, so it
> >>>>>>>> will be much easier to change the different behavior tests to API tests
> >>>>>>>> if we use script to separate them, and vice versa.
> >>>>>>>>
> >>>>>>>> 5. [bootclasspath],
> >>>>>>>> which covers 4 categories you mentioned, the test cases are in same
> >>>>>>>> package with implementions
> >>>>>>>> e.g.
> >>>>>>>> java.util is test package for java.util
> >>>>>>>> and
> >>>>>>>> o.a.h.luni.util is test packag for o.a.h.luni.util
> >>>>>>>>
> >>>>>>>> As a summary, we'll have four kind of test packages:
> >>>>>>>>
> >>>>>>>> a. o.a.h.<module>.tests.api.java.*** for case 1,4 above
> >>>>>>>> b. o.a.h.<module>.tests.*** for case 2 above
> >>>>>>>> c. java.***, for case 5 above
> >>>>>>>> d. o.a.h.***, for case 5 above
> >>>>>>>>
> >>>>>>>> comments?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Mikhail
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> Mikhail Loenko wrote:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Geir,
> >>>>>>>>>>>
> >>>>>>>>>>> The problem is that no one yet suggested a consistent solution
> >>>>>>>>>>> that would fit for all the tests and would not get into packages like
> >>>>>>>>>>>
> >>>>>>>>>>> org.apache.harmony.security.tests.org.apache.harmony.security.util
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Everybody seems to agree that SOME implementation specific tests are
> >>>>>>>>>>> in the same package as implementation.
> >>>>>>>>>>>
> >>>>>>>>>>> But if we have ALL the tests in the same package as implementation,
> >>>>>>>>>>> then we can't run e.g. tests against java.beans.Beans from classpath, so
> >>>>>>>>>>> we can't run regression test for HARMONY-358
> >>>>>>>>>>>
> >>>>>>>>>>> The question is where we put 'classpath' test. If we put them into
> >>>>>>>>>>> o.a.h.module.tests.[package under test] we result in that ugly package
> >>>>>>>>>>> names like
> >>>>>>>>>>> o.a.h.security.tests.o.a.h.security.util
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> Mikhail,
> >>>>>>>>>>
> >>>>>>>>>> Why my prior proposal that o.a.h.security.tests.impl.util is unacceptable?
> >>>>>>>>>>
> >>>>>>>>>> And I have no objection if "impl" considered redundant, too.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> Mikhail
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> 2006/4/27, Geir Magnusson Jr <ge...@pobox.com>:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> Geir Magnusson Jr wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> George Harley wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>> >>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>> Of course, the text module has only "implementation-independent tests
> >>>>>>>>>>>>>> that designed to be run from classpath". For modules that have got
> >>>>>>>>>>>>>> implementation-specific tests then I suppose we could use something
> >>>>>>>>>>>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
> >>>>>>>>>>>>>> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
> >>>>>>>>>>>>>> I've got no preference.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> -1
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Put them in the same package as the class being tested, please, if there
> >>>>>>>>>>>>> is no problem being run that way.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>> Sorry to be so snippy. Been a long day.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Why would we want this convoluted package name? Why wouldn't we want to
> >>>>>>>>>>>> let them be in the same package as the class being tested since it's
> >>>>>>>>>>>> implementation testing?
> >>>>>>>>>>>>
> >>>>>>>>>>>> geir
> >>>>>>>>>>>>
> >>>>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Paulex Yang
> >>>>>>>>>> China Software Development Lab
> >>>>>>>>>> IBM
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> --
> >>>>>>>> Paulex Yang
> >>>>>>>> China Software Development Lab
> >>>>>>>> IBM
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ---------------------------------------------------------------------
> >>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>> ---------------------------------------------------------------------
> >>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> --
> >>>>>> Paulex Yang
> >>>>>> China Software Development Lab
> >>>>>> IBM
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ---------------------------------------------------------------------
> >>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>> --
> >>>> Paulex Yang
> >>>> China Software Development Lab
> >>>> IBM
> >>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>
> >>>>
> >>>>
> >>>>
> >>> ---------------------------------------------------------------------
> >>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>
> >>>
> >>>
> >>>
> >> --
> >> Paulex Yang
> >> China Software Development Lab
> >> IBM
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
> >
>
>
> --
> Paulex Yang
> China Software Development Lab
> IBM
>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Paulex Yang <pa...@gmail.com>.
Mikhail Loenko wrote:
> Then how to run only independent tests?
>
> Such a run would make sense to validate the tests or
> if we switch to different RI version
>
It should be easy for ant script to identify independent tests by
package name convention, e.g, we have a keyword "API" for the
independent tests (1), and 2 can be easily excluded.
For the intended different behavior (4), test can be added to ant's
exclude list (I don't expect there are too many such cases)
> Thanks,
> Mikhail
>
> 2006/4/28, Paulex Yang <pa...@gmail.com>:
>
>> Mikhail Loenko wrote:
>>
>>> And how would you split 1,2,4,5 between these two folders?
>>>
>>>
>> I thought it is obvious.
>> test 1,2,4 goes to src/test
>> test 5 goes to src/impltest (internl/injected/boot anything is fine)
>> and all implementation goes to src/main
>>
>> and they can be compiled to corresponding bin dir, i.e.
>> bin/test
>> bin/impltest
>> bin/main
>>
>> bin/main and bin/impltest goes to bootclasspath
>> bin/test goes to classpath
>>
>> Pls. correct me if I made mistake or missed sth.
>>
>>> Thanks,
>>> Mikhail
>>>
>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>
>>>
>>>> Mikhail Loenko wrote:
>>>>
>>>>
>>>>> If it's separated, I'm OK with it
>>>>>
>>>>> Can you suggest dir names?
>>>>>
>>>>>
>>>>>
>>>> Great!
>>>>
>>>> Either boot or injected(as Mark said) sounds good keyword to me. so it
>>>> can be one of below:
>>>> 1. src/test.injected/
>>>> 2. src/test/injected
>>>> 3. src/test.boot/
>>>> 4. src/test/boot
>>>>
>>>> Mmm...seems injected is more self-descriptive, I prefer option 1. So
>>>> the layout would be:
>>>> src/main
>>>> src/test
>>>> src/test.injected
>>>>
>>>> ideas?
>>>>
>>>>
>>>>> Thanks,
>>>>> Mikhail
>>>>>
>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>>>
>>>>>
>>>>>
>>>>>> Mikhail Loenko wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> The problem is the tests will be mixed like that:
>>>>>>>
>>>>>>> org.apache.harmony.security.asn1 -- tests in bootclasspath
>>>>>>> org.apache.harmony.security.cert -- tests in bootclasspath
>>>>>>> org.apache.harmony.security.tests -- tests in classpath
>>>>>>> org.apache.harmony.security.x509 -- tests in bootclasspath
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Good catch! :)
>>>>>>
>>>>>> It's not so hard for script to separate, if the package name doesn't
>>>>>> contains "test", it belongs to bootclasspath
>>>>>>
>>>>>> And if you concern about human reading, we can separate src directory
>>>>>> for them, i.e.
>>>>>> src/main/java
>>>>>> src/test/java
>>>>>> src/boottest/java or src/test/boot/java
>>>>>>
>>>>>> comments?
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Thanks,
>>>>>>> Mikhail
>>>>>>>
>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Mikhail Loenko wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Paulex,
>>>>>>>>>
>>>>>>>>> we have at least 8 categories of tests:
>>>>>>>>> running from classpath or bootclasspath
>>>>>>>>> implementation specific or independent
>>>>>>>>> testing org.apache.harmony.* or java.*
>>>>>>>>> Could you please list how all the tests will be named
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Mikhail,
>>>>>>>>
>>>>>>>> OK, let me try. I'm not sure if all 8 categories are necessary,
>>>>>>>>
>>>>>>>> 1. [classpath], [independent], [java.*]
>>>>>>>> o.a.h.<module name>.tests.api.<package name>, e.g.,
>>>>>>>> o.a.h.luni.tests.api.java.util is test package for java.util
>>>>>>>> I consider the module name is optional and can be removed, so it can be
>>>>>>>> o.a.h.tests.api.java.util for java.util
>>>>>>>>
>>>>>>>> 2. [classpath], [dependent], [o.a.h.*]
>>>>>>>> o.a.h.<module name>.tests.<package name without o.a.h...prefix>,
>>>>>>>> e.g., o.a.h.luni.tests.util is test package for o.a.h.luni.util
>>>>>>>> Again, the module name is optional, but if module name is not
>>>>>>>> prefix, it should be part of package name, so it is:
>>>>>>>> o.a.h.tests.luni.util for o.a.h.luni.util
>>>>>>>>
>>>>>>>> 3. [classpath], [independent], [o.a.h.*]
>>>>>>>> I don't expect these classes/tests exist.
>>>>>>>>
>>>>>>>> 4. [classpath], [dependent], [java.*]
>>>>>>>> if my understanding is correct, this category means the tests for
>>>>>>>> intended different behavior between Harmony and RI, I consider these
>>>>>>>> tests should be put in same package with API tests, and separate them by
>>>>>>>> running script instead of different package, because RI may fix bug in
>>>>>>>> the revision later(e.g, Sun JDK 1.5.1 maybe?), and IIRC in another
>>>>>>>> thread we have agreed that we always follow latest version of RI, so it
>>>>>>>> will be much easier to change the different behavior tests to API tests
>>>>>>>> if we use script to separate them, and vice versa.
>>>>>>>>
>>>>>>>> 5. [bootclasspath],
>>>>>>>> which covers 4 categories you mentioned, the test cases are in same
>>>>>>>> package with implementions
>>>>>>>> e.g.
>>>>>>>> java.util is test package for java.util
>>>>>>>> and
>>>>>>>> o.a.h.luni.util is test packag for o.a.h.luni.util
>>>>>>>>
>>>>>>>> As a summary, we'll have four kind of test packages:
>>>>>>>>
>>>>>>>> a. o.a.h.<module>.tests.api.java.*** for case 1,4 above
>>>>>>>> b. o.a.h.<module>.tests.*** for case 2 above
>>>>>>>> c. java.***, for case 5 above
>>>>>>>> d. o.a.h.***, for case 5 above
>>>>>>>>
>>>>>>>> comments?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Mikhail
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Mikhail Loenko wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Geir,
>>>>>>>>>>>
>>>>>>>>>>> The problem is that no one yet suggested a consistent solution
>>>>>>>>>>> that would fit for all the tests and would not get into packages like
>>>>>>>>>>>
>>>>>>>>>>> org.apache.harmony.security.tests.org.apache.harmony.security.util
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Everybody seems to agree that SOME implementation specific tests are
>>>>>>>>>>> in the same package as implementation.
>>>>>>>>>>>
>>>>>>>>>>> But if we have ALL the tests in the same package as implementation,
>>>>>>>>>>> then we can't run e.g. tests against java.beans.Beans from classpath, so
>>>>>>>>>>> we can't run regression test for HARMONY-358
>>>>>>>>>>>
>>>>>>>>>>> The question is where we put 'classpath' test. If we put them into
>>>>>>>>>>> o.a.h.module.tests.[package under test] we result in that ugly package
>>>>>>>>>>> names like
>>>>>>>>>>> o.a.h.security.tests.o.a.h.security.util
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Mikhail,
>>>>>>>>>>
>>>>>>>>>> Why my prior proposal that o.a.h.security.tests.impl.util is unacceptable?
>>>>>>>>>>
>>>>>>>>>> And I have no objection if "impl" considered redundant, too.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Mikhail
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2006/4/27, Geir Magnusson Jr <ge...@pobox.com>:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Geir Magnusson Jr wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> George Harley wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> >>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>> Of course, the text module has only "implementation-independent tests
>>>>>>>>>>>>>> that designed to be run from classpath". For modules that have got
>>>>>>>>>>>>>> implementation-specific tests then I suppose we could use something
>>>>>>>>>>>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
>>>>>>>>>>>>>> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
>>>>>>>>>>>>>> I've got no preference.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> -1
>>>>>>>>>>>>>
>>>>>>>>>>>>> Put them in the same package as the class being tested, please, if there
>>>>>>>>>>>>> is no problem being run that way.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> Sorry to be so snippy. Been a long day.
>>>>>>>>>>>>
>>>>>>>>>>>> Why would we want this convoluted package name? Why wouldn't we want to
>>>>>>>>>>>> let them be in the same package as the class being tested since it's
>>>>>>>>>>>> implementation testing?
>>>>>>>>>>>>
>>>>>>>>>>>> geir
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Paulex Yang
>>>>>>>>>> China Software Development Lab
>>>>>>>>>> IBM
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Paulex Yang
>>>>>>>> China Software Development Lab
>>>>>>>> IBM
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Paulex Yang
>>>>>> China Software Development Lab
>>>>>> IBM
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> Paulex Yang
>>>> China Software Development Lab
>>>> IBM
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>>>
>>>
>> --
>> Paulex Yang
>> China Software Development Lab
>> IBM
>>
>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
>
--
Paulex Yang
China Software Development Lab
IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Mikhail Loenko <ml...@gmail.com>.
Then how to run only independent tests?
Such a run would make sense to validate the tests or
if we switch to different RI version
Thanks,
Mikhail
2006/4/28, Paulex Yang <pa...@gmail.com>:
> Mikhail Loenko wrote:
> > And how would you split 1,2,4,5 between these two folders?
> >
> I thought it is obvious.
> test 1,2,4 goes to src/test
> test 5 goes to src/impltest (internl/injected/boot anything is fine)
> and all implementation goes to src/main
>
> and they can be compiled to corresponding bin dir, i.e.
> bin/test
> bin/impltest
> bin/main
>
> bin/main and bin/impltest goes to bootclasspath
> bin/test goes to classpath
>
> Pls. correct me if I made mistake or missed sth.
> > Thanks,
> > Mikhail
> >
> > 2006/4/27, Paulex Yang <pa...@gmail.com>:
> >
> >> Mikhail Loenko wrote:
> >>
> >>> If it's separated, I'm OK with it
> >>>
> >>> Can you suggest dir names?
> >>>
> >>>
> >> Great!
> >>
> >> Either boot or injected(as Mark said) sounds good keyword to me. so it
> >> can be one of below:
> >> 1. src/test.injected/
> >> 2. src/test/injected
> >> 3. src/test.boot/
> >> 4. src/test/boot
> >>
> >> Mmm...seems injected is more self-descriptive, I prefer option 1. So
> >> the layout would be:
> >> src/main
> >> src/test
> >> src/test.injected
> >>
> >> ideas?
> >>
> >>> Thanks,
> >>> Mikhail
> >>>
> >>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
> >>>
> >>>
> >>>> Mikhail Loenko wrote:
> >>>>
> >>>>
> >>>>> The problem is the tests will be mixed like that:
> >>>>>
> >>>>> org.apache.harmony.security.asn1 -- tests in bootclasspath
> >>>>> org.apache.harmony.security.cert -- tests in bootclasspath
> >>>>> org.apache.harmony.security.tests -- tests in classpath
> >>>>> org.apache.harmony.security.x509 -- tests in bootclasspath
> >>>>>
> >>>>>
> >>>>>
> >>>> Good catch! :)
> >>>>
> >>>> It's not so hard for script to separate, if the package name doesn't
> >>>> contains "test", it belongs to bootclasspath
> >>>>
> >>>> And if you concern about human reading, we can separate src directory
> >>>> for them, i.e.
> >>>> src/main/java
> >>>> src/test/java
> >>>> src/boottest/java or src/test/boot/java
> >>>>
> >>>> comments?
> >>>>
> >>>>
> >>>>> Thanks,
> >>>>> Mikhail
> >>>>>
> >>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
> >>>>>
> >>>>>
> >>>>>
> >>>>>> Mikhail Loenko wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Paulex,
> >>>>>>>
> >>>>>>> we have at least 8 categories of tests:
> >>>>>>> running from classpath or bootclasspath
> >>>>>>> implementation specific or independent
> >>>>>>> testing org.apache.harmony.* or java.*
> >>>>>>> Could you please list how all the tests will be named
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> Mikhail,
> >>>>>>
> >>>>>> OK, let me try. I'm not sure if all 8 categories are necessary,
> >>>>>>
> >>>>>> 1. [classpath], [independent], [java.*]
> >>>>>> o.a.h.<module name>.tests.api.<package name>, e.g.,
> >>>>>> o.a.h.luni.tests.api.java.util is test package for java.util
> >>>>>> I consider the module name is optional and can be removed, so it can be
> >>>>>> o.a.h.tests.api.java.util for java.util
> >>>>>>
> >>>>>> 2. [classpath], [dependent], [o.a.h.*]
> >>>>>> o.a.h.<module name>.tests.<package name without o.a.h...prefix>,
> >>>>>> e.g., o.a.h.luni.tests.util is test package for o.a.h.luni.util
> >>>>>> Again, the module name is optional, but if module name is not
> >>>>>> prefix, it should be part of package name, so it is:
> >>>>>> o.a.h.tests.luni.util for o.a.h.luni.util
> >>>>>>
> >>>>>> 3. [classpath], [independent], [o.a.h.*]
> >>>>>> I don't expect these classes/tests exist.
> >>>>>>
> >>>>>> 4. [classpath], [dependent], [java.*]
> >>>>>> if my understanding is correct, this category means the tests for
> >>>>>> intended different behavior between Harmony and RI, I consider these
> >>>>>> tests should be put in same package with API tests, and separate them by
> >>>>>> running script instead of different package, because RI may fix bug in
> >>>>>> the revision later(e.g, Sun JDK 1.5.1 maybe?), and IIRC in another
> >>>>>> thread we have agreed that we always follow latest version of RI, so it
> >>>>>> will be much easier to change the different behavior tests to API tests
> >>>>>> if we use script to separate them, and vice versa.
> >>>>>>
> >>>>>> 5. [bootclasspath],
> >>>>>> which covers 4 categories you mentioned, the test cases are in same
> >>>>>> package with implementions
> >>>>>> e.g.
> >>>>>> java.util is test package for java.util
> >>>>>> and
> >>>>>> o.a.h.luni.util is test packag for o.a.h.luni.util
> >>>>>>
> >>>>>> As a summary, we'll have four kind of test packages:
> >>>>>>
> >>>>>> a. o.a.h.<module>.tests.api.java.*** for case 1,4 above
> >>>>>> b. o.a.h.<module>.tests.*** for case 2 above
> >>>>>> c. java.***, for case 5 above
> >>>>>> d. o.a.h.***, for case 5 above
> >>>>>>
> >>>>>> comments?
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Thanks,
> >>>>>>> Mikhail
> >>>>>>>
> >>>>>>>
> >>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> Mikhail Loenko wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Geir,
> >>>>>>>>>
> >>>>>>>>> The problem is that no one yet suggested a consistent solution
> >>>>>>>>> that would fit for all the tests and would not get into packages like
> >>>>>>>>>
> >>>>>>>>> org.apache.harmony.security.tests.org.apache.harmony.security.util
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Everybody seems to agree that SOME implementation specific tests are
> >>>>>>>>> in the same package as implementation.
> >>>>>>>>>
> >>>>>>>>> But if we have ALL the tests in the same package as implementation,
> >>>>>>>>> then we can't run e.g. tests against java.beans.Beans from classpath, so
> >>>>>>>>> we can't run regression test for HARMONY-358
> >>>>>>>>>
> >>>>>>>>> The question is where we put 'classpath' test. If we put them into
> >>>>>>>>> o.a.h.module.tests.[package under test] we result in that ugly package
> >>>>>>>>> names like
> >>>>>>>>> o.a.h.security.tests.o.a.h.security.util
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> Mikhail,
> >>>>>>>>
> >>>>>>>> Why my prior proposal that o.a.h.security.tests.impl.util is unacceptable?
> >>>>>>>>
> >>>>>>>> And I have no objection if "impl" considered redundant, too.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Mikhail
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 2006/4/27, Geir Magnusson Jr <ge...@pobox.com>:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> Geir Magnusson Jr wrote:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> George Harley wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> >>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>> Of course, the text module has only "implementation-independent tests
> >>>>>>>>>>>> that designed to be run from classpath". For modules that have got
> >>>>>>>>>>>> implementation-specific tests then I suppose we could use something
> >>>>>>>>>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
> >>>>>>>>>>>> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
> >>>>>>>>>>>> I've got no preference.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>> -1
> >>>>>>>>>>>
> >>>>>>>>>>> Put them in the same package as the class being tested, please, if there
> >>>>>>>>>>> is no problem being run that way.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> Sorry to be so snippy. Been a long day.
> >>>>>>>>>>
> >>>>>>>>>> Why would we want this convoluted package name? Why wouldn't we want to
> >>>>>>>>>> let them be in the same package as the class being tested since it's
> >>>>>>>>>> implementation testing?
> >>>>>>>>>>
> >>>>>>>>>> geir
> >>>>>>>>>>
> >>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> --
> >>>>>>>> Paulex Yang
> >>>>>>>> China Software Development Lab
> >>>>>>>> IBM
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ---------------------------------------------------------------------
> >>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>> ---------------------------------------------------------------------
> >>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> --
> >>>>>> Paulex Yang
> >>>>>> China Software Development Lab
> >>>>>> IBM
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ---------------------------------------------------------------------
> >>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>> --
> >>>> Paulex Yang
> >>>> China Software Development Lab
> >>>> IBM
> >>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>
> >>>>
> >>>>
> >>>>
> >>> ---------------------------------------------------------------------
> >>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>
> >>>
> >>>
> >>>
> >> --
> >> Paulex Yang
> >> China Software Development Lab
> >> IBM
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
> >
>
>
> --
> Paulex Yang
> China Software Development Lab
> IBM
>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Paulex Yang <pa...@gmail.com>.
Mikhail Loenko wrote:
> And how would you split 1,2,4,5 between these two folders?
>
I thought it is obvious.
test 1,2,4 goes to src/test
test 5 goes to src/impltest (internl/injected/boot anything is fine)
and all implementation goes to src/main
and they can be compiled to corresponding bin dir, i.e.
bin/test
bin/impltest
bin/main
bin/main and bin/impltest goes to bootclasspath
bin/test goes to classpath
Pls. correct me if I made mistake or missed sth.
> Thanks,
> Mikhail
>
> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>
>> Mikhail Loenko wrote:
>>
>>> If it's separated, I'm OK with it
>>>
>>> Can you suggest dir names?
>>>
>>>
>> Great!
>>
>> Either boot or injected(as Mark said) sounds good keyword to me. so it
>> can be one of below:
>> 1. src/test.injected/
>> 2. src/test/injected
>> 3. src/test.boot/
>> 4. src/test/boot
>>
>> Mmm...seems injected is more self-descriptive, I prefer option 1. So
>> the layout would be:
>> src/main
>> src/test
>> src/test.injected
>>
>> ideas?
>>
>>> Thanks,
>>> Mikhail
>>>
>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>
>>>
>>>> Mikhail Loenko wrote:
>>>>
>>>>
>>>>> The problem is the tests will be mixed like that:
>>>>>
>>>>> org.apache.harmony.security.asn1 -- tests in bootclasspath
>>>>> org.apache.harmony.security.cert -- tests in bootclasspath
>>>>> org.apache.harmony.security.tests -- tests in classpath
>>>>> org.apache.harmony.security.x509 -- tests in bootclasspath
>>>>>
>>>>>
>>>>>
>>>> Good catch! :)
>>>>
>>>> It's not so hard for script to separate, if the package name doesn't
>>>> contains "test", it belongs to bootclasspath
>>>>
>>>> And if you concern about human reading, we can separate src directory
>>>> for them, i.e.
>>>> src/main/java
>>>> src/test/java
>>>> src/boottest/java or src/test/boot/java
>>>>
>>>> comments?
>>>>
>>>>
>>>>> Thanks,
>>>>> Mikhail
>>>>>
>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>>>
>>>>>
>>>>>
>>>>>> Mikhail Loenko wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Paulex,
>>>>>>>
>>>>>>> we have at least 8 categories of tests:
>>>>>>> running from classpath or bootclasspath
>>>>>>> implementation specific or independent
>>>>>>> testing org.apache.harmony.* or java.*
>>>>>>> Could you please list how all the tests will be named
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Mikhail,
>>>>>>
>>>>>> OK, let me try. I'm not sure if all 8 categories are necessary,
>>>>>>
>>>>>> 1. [classpath], [independent], [java.*]
>>>>>> o.a.h.<module name>.tests.api.<package name>, e.g.,
>>>>>> o.a.h.luni.tests.api.java.util is test package for java.util
>>>>>> I consider the module name is optional and can be removed, so it can be
>>>>>> o.a.h.tests.api.java.util for java.util
>>>>>>
>>>>>> 2. [classpath], [dependent], [o.a.h.*]
>>>>>> o.a.h.<module name>.tests.<package name without o.a.h...prefix>,
>>>>>> e.g., o.a.h.luni.tests.util is test package for o.a.h.luni.util
>>>>>> Again, the module name is optional, but if module name is not
>>>>>> prefix, it should be part of package name, so it is:
>>>>>> o.a.h.tests.luni.util for o.a.h.luni.util
>>>>>>
>>>>>> 3. [classpath], [independent], [o.a.h.*]
>>>>>> I don't expect these classes/tests exist.
>>>>>>
>>>>>> 4. [classpath], [dependent], [java.*]
>>>>>> if my understanding is correct, this category means the tests for
>>>>>> intended different behavior between Harmony and RI, I consider these
>>>>>> tests should be put in same package with API tests, and separate them by
>>>>>> running script instead of different package, because RI may fix bug in
>>>>>> the revision later(e.g, Sun JDK 1.5.1 maybe?), and IIRC in another
>>>>>> thread we have agreed that we always follow latest version of RI, so it
>>>>>> will be much easier to change the different behavior tests to API tests
>>>>>> if we use script to separate them, and vice versa.
>>>>>>
>>>>>> 5. [bootclasspath],
>>>>>> which covers 4 categories you mentioned, the test cases are in same
>>>>>> package with implementions
>>>>>> e.g.
>>>>>> java.util is test package for java.util
>>>>>> and
>>>>>> o.a.h.luni.util is test packag for o.a.h.luni.util
>>>>>>
>>>>>> As a summary, we'll have four kind of test packages:
>>>>>>
>>>>>> a. o.a.h.<module>.tests.api.java.*** for case 1,4 above
>>>>>> b. o.a.h.<module>.tests.*** for case 2 above
>>>>>> c. java.***, for case 5 above
>>>>>> d. o.a.h.***, for case 5 above
>>>>>>
>>>>>> comments?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Thanks,
>>>>>>> Mikhail
>>>>>>>
>>>>>>>
>>>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Mikhail Loenko wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Geir,
>>>>>>>>>
>>>>>>>>> The problem is that no one yet suggested a consistent solution
>>>>>>>>> that would fit for all the tests and would not get into packages like
>>>>>>>>>
>>>>>>>>> org.apache.harmony.security.tests.org.apache.harmony.security.util
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Everybody seems to agree that SOME implementation specific tests are
>>>>>>>>> in the same package as implementation.
>>>>>>>>>
>>>>>>>>> But if we have ALL the tests in the same package as implementation,
>>>>>>>>> then we can't run e.g. tests against java.beans.Beans from classpath, so
>>>>>>>>> we can't run regression test for HARMONY-358
>>>>>>>>>
>>>>>>>>> The question is where we put 'classpath' test. If we put them into
>>>>>>>>> o.a.h.module.tests.[package under test] we result in that ugly package
>>>>>>>>> names like
>>>>>>>>> o.a.h.security.tests.o.a.h.security.util
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Mikhail,
>>>>>>>>
>>>>>>>> Why my prior proposal that o.a.h.security.tests.impl.util is unacceptable?
>>>>>>>>
>>>>>>>> And I have no objection if "impl" considered redundant, too.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Mikhail
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2006/4/27, Geir Magnusson Jr <ge...@pobox.com>:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Geir Magnusson Jr wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> George Harley wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> >>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> Of course, the text module has only "implementation-independent tests
>>>>>>>>>>>> that designed to be run from classpath". For modules that have got
>>>>>>>>>>>> implementation-specific tests then I suppose we could use something
>>>>>>>>>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
>>>>>>>>>>>> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
>>>>>>>>>>>> I've got no preference.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> -1
>>>>>>>>>>>
>>>>>>>>>>> Put them in the same package as the class being tested, please, if there
>>>>>>>>>>> is no problem being run that way.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Sorry to be so snippy. Been a long day.
>>>>>>>>>>
>>>>>>>>>> Why would we want this convoluted package name? Why wouldn't we want to
>>>>>>>>>> let them be in the same package as the class being tested since it's
>>>>>>>>>> implementation testing?
>>>>>>>>>>
>>>>>>>>>> geir
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Paulex Yang
>>>>>>>> China Software Development Lab
>>>>>>>> IBM
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Paulex Yang
>>>>>> China Software Development Lab
>>>>>> IBM
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> Paulex Yang
>>>> China Software Development Lab
>>>> IBM
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>>>
>>>
>> --
>> Paulex Yang
>> China Software Development Lab
>> IBM
>>
>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
>
--
Paulex Yang
China Software Development Lab
IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Mikhail Loenko <ml...@gmail.com>.
And how would you split 1,2,4,5 between these two folders?
Thanks,
Mikhail
2006/4/27, Paulex Yang <pa...@gmail.com>:
> Mikhail Loenko wrote:
> > If it's separated, I'm OK with it
> >
> > Can you suggest dir names?
> >
> Great!
>
> Either boot or injected(as Mark said) sounds good keyword to me. so it
> can be one of below:
> 1. src/test.injected/
> 2. src/test/injected
> 3. src/test.boot/
> 4. src/test/boot
>
> Mmm...seems injected is more self-descriptive, I prefer option 1. So
> the layout would be:
> src/main
> src/test
> src/test.injected
>
> ideas?
> > Thanks,
> > Mikhail
> >
> > 2006/4/27, Paulex Yang <pa...@gmail.com>:
> >
> >> Mikhail Loenko wrote:
> >>
> >>> The problem is the tests will be mixed like that:
> >>>
> >>> org.apache.harmony.security.asn1 -- tests in bootclasspath
> >>> org.apache.harmony.security.cert -- tests in bootclasspath
> >>> org.apache.harmony.security.tests -- tests in classpath
> >>> org.apache.harmony.security.x509 -- tests in bootclasspath
> >>>
> >>>
> >> Good catch! :)
> >>
> >> It's not so hard for script to separate, if the package name doesn't
> >> contains "test", it belongs to bootclasspath
> >>
> >> And if you concern about human reading, we can separate src directory
> >> for them, i.e.
> >> src/main/java
> >> src/test/java
> >> src/boottest/java or src/test/boot/java
> >>
> >> comments?
> >>
> >>> Thanks,
> >>> Mikhail
> >>>
> >>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
> >>>
> >>>
> >>>> Mikhail Loenko wrote:
> >>>>
> >>>>
> >>>>> Paulex,
> >>>>>
> >>>>> we have at least 8 categories of tests:
> >>>>> running from classpath or bootclasspath
> >>>>> implementation specific or independent
> >>>>> testing org.apache.harmony.* or java.*
> >>>>> Could you please list how all the tests will be named
> >>>>>
> >>>>>
> >>>>>
> >>>> Mikhail,
> >>>>
> >>>> OK, let me try. I'm not sure if all 8 categories are necessary,
> >>>>
> >>>> 1. [classpath], [independent], [java.*]
> >>>> o.a.h.<module name>.tests.api.<package name>, e.g.,
> >>>> o.a.h.luni.tests.api.java.util is test package for java.util
> >>>> I consider the module name is optional and can be removed, so it can be
> >>>> o.a.h.tests.api.java.util for java.util
> >>>>
> >>>> 2. [classpath], [dependent], [o.a.h.*]
> >>>> o.a.h.<module name>.tests.<package name without o.a.h...prefix>,
> >>>> e.g., o.a.h.luni.tests.util is test package for o.a.h.luni.util
> >>>> Again, the module name is optional, but if module name is not
> >>>> prefix, it should be part of package name, so it is:
> >>>> o.a.h.tests.luni.util for o.a.h.luni.util
> >>>>
> >>>> 3. [classpath], [independent], [o.a.h.*]
> >>>> I don't expect these classes/tests exist.
> >>>>
> >>>> 4. [classpath], [dependent], [java.*]
> >>>> if my understanding is correct, this category means the tests for
> >>>> intended different behavior between Harmony and RI, I consider these
> >>>> tests should be put in same package with API tests, and separate them by
> >>>> running script instead of different package, because RI may fix bug in
> >>>> the revision later(e.g, Sun JDK 1.5.1 maybe?), and IIRC in another
> >>>> thread we have agreed that we always follow latest version of RI, so it
> >>>> will be much easier to change the different behavior tests to API tests
> >>>> if we use script to separate them, and vice versa.
> >>>>
> >>>> 5. [bootclasspath],
> >>>> which covers 4 categories you mentioned, the test cases are in same
> >>>> package with implementions
> >>>> e.g.
> >>>> java.util is test package for java.util
> >>>> and
> >>>> o.a.h.luni.util is test packag for o.a.h.luni.util
> >>>>
> >>>> As a summary, we'll have four kind of test packages:
> >>>>
> >>>> a. o.a.h.<module>.tests.api.java.*** for case 1,4 above
> >>>> b. o.a.h.<module>.tests.*** for case 2 above
> >>>> c. java.***, for case 5 above
> >>>> d. o.a.h.***, for case 5 above
> >>>>
> >>>> comments?
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> Thanks,
> >>>>> Mikhail
> >>>>>
> >>>>>
> >>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
> >>>>>
> >>>>>
> >>>>>
> >>>>>> Mikhail Loenko wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Geir,
> >>>>>>>
> >>>>>>> The problem is that no one yet suggested a consistent solution
> >>>>>>> that would fit for all the tests and would not get into packages like
> >>>>>>>
> >>>>>>> org.apache.harmony.security.tests.org.apache.harmony.security.util
> >>>>>>>
> >>>>>>>
> >>>>>>> Everybody seems to agree that SOME implementation specific tests are
> >>>>>>> in the same package as implementation.
> >>>>>>>
> >>>>>>> But if we have ALL the tests in the same package as implementation,
> >>>>>>> then we can't run e.g. tests against java.beans.Beans from classpath, so
> >>>>>>> we can't run regression test for HARMONY-358
> >>>>>>>
> >>>>>>> The question is where we put 'classpath' test. If we put them into
> >>>>>>> o.a.h.module.tests.[package under test] we result in that ugly package
> >>>>>>> names like
> >>>>>>> o.a.h.security.tests.o.a.h.security.util
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> Mikhail,
> >>>>>>
> >>>>>> Why my prior proposal that o.a.h.security.tests.impl.util is unacceptable?
> >>>>>>
> >>>>>> And I have no objection if "impl" considered redundant, too.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Thanks,
> >>>>>>> Mikhail
> >>>>>>>
> >>>>>>>
> >>>>>>> 2006/4/27, Geir Magnusson Jr <ge...@pobox.com>:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> Geir Magnusson Jr wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> George Harley wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> >>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>> Of course, the text module has only "implementation-independent tests
> >>>>>>>>>> that designed to be run from classpath". For modules that have got
> >>>>>>>>>> implementation-specific tests then I suppose we could use something
> >>>>>>>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
> >>>>>>>>>> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
> >>>>>>>>>> I've got no preference.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> -1
> >>>>>>>>>
> >>>>>>>>> Put them in the same package as the class being tested, please, if there
> >>>>>>>>> is no problem being run that way.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> Sorry to be so snippy. Been a long day.
> >>>>>>>>
> >>>>>>>> Why would we want this convoluted package name? Why wouldn't we want to
> >>>>>>>> let them be in the same package as the class being tested since it's
> >>>>>>>> implementation testing?
> >>>>>>>>
> >>>>>>>> geir
> >>>>>>>>
> >>>>>>>> ---------------------------------------------------------------------
> >>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>> ---------------------------------------------------------------------
> >>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> --
> >>>>>> Paulex Yang
> >>>>>> China Software Development Lab
> >>>>>> IBM
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ---------------------------------------------------------------------
> >>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>> --
> >>>> Paulex Yang
> >>>> China Software Development Lab
> >>>> IBM
> >>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>
> >>>>
> >>>>
> >>>>
> >>> ---------------------------------------------------------------------
> >>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>
> >>>
> >>>
> >>>
> >> --
> >> Paulex Yang
> >> China Software Development Lab
> >> IBM
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
> >
>
>
> --
> Paulex Yang
> China Software Development Lab
> IBM
>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Paulex Yang <pa...@gmail.com>.
Mikhail Loenko wrote:
> If it's separated, I'm OK with it
>
> Can you suggest dir names?
>
Great!
Either boot or injected(as Mark said) sounds good keyword to me. so it
can be one of below:
1. src/test.injected/
2. src/test/injected
3. src/test.boot/
4. src/test/boot
Mmm...seems injected is more self-descriptive, I prefer option 1. So
the layout would be:
src/main
src/test
src/test.injected
ideas?
> Thanks,
> Mikhail
>
> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>
>> Mikhail Loenko wrote:
>>
>>> The problem is the tests will be mixed like that:
>>>
>>> org.apache.harmony.security.asn1 -- tests in bootclasspath
>>> org.apache.harmony.security.cert -- tests in bootclasspath
>>> org.apache.harmony.security.tests -- tests in classpath
>>> org.apache.harmony.security.x509 -- tests in bootclasspath
>>>
>>>
>> Good catch! :)
>>
>> It's not so hard for script to separate, if the package name doesn't
>> contains "test", it belongs to bootclasspath
>>
>> And if you concern about human reading, we can separate src directory
>> for them, i.e.
>> src/main/java
>> src/test/java
>> src/boottest/java or src/test/boot/java
>>
>> comments?
>>
>>> Thanks,
>>> Mikhail
>>>
>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>
>>>
>>>> Mikhail Loenko wrote:
>>>>
>>>>
>>>>> Paulex,
>>>>>
>>>>> we have at least 8 categories of tests:
>>>>> running from classpath or bootclasspath
>>>>> implementation specific or independent
>>>>> testing org.apache.harmony.* or java.*
>>>>> Could you please list how all the tests will be named
>>>>>
>>>>>
>>>>>
>>>> Mikhail,
>>>>
>>>> OK, let me try. I'm not sure if all 8 categories are necessary,
>>>>
>>>> 1. [classpath], [independent], [java.*]
>>>> o.a.h.<module name>.tests.api.<package name>, e.g.,
>>>> o.a.h.luni.tests.api.java.util is test package for java.util
>>>> I consider the module name is optional and can be removed, so it can be
>>>> o.a.h.tests.api.java.util for java.util
>>>>
>>>> 2. [classpath], [dependent], [o.a.h.*]
>>>> o.a.h.<module name>.tests.<package name without o.a.h...prefix>,
>>>> e.g., o.a.h.luni.tests.util is test package for o.a.h.luni.util
>>>> Again, the module name is optional, but if module name is not
>>>> prefix, it should be part of package name, so it is:
>>>> o.a.h.tests.luni.util for o.a.h.luni.util
>>>>
>>>> 3. [classpath], [independent], [o.a.h.*]
>>>> I don't expect these classes/tests exist.
>>>>
>>>> 4. [classpath], [dependent], [java.*]
>>>> if my understanding is correct, this category means the tests for
>>>> intended different behavior between Harmony and RI, I consider these
>>>> tests should be put in same package with API tests, and separate them by
>>>> running script instead of different package, because RI may fix bug in
>>>> the revision later(e.g, Sun JDK 1.5.1 maybe?), and IIRC in another
>>>> thread we have agreed that we always follow latest version of RI, so it
>>>> will be much easier to change the different behavior tests to API tests
>>>> if we use script to separate them, and vice versa.
>>>>
>>>> 5. [bootclasspath],
>>>> which covers 4 categories you mentioned, the test cases are in same
>>>> package with implementions
>>>> e.g.
>>>> java.util is test package for java.util
>>>> and
>>>> o.a.h.luni.util is test packag for o.a.h.luni.util
>>>>
>>>> As a summary, we'll have four kind of test packages:
>>>>
>>>> a. o.a.h.<module>.tests.api.java.*** for case 1,4 above
>>>> b. o.a.h.<module>.tests.*** for case 2 above
>>>> c. java.***, for case 5 above
>>>> d. o.a.h.***, for case 5 above
>>>>
>>>> comments?
>>>>
>>>>
>>>>
>>>>
>>>>> Thanks,
>>>>> Mikhail
>>>>>
>>>>>
>>>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>>>
>>>>>
>>>>>
>>>>>> Mikhail Loenko wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Geir,
>>>>>>>
>>>>>>> The problem is that no one yet suggested a consistent solution
>>>>>>> that would fit for all the tests and would not get into packages like
>>>>>>>
>>>>>>> org.apache.harmony.security.tests.org.apache.harmony.security.util
>>>>>>>
>>>>>>>
>>>>>>> Everybody seems to agree that SOME implementation specific tests are
>>>>>>> in the same package as implementation.
>>>>>>>
>>>>>>> But if we have ALL the tests in the same package as implementation,
>>>>>>> then we can't run e.g. tests against java.beans.Beans from classpath, so
>>>>>>> we can't run regression test for HARMONY-358
>>>>>>>
>>>>>>> The question is where we put 'classpath' test. If we put them into
>>>>>>> o.a.h.module.tests.[package under test] we result in that ugly package
>>>>>>> names like
>>>>>>> o.a.h.security.tests.o.a.h.security.util
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Mikhail,
>>>>>>
>>>>>> Why my prior proposal that o.a.h.security.tests.impl.util is unacceptable?
>>>>>>
>>>>>> And I have no objection if "impl" considered redundant, too.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Thanks,
>>>>>>> Mikhail
>>>>>>>
>>>>>>>
>>>>>>> 2006/4/27, Geir Magnusson Jr <ge...@pobox.com>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Geir Magnusson Jr wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> George Harley wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> >>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>> Of course, the text module has only "implementation-independent tests
>>>>>>>>>> that designed to be run from classpath". For modules that have got
>>>>>>>>>> implementation-specific tests then I suppose we could use something
>>>>>>>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
>>>>>>>>>> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
>>>>>>>>>> I've got no preference.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> -1
>>>>>>>>>
>>>>>>>>> Put them in the same package as the class being tested, please, if there
>>>>>>>>> is no problem being run that way.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Sorry to be so snippy. Been a long day.
>>>>>>>>
>>>>>>>> Why would we want this convoluted package name? Why wouldn't we want to
>>>>>>>> let them be in the same package as the class being tested since it's
>>>>>>>> implementation testing?
>>>>>>>>
>>>>>>>> geir
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Paulex Yang
>>>>>> China Software Development Lab
>>>>>> IBM
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> Paulex Yang
>>>> China Software Development Lab
>>>> IBM
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>>>
>>>
>> --
>> Paulex Yang
>> China Software Development Lab
>> IBM
>>
>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
>
--
Paulex Yang
China Software Development Lab
IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Mikhail Loenko <ml...@gmail.com>.
If it's separated, I'm OK with it
Can you suggest dir names?
Thanks,
Mikhail
2006/4/27, Paulex Yang <pa...@gmail.com>:
> Mikhail Loenko wrote:
> > The problem is the tests will be mixed like that:
> >
> > org.apache.harmony.security.asn1 -- tests in bootclasspath
> > org.apache.harmony.security.cert -- tests in bootclasspath
> > org.apache.harmony.security.tests -- tests in classpath
> > org.apache.harmony.security.x509 -- tests in bootclasspath
> >
> Good catch! :)
>
> It's not so hard for script to separate, if the package name doesn't
> contains "test", it belongs to bootclasspath
>
> And if you concern about human reading, we can separate src directory
> for them, i.e.
> src/main/java
> src/test/java
> src/boottest/java or src/test/boot/java
>
> comments?
> > Thanks,
> > Mikhail
> >
> > 2006/4/27, Paulex Yang <pa...@gmail.com>:
> >
> >> Mikhail Loenko wrote:
> >>
> >>> Paulex,
> >>>
> >>> we have at least 8 categories of tests:
> >>> running from classpath or bootclasspath
> >>> implementation specific or independent
> >>> testing org.apache.harmony.* or java.*
> >>> Could you please list how all the tests will be named
> >>>
> >>>
> >> Mikhail,
> >>
> >> OK, let me try. I'm not sure if all 8 categories are necessary,
> >>
> >> 1. [classpath], [independent], [java.*]
> >> o.a.h.<module name>.tests.api.<package name>, e.g.,
> >> o.a.h.luni.tests.api.java.util is test package for java.util
> >> I consider the module name is optional and can be removed, so it can be
> >> o.a.h.tests.api.java.util for java.util
> >>
> >> 2. [classpath], [dependent], [o.a.h.*]
> >> o.a.h.<module name>.tests.<package name without o.a.h...prefix>,
> >> e.g., o.a.h.luni.tests.util is test package for o.a.h.luni.util
> >> Again, the module name is optional, but if module name is not
> >> prefix, it should be part of package name, so it is:
> >> o.a.h.tests.luni.util for o.a.h.luni.util
> >>
> >> 3. [classpath], [independent], [o.a.h.*]
> >> I don't expect these classes/tests exist.
> >>
> >> 4. [classpath], [dependent], [java.*]
> >> if my understanding is correct, this category means the tests for
> >> intended different behavior between Harmony and RI, I consider these
> >> tests should be put in same package with API tests, and separate them by
> >> running script instead of different package, because RI may fix bug in
> >> the revision later(e.g, Sun JDK 1.5.1 maybe?), and IIRC in another
> >> thread we have agreed that we always follow latest version of RI, so it
> >> will be much easier to change the different behavior tests to API tests
> >> if we use script to separate them, and vice versa.
> >>
> >> 5. [bootclasspath],
> >> which covers 4 categories you mentioned, the test cases are in same
> >> package with implementions
> >> e.g.
> >> java.util is test package for java.util
> >> and
> >> o.a.h.luni.util is test packag for o.a.h.luni.util
> >>
> >> As a summary, we'll have four kind of test packages:
> >>
> >> a. o.a.h.<module>.tests.api.java.*** for case 1,4 above
> >> b. o.a.h.<module>.tests.*** for case 2 above
> >> c. java.***, for case 5 above
> >> d. o.a.h.***, for case 5 above
> >>
> >> comments?
> >>
> >>
> >>
> >>> Thanks,
> >>> Mikhail
> >>>
> >>>
> >>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
> >>>
> >>>
> >>>> Mikhail Loenko wrote:
> >>>>
> >>>>
> >>>>> Geir,
> >>>>>
> >>>>> The problem is that no one yet suggested a consistent solution
> >>>>> that would fit for all the tests and would not get into packages like
> >>>>>
> >>>>> org.apache.harmony.security.tests.org.apache.harmony.security.util
> >>>>>
> >>>>>
> >>>>> Everybody seems to agree that SOME implementation specific tests are
> >>>>> in the same package as implementation.
> >>>>>
> >>>>> But if we have ALL the tests in the same package as implementation,
> >>>>> then we can't run e.g. tests against java.beans.Beans from classpath, so
> >>>>> we can't run regression test for HARMONY-358
> >>>>>
> >>>>> The question is where we put 'classpath' test. If we put them into
> >>>>> o.a.h.module.tests.[package under test] we result in that ugly package
> >>>>> names like
> >>>>> o.a.h.security.tests.o.a.h.security.util
> >>>>>
> >>>>>
> >>>>>
> >>>> Mikhail,
> >>>>
> >>>> Why my prior proposal that o.a.h.security.tests.impl.util is unacceptable?
> >>>>
> >>>> And I have no objection if "impl" considered redundant, too.
> >>>>
> >>>>
> >>>>
> >>>>> Thanks,
> >>>>> Mikhail
> >>>>>
> >>>>>
> >>>>> 2006/4/27, Geir Magnusson Jr <ge...@pobox.com>:
> >>>>>
> >>>>>
> >>>>>
> >>>>>> Geir Magnusson Jr wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> George Harley wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> >>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>> Of course, the text module has only "implementation-independent tests
> >>>>>>>> that designed to be run from classpath". For modules that have got
> >>>>>>>> implementation-specific tests then I suppose we could use something
> >>>>>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
> >>>>>>>> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
> >>>>>>>> I've got no preference.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>> -1
> >>>>>>>
> >>>>>>> Put them in the same package as the class being tested, please, if there
> >>>>>>> is no problem being run that way.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> Sorry to be so snippy. Been a long day.
> >>>>>>
> >>>>>> Why would we want this convoluted package name? Why wouldn't we want to
> >>>>>> let them be in the same package as the class being tested since it's
> >>>>>> implementation testing?
> >>>>>>
> >>>>>> geir
> >>>>>>
> >>>>>> ---------------------------------------------------------------------
> >>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>> --
> >>>> Paulex Yang
> >>>> China Software Development Lab
> >>>> IBM
> >>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>
> >>>>
> >>>>
> >>>>
> >>> ---------------------------------------------------------------------
> >>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>
> >>>
> >>>
> >>>
> >> --
> >> Paulex Yang
> >> China Software Development Lab
> >> IBM
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
> >
>
>
> --
> Paulex Yang
> China Software Development Lab
> IBM
>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Paulex Yang <pa...@gmail.com>.
Mikhail Loenko wrote:
> The problem is the tests will be mixed like that:
>
> org.apache.harmony.security.asn1 -- tests in bootclasspath
> org.apache.harmony.security.cert -- tests in bootclasspath
> org.apache.harmony.security.tests -- tests in classpath
> org.apache.harmony.security.x509 -- tests in bootclasspath
>
Good catch! :)
It's not so hard for script to separate, if the package name doesn't
contains "test", it belongs to bootclasspath
And if you concern about human reading, we can separate src directory
for them, i.e.
src/main/java
src/test/java
src/boottest/java or src/test/boot/java
comments?
> Thanks,
> Mikhail
>
> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>
>> Mikhail Loenko wrote:
>>
>>> Paulex,
>>>
>>> we have at least 8 categories of tests:
>>> running from classpath or bootclasspath
>>> implementation specific or independent
>>> testing org.apache.harmony.* or java.*
>>> Could you please list how all the tests will be named
>>>
>>>
>> Mikhail,
>>
>> OK, let me try. I'm not sure if all 8 categories are necessary,
>>
>> 1. [classpath], [independent], [java.*]
>> o.a.h.<module name>.tests.api.<package name>, e.g.,
>> o.a.h.luni.tests.api.java.util is test package for java.util
>> I consider the module name is optional and can be removed, so it can be
>> o.a.h.tests.api.java.util for java.util
>>
>> 2. [classpath], [dependent], [o.a.h.*]
>> o.a.h.<module name>.tests.<package name without o.a.h...prefix>,
>> e.g., o.a.h.luni.tests.util is test package for o.a.h.luni.util
>> Again, the module name is optional, but if module name is not
>> prefix, it should be part of package name, so it is:
>> o.a.h.tests.luni.util for o.a.h.luni.util
>>
>> 3. [classpath], [independent], [o.a.h.*]
>> I don't expect these classes/tests exist.
>>
>> 4. [classpath], [dependent], [java.*]
>> if my understanding is correct, this category means the tests for
>> intended different behavior between Harmony and RI, I consider these
>> tests should be put in same package with API tests, and separate them by
>> running script instead of different package, because RI may fix bug in
>> the revision later(e.g, Sun JDK 1.5.1 maybe?), and IIRC in another
>> thread we have agreed that we always follow latest version of RI, so it
>> will be much easier to change the different behavior tests to API tests
>> if we use script to separate them, and vice versa.
>>
>> 5. [bootclasspath],
>> which covers 4 categories you mentioned, the test cases are in same
>> package with implementions
>> e.g.
>> java.util is test package for java.util
>> and
>> o.a.h.luni.util is test packag for o.a.h.luni.util
>>
>> As a summary, we'll have four kind of test packages:
>>
>> a. o.a.h.<module>.tests.api.java.*** for case 1,4 above
>> b. o.a.h.<module>.tests.*** for case 2 above
>> c. java.***, for case 5 above
>> d. o.a.h.***, for case 5 above
>>
>> comments?
>>
>>
>>
>>> Thanks,
>>> Mikhail
>>>
>>>
>>> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>>>
>>>
>>>> Mikhail Loenko wrote:
>>>>
>>>>
>>>>> Geir,
>>>>>
>>>>> The problem is that no one yet suggested a consistent solution
>>>>> that would fit for all the tests and would not get into packages like
>>>>>
>>>>> org.apache.harmony.security.tests.org.apache.harmony.security.util
>>>>>
>>>>>
>>>>> Everybody seems to agree that SOME implementation specific tests are
>>>>> in the same package as implementation.
>>>>>
>>>>> But if we have ALL the tests in the same package as implementation,
>>>>> then we can't run e.g. tests against java.beans.Beans from classpath, so
>>>>> we can't run regression test for HARMONY-358
>>>>>
>>>>> The question is where we put 'classpath' test. If we put them into
>>>>> o.a.h.module.tests.[package under test] we result in that ugly package
>>>>> names like
>>>>> o.a.h.security.tests.o.a.h.security.util
>>>>>
>>>>>
>>>>>
>>>> Mikhail,
>>>>
>>>> Why my prior proposal that o.a.h.security.tests.impl.util is unacceptable?
>>>>
>>>> And I have no objection if "impl" considered redundant, too.
>>>>
>>>>
>>>>
>>>>> Thanks,
>>>>> Mikhail
>>>>>
>>>>>
>>>>> 2006/4/27, Geir Magnusson Jr <ge...@pobox.com>:
>>>>>
>>>>>
>>>>>
>>>>>> Geir Magnusson Jr wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> George Harley wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> >>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>> Of course, the text module has only "implementation-independent tests
>>>>>>>> that designed to be run from classpath". For modules that have got
>>>>>>>> implementation-specific tests then I suppose we could use something
>>>>>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
>>>>>>>> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
>>>>>>>> I've got no preference.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> -1
>>>>>>>
>>>>>>> Put them in the same package as the class being tested, please, if there
>>>>>>> is no problem being run that way.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Sorry to be so snippy. Been a long day.
>>>>>>
>>>>>> Why would we want this convoluted package name? Why wouldn't we want to
>>>>>> let them be in the same package as the class being tested since it's
>>>>>> implementation testing?
>>>>>>
>>>>>> geir
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> Paulex Yang
>>>> China Software Development Lab
>>>> IBM
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>>>
>>>
>> --
>> Paulex Yang
>> China Software Development Lab
>> IBM
>>
>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
>
--
Paulex Yang
China Software Development Lab
IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Mikhail Loenko <ml...@gmail.com>.
The problem is the tests will be mixed like that:
org.apache.harmony.security.asn1 -- tests in bootclasspath
org.apache.harmony.security.cert -- tests in bootclasspath
org.apache.harmony.security.tests -- tests in classpath
org.apache.harmony.security.x509 -- tests in bootclasspath
Thanks,
Mikhail
2006/4/27, Paulex Yang <pa...@gmail.com>:
> Mikhail Loenko wrote:
> > Paulex,
> >
> > we have at least 8 categories of tests:
> > running from classpath or bootclasspath
> > implementation specific or independent
> > testing org.apache.harmony.* or java.*
> > Could you please list how all the tests will be named
> >
> Mikhail,
>
> OK, let me try. I'm not sure if all 8 categories are necessary,
>
> 1. [classpath], [independent], [java.*]
> o.a.h.<module name>.tests.api.<package name>, e.g.,
> o.a.h.luni.tests.api.java.util is test package for java.util
> I consider the module name is optional and can be removed, so it can be
> o.a.h.tests.api.java.util for java.util
>
> 2. [classpath], [dependent], [o.a.h.*]
> o.a.h.<module name>.tests.<package name without o.a.h...prefix>,
> e.g., o.a.h.luni.tests.util is test package for o.a.h.luni.util
> Again, the module name is optional, but if module name is not
> prefix, it should be part of package name, so it is:
> o.a.h.tests.luni.util for o.a.h.luni.util
>
> 3. [classpath], [independent], [o.a.h.*]
> I don't expect these classes/tests exist.
>
> 4. [classpath], [dependent], [java.*]
> if my understanding is correct, this category means the tests for
> intended different behavior between Harmony and RI, I consider these
> tests should be put in same package with API tests, and separate them by
> running script instead of different package, because RI may fix bug in
> the revision later(e.g, Sun JDK 1.5.1 maybe?), and IIRC in another
> thread we have agreed that we always follow latest version of RI, so it
> will be much easier to change the different behavior tests to API tests
> if we use script to separate them, and vice versa.
>
> 5. [bootclasspath],
> which covers 4 categories you mentioned, the test cases are in same
> package with implementions
> e.g.
> java.util is test package for java.util
> and
> o.a.h.luni.util is test packag for o.a.h.luni.util
>
> As a summary, we'll have four kind of test packages:
>
> a. o.a.h.<module>.tests.api.java.*** for case 1,4 above
> b. o.a.h.<module>.tests.*** for case 2 above
> c. java.***, for case 5 above
> d. o.a.h.***, for case 5 above
>
> comments?
>
>
> > Thanks,
> > Mikhail
> >
> >
> > 2006/4/27, Paulex Yang <pa...@gmail.com>:
> >
> >> Mikhail Loenko wrote:
> >>
> >>> Geir,
> >>>
> >>> The problem is that no one yet suggested a consistent solution
> >>> that would fit for all the tests and would not get into packages like
> >>>
> >>> org.apache.harmony.security.tests.org.apache.harmony.security.util
> >>>
> >>>
> >>> Everybody seems to agree that SOME implementation specific tests are
> >>> in the same package as implementation.
> >>>
> >>> But if we have ALL the tests in the same package as implementation,
> >>> then we can't run e.g. tests against java.beans.Beans from classpath, so
> >>> we can't run regression test for HARMONY-358
> >>>
> >>> The question is where we put 'classpath' test. If we put them into
> >>> o.a.h.module.tests.[package under test] we result in that ugly package
> >>> names like
> >>> o.a.h.security.tests.o.a.h.security.util
> >>>
> >>>
> >> Mikhail,
> >>
> >> Why my prior proposal that o.a.h.security.tests.impl.util is unacceptable?
> >>
> >> And I have no objection if "impl" considered redundant, too.
> >>
> >>
> >>> Thanks,
> >>> Mikhail
> >>>
> >>>
> >>> 2006/4/27, Geir Magnusson Jr <ge...@pobox.com>:
> >>>
> >>>
> >>>> Geir Magnusson Jr wrote:
> >>>>
> >>>>
> >>>>> George Harley wrote:
> >>>>>
> >>>>>
> >>>> >>
> >>>>
> >>>>
> >>>>>> Of course, the text module has only "implementation-independent tests
> >>>>>> that designed to be run from classpath". For modules that have got
> >>>>>> implementation-specific tests then I suppose we could use something
> >>>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
> >>>>>> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
> >>>>>> I've got no preference.
> >>>>>>
> >>>>>>
> >>>>> -1
> >>>>>
> >>>>> Put them in the same package as the class being tested, please, if there
> >>>>> is no problem being run that way.
> >>>>>
> >>>>>
> >>>> Sorry to be so snippy. Been a long day.
> >>>>
> >>>> Why would we want this convoluted package name? Why wouldn't we want to
> >>>> let them be in the same package as the class being tested since it's
> >>>> implementation testing?
> >>>>
> >>>> geir
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>>
> >>>>
> >>>>
> >>>>
> >>> ---------------------------------------------------------------------
> >>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>
> >>>
> >>>
> >>>
> >> --
> >> Paulex Yang
> >> China Software Development Lab
> >> IBM
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
> >
>
>
> --
> Paulex Yang
> China Software Development Lab
> IBM
>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Paulex Yang <pa...@gmail.com>.
Mikhail Loenko wrote:
> Paulex,
>
> we have at least 8 categories of tests:
> running from classpath or bootclasspath
> implementation specific or independent
> testing org.apache.harmony.* or java.*
> Could you please list how all the tests will be named
>
Mikhail,
OK, let me try. I'm not sure if all 8 categories are necessary,
1. [classpath], [independent], [java.*]
o.a.h.<module name>.tests.api.<package name>, e.g.,
o.a.h.luni.tests.api.java.util is test package for java.util
I consider the module name is optional and can be removed, so it can be
o.a.h.tests.api.java.util for java.util
2. [classpath], [dependent], [o.a.h.*]
o.a.h.<module name>.tests.<package name without o.a.h...prefix>,
e.g., o.a.h.luni.tests.util is test package for o.a.h.luni.util
Again, the module name is optional, but if module name is not
prefix, it should be part of package name, so it is:
o.a.h.tests.luni.util for o.a.h.luni.util
3. [classpath], [independent], [o.a.h.*]
I don't expect these classes/tests exist.
4. [classpath], [dependent], [java.*]
if my understanding is correct, this category means the tests for
intended different behavior between Harmony and RI, I consider these
tests should be put in same package with API tests, and separate them by
running script instead of different package, because RI may fix bug in
the revision later(e.g, Sun JDK 1.5.1 maybe?), and IIRC in another
thread we have agreed that we always follow latest version of RI, so it
will be much easier to change the different behavior tests to API tests
if we use script to separate them, and vice versa.
5. [bootclasspath],
which covers 4 categories you mentioned, the test cases are in same
package with implementions
e.g.
java.util is test package for java.util
and
o.a.h.luni.util is test packag for o.a.h.luni.util
As a summary, we'll have four kind of test packages:
a. o.a.h.<module>.tests.api.java.*** for case 1,4 above
b. o.a.h.<module>.tests.*** for case 2 above
c. java.***, for case 5 above
d. o.a.h.***, for case 5 above
comments?
> Thanks,
> Mikhail
>
>
> 2006/4/27, Paulex Yang <pa...@gmail.com>:
>
>> Mikhail Loenko wrote:
>>
>>> Geir,
>>>
>>> The problem is that no one yet suggested a consistent solution
>>> that would fit for all the tests and would not get into packages like
>>>
>>> org.apache.harmony.security.tests.org.apache.harmony.security.util
>>>
>>>
>>> Everybody seems to agree that SOME implementation specific tests are
>>> in the same package as implementation.
>>>
>>> But if we have ALL the tests in the same package as implementation,
>>> then we can't run e.g. tests against java.beans.Beans from classpath, so
>>> we can't run regression test for HARMONY-358
>>>
>>> The question is where we put 'classpath' test. If we put them into
>>> o.a.h.module.tests.[package under test] we result in that ugly package
>>> names like
>>> o.a.h.security.tests.o.a.h.security.util
>>>
>>>
>> Mikhail,
>>
>> Why my prior proposal that o.a.h.security.tests.impl.util is unacceptable?
>>
>> And I have no objection if "impl" considered redundant, too.
>>
>>
>>> Thanks,
>>> Mikhail
>>>
>>>
>>> 2006/4/27, Geir Magnusson Jr <ge...@pobox.com>:
>>>
>>>
>>>> Geir Magnusson Jr wrote:
>>>>
>>>>
>>>>> George Harley wrote:
>>>>>
>>>>>
>>>> >>
>>>>
>>>>
>>>>>> Of course, the text module has only "implementation-independent tests
>>>>>> that designed to be run from classpath". For modules that have got
>>>>>> implementation-specific tests then I suppose we could use something
>>>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
>>>>>> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
>>>>>> I've got no preference.
>>>>>>
>>>>>>
>>>>> -1
>>>>>
>>>>> Put them in the same package as the class being tested, please, if there
>>>>> is no problem being run that way.
>>>>>
>>>>>
>>>> Sorry to be so snippy. Been a long day.
>>>>
>>>> Why would we want this convoluted package name? Why wouldn't we want to
>>>> let them be in the same package as the class being tested since it's
>>>> implementation testing?
>>>>
>>>> geir
>>>>
>>>> ---------------------------------------------------------------------
>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>>>
>>>
>> --
>> Paulex Yang
>> China Software Development Lab
>> IBM
>>
>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
>
--
Paulex Yang
China Software Development Lab
IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Mikhail Loenko <ml...@gmail.com>.
Paulex,
we have at least 8 categories of tests:
running from classpath or bootclasspath
implementation specific or independent
testing org.apache.harmony.* or java.*
Could you please list how all the tests will be named
Thanks,
Mikhail
2006/4/27, Paulex Yang <pa...@gmail.com>:
> Mikhail Loenko wrote:
> > Geir,
> >
> > The problem is that no one yet suggested a consistent solution
> > that would fit for all the tests and would not get into packages like
> >
> > org.apache.harmony.security.tests.org.apache.harmony.security.util
> >
> >
> > Everybody seems to agree that SOME implementation specific tests are
> > in the same package as implementation.
> >
> > But if we have ALL the tests in the same package as implementation,
> > then we can't run e.g. tests against java.beans.Beans from classpath, so
> > we can't run regression test for HARMONY-358
> >
> > The question is where we put 'classpath' test. If we put them into
> > o.a.h.module.tests.[package under test] we result in that ugly package
> > names like
> > o.a.h.security.tests.o.a.h.security.util
> >
> Mikhail,
>
> Why my prior proposal that o.a.h.security.tests.impl.util is unacceptable?
>
> And I have no objection if "impl" considered redundant, too.
>
> > Thanks,
> > Mikhail
> >
> >
> > 2006/4/27, Geir Magnusson Jr <ge...@pobox.com>:
> >
> >> Geir Magnusson Jr wrote:
> >>
> >>> George Harley wrote:
> >>>
> >> >>
> >>
> >>>> Of course, the text module has only "implementation-independent tests
> >>>> that designed to be run from classpath". For modules that have got
> >>>> implementation-specific tests then I suppose we could use something
> >>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
> >>>> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
> >>>> I've got no preference.
> >>>>
> >>> -1
> >>>
> >>> Put them in the same package as the class being tested, please, if there
> >>> is no problem being run that way.
> >>>
> >> Sorry to be so snippy. Been a long day.
> >>
> >> Why would we want this convoluted package name? Why wouldn't we want to
> >> let them be in the same package as the class being tested since it's
> >> implementation testing?
> >>
> >> geir
> >>
> >> ---------------------------------------------------------------------
> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
> >
>
>
> --
> Paulex Yang
> China Software Development Lab
> IBM
>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Paulex Yang <pa...@gmail.com>.
Mikhail Loenko wrote:
> Geir,
>
> The problem is that no one yet suggested a consistent solution
> that would fit for all the tests and would not get into packages like
>
> org.apache.harmony.security.tests.org.apache.harmony.security.util
>
>
> Everybody seems to agree that SOME implementation specific tests are
> in the same package as implementation.
>
> But if we have ALL the tests in the same package as implementation,
> then we can't run e.g. tests against java.beans.Beans from classpath, so
> we can't run regression test for HARMONY-358
>
> The question is where we put 'classpath' test. If we put them into
> o.a.h.module.tests.[package under test] we result in that ugly package
> names like
> o.a.h.security.tests.o.a.h.security.util
>
Mikhail,
Why my prior proposal that o.a.h.security.tests.impl.util is unacceptable?
And I have no objection if "impl" considered redundant, too.
> Thanks,
> Mikhail
>
>
> 2006/4/27, Geir Magnusson Jr <ge...@pobox.com>:
>
>> Geir Magnusson Jr wrote:
>>
>>> George Harley wrote:
>>>
>> >>
>>
>>>> Of course, the text module has only "implementation-independent tests
>>>> that designed to be run from classpath". For modules that have got
>>>> implementation-specific tests then I suppose we could use something
>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
>>>> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
>>>> I've got no preference.
>>>>
>>> -1
>>>
>>> Put them in the same package as the class being tested, please, if there
>>> is no problem being run that way.
>>>
>> Sorry to be so snippy. Been a long day.
>>
>> Why would we want this convoluted package name? Why wouldn't we want to
>> let them be in the same package as the class being tested since it's
>> implementation testing?
>>
>> geir
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
>
--
Paulex Yang
China Software Development Lab
IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Geir Magnusson Jr <ge...@pobox.com>.
Mikhail Loenko wrote:
> Geir,
>
> The problem is that no one yet suggested a consistent solution
> that would fit for all the tests and would not get into packages like
>
> org.apache.harmony.security.tests.org.apache.harmony.security.util
>
>
> Everybody seems to agree that SOME implementation specific tests are
> in the same package as implementation.
>
> But if we have ALL the tests in the same package as implementation,
> then we can't run e.g. tests against java.beans.Beans from classpath, so
> we can't run regression test for HARMONY-358
Right - I thought that some tests can't be in implementation package -
i.e. the API tests that we wish to run on classpath - and therefore we
put those somewhere else.
I thought that "somewhere else" was o.a.security.tests.<whatever>
>
> The question is where we put 'classpath' test. If we put them into
> o.a.h.module.tests.[package under test] we result in that ugly package
> names like
> o.a.h.security.tests.o.a.h.security.util
Why not
o.a.h.security.tests.util
?
geir
>
> Thanks,
> Mikhail
>
>
> 2006/4/27, Geir Magnusson Jr <ge...@pobox.com>:
>> Geir Magnusson Jr wrote:
>>> George Harley wrote:
>> >>
>>>> Of course, the text module has only "implementation-independent tests
>>>> that designed to be run from classpath". For modules that have got
>>>> implementation-specific tests then I suppose we could use something
>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
>>>> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
>>>> I've got no preference.
>>> -1
>>>
>>> Put them in the same package as the class being tested, please, if there
>>> is no problem being run that way.
>> Sorry to be so snippy. Been a long day.
>>
>> Why would we want this convoluted package name? Why wouldn't we want to
>> let them be in the same package as the class being tested since it's
>> implementation testing?
>>
>> geir
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Mikhail Loenko <ml...@gmail.com>.
Geir,
The problem is that no one yet suggested a consistent solution
that would fit for all the tests and would not get into packages like
org.apache.harmony.security.tests.org.apache.harmony.security.util
Everybody seems to agree that SOME implementation specific tests are
in the same package as implementation.
But if we have ALL the tests in the same package as implementation,
then we can't run e.g. tests against java.beans.Beans from classpath, so
we can't run regression test for HARMONY-358
The question is where we put 'classpath' test. If we put them into
o.a.h.module.tests.[package under test] we result in that ugly package
names like
o.a.h.security.tests.o.a.h.security.util
Thanks,
Mikhail
2006/4/27, Geir Magnusson Jr <ge...@pobox.com>:
>
> Geir Magnusson Jr wrote:
> >
> > George Harley wrote:
> >>
> >> Of course, the text module has only "implementation-independent tests
> >> that designed to be run from classpath". For modules that have got
> >> implementation-specific tests then I suppose we could use something
> >> like "org.apache.harmony.[module].tests.impl.[package under test]" or
> >> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
> >> I've got no preference.
> >
> > -1
> >
> > Put them in the same package as the class being tested, please, if there
> > is no problem being run that way.
>
> Sorry to be so snippy. Been a long day.
>
> Why would we want this convoluted package name? Why wouldn't we want to
> let them be in the same package as the class being tested since it's
> implementation testing?
>
> geir
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Geir Magnusson Jr <ge...@pobox.com>.
Geir Magnusson Jr wrote:
>
> George Harley wrote:
>>
>> Of course, the text module has only "implementation-independent tests
>> that designed to be run from classpath". For modules that have got
>> implementation-specific tests then I suppose we could use something
>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
>> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
>> I've got no preference.
>
> -1
>
> Put them in the same package as the class being tested, please, if there
> is no problem being run that way.
Sorry to be so snippy. Been a long day.
Why would we want this convoluted package name? Why wouldn't we want to
let them be in the same package as the class being tested since it's
implementation testing?
geir
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Geir Magnusson Jr <ge...@pobox.com>.
George Harley wrote:
> Mikhail Loenko wrote:
>> Hello
>>
>> I'd like to bring this thread back.
>>
>> Number of tests is growing and it is time to put them in order.
>>
>> So far we may have:
>>
>> 1) implementation-specific tests that designed to be run from
>> bootclasspath
>> 2) implementation-specific tests that might be run from classpath
>> 3) implementation-specific tests that designed to be run from classpath
>> 4) implementation-independent tests that designed to be run from
>> bootclasspath
>> 5) implementation-independent tests that might be run from classpath
>> 6) implementation-independent tests that designed to be run from
>> classpath
>>
>> Also we seem to have the following packages, where the tests are:
>>
>> 1) the same package as implementation
>> 2) org.apache.harmony.tests.[the same package as implementation]
>> 3) tests.api.[the same package as implementation]
>>
>> I suggest that we work out step-by-step solution as we could not reach
>> an agreement for the general universal one
>>
>> So for the first step I suggest that we separate i-independent tests
>> that must or may be run from classpath
>>
>> I suggest that we put them into package
>> tests.module.compatible.[package of implementation being tested]
>>
>> Comments?
>>
>> Thanks,
>> Mikhail
>>
>
> Hi Mikhail,
>
> I've just started working through the modules to merge test packages
> "tests.api.[same package as implementation]" and "tests.api.[same
> package as implementation]" into one package space. Using the class
> library package naming guidelines from off the web site [1], all of the
> tests for the text module have been consolidated under
> org.apache.harmony.text.tests.[package under test].
Are these API spec tests or implementation tests? If the former, +1.
if the latter, lets talk about it.
>
> Of course, the text module has only "implementation-independent tests
> that designed to be run from classpath". For modules that have got
> implementation-specific tests then I suppose we could use something like
> "org.apache.harmony.[module].tests.impl.[package under test]" or
> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
> I've got no preference.
-1
Put them in the same package as the class being tested, please, if there
is no problem being run that way.
geir
>
> Best regards,
> George
>
>
> [1]
> http://incubator.apache.org/harmony/subcomponents/classlibrary/pkgnaming.html
>
>
>
>>
>> 2006/3/24, George Harley <ge...@googlemail.com>:
>>
>>> Geir Magnusson Jr wrote:
>>>
>>>> Leo Simons wrote:
>>>>
>>>>> On Wed, Mar 22, 2006 at 08:02:44AM -0500, Geir Magnusson Jr wrote:
>>>>>
>>>>>> Leo Simons wrote:
>>>>>>
>>>>>>> On Wed, Mar 22, 2006 at 07:15:28AM -0500, Geir Magnusson Jr wrote:
>>>>>>>
>>>>>>>> Pulling out of the various threads where we have been discussing,
>>>>>>>> can we agree on the problem :
>>>>>>>>
>>>>>>>> We have unique problems compared to other Java projects because we
>>>>>>>> need to find a way to reliably test the things that are commonly
>>>>>>>> expected to be a solid point of reference - namely the core class
>>>>>>>> library.
>>>>>>>>
>>>>>>>> Further, we've been implicitly doing "integration testing" because
>>>>>>>> - so far - the only way we've been testing our code has been 'in
>>>>>>>> situ' in the VM - not in an isolated test harness. To me, this
>>>>>>>> turns it into an integration test.
>>>>>>>>
>>>>>>>> Sure, we're using JUnit, but because of the fact we are
>>>>>>>> implmenting core java.* APIs, we aren't testing with a framework
>>>>>>>> that has been independently tested for correctness, like we would
>>>>>>>> when testing any other code.
>>>>>>>>
>>>>>>>> I hope I got that idea across - I believe that we have to go
>>>>>>>> beyond normal testing approaches because we don't have a normal
>>>>>>>> situation.
>>>>>>>>
>>>>>>> Where we define 'normal situation' as "running a test framework on
>>>>>>> top of
>>>>>>> the sun jdk and expecting any bugs to not be in that jdk". There's
>>>>>>> plenty
>>>>>>> of projects out there that have to test things without having such a
>>>>>>> "stable reference JDK" luxury.....I imagine that testing GCC is
>>>>>>> just as
>>>>>>> hard as this problem we have here :-)
>>>>>>>
>>>>>> Is it the same? We need to have a running JVM+classlibarary to test
>>>>>> the classlibrary code.
>>>>>>
>>>>> Well you need a working C compiler and standard C library to
>>>>> compile the
>>>>> compiler so you can compile make so you can build bash so you can run
>>>>> perl (which uses the standard C library functions all over the
>>>>> place of
>>>>> course) so you can run the standard C library tests so that you know
>>>>> that
>>>>> the library you used when compiling the compiler were correct so
>>>>> you can
>>>>> run the compiler tests. I don't think they actually do things that
>>>>> way, but
>>>>> it seems like basically the same problem. Having a virtual machine
>>>>> just
>>>>> makes it easier since you still assume "the native world" as a
>>>>> baseline,
>>>>> which is a lot more than "the hardware".
>>>>>
>>>> There's a difference. You can use a completely separate toolchain to
>>>> build, test and verify the output of the C compiler.
>>>>
>>>> In our case, we are using the thing we are testing to test itself.
>>>> There is no "known good" element possible right now.
>>>>
>>>> We use the classlibrary we are trying to test to execute the test
>>>> framework that tests the classlibrary that is running it.
>>>>
>>>> The tool is testing itself.
>>>>
>>>>
>>>>
>>>>>>>> So I think there are three things we want to do (adopting the
>>>>>>>> terminology that came from the discussion with Tim and Leo ) :
>>>>>>>>
>>>>>>>> 1) implementation tests
>>>>>>>> 2) spec/API tests (I'll bundle together)
>>>>>>>> 3) integration/functional tests
>>>>>>>>
>>>>>>>> I believe that for #1, the issues related to being on the
>>>>>>>> bootclasspath don't matter, because we aren't testing that aspect
>>>>>>>> of the classes (which is how they behave integrated w/ the VM and
>>>>>>>> security system) but rather the basic internal functioning.
>>>>>>>>
>>>>>>>> I'm not sure how to approach this, but I'll try. I'd love to hear
>>>>>>>> how Sun, IBM or BEA deals with this, or be told why it isn't an
>>>>>>>> issue :)
>>>>>>>>
>>>>>>>> Implementation tests : I'd like to see us be able to do #1 via the
>>>>>>>> standard same-package technique (i.e. testing a.b.C w/ a.b.CTest)
>>>>>>>> but we'll run into a tangle of classloader problems, I suspect,
>>>>>>>> becuase we want to be testing java.* code in a system that already
>>>>>>>> has java.* code. Can anyone see a way we can do this - test the
>>>>>>>> classlibrary from the integration point of view - using some test
>>>>>>>> harness + any known-good JRE, like Sun's or IBM's?
>>>>>>>>
>>>>>>> Ew, that won't work in the end since we should assume our own JRE
>>>>>>> is going
>>>>>>> to be "known-better" :-). But it might be a nice way to "bootstrap"
>>>>>>> (eg
>>>>>>> we test with an external JRE until we satisfy the tests and then we
>>>>>>> switch
>>>>>>> to testing with an earlier build).
>>>>>>>
>>>>>> Lets be clear - even using our own "earlier build" doesn't solve the
>>>>>> problem I'm describing, because as it stands now, we don't use
>>>>>> "earlier build" classes to test with - we use the code we want to
>>>>>> test as the clsaslibrary for the JRE that's running the test
>>>>>> framework.
>>>>>>
>>>>>> The classes that we are testing are also the classes used by the
>>>>>> testing framework. IOW, any of the java.* classes that JUnit itself
>>>>>> needs (ex. java.util.HashMap) are exactly the same implementation
>>>>>> that it's testing.
>>>>>>
>>>>>> That's why I think it's subtly different than a "bootstrap and use
>>>>>> version - 1 to test" problem. See what I mean?
>>>>>>
>>>>> Yeah yeah, I was already way beyond thinking "just" JUnit is usable
>>>>> for the
>>>>> kind of test you're describing. At some point, fundamentally, you
>>>>> either trust
>>>>> something external (whether its the sun jdk or the intel compiler
>>>>> designers,
>>>>> at some point you do draw a line) or you find a way to bootstrap.
>>>>>
>>>> Well, we do trust the Sun JDK.
>>>>
>>>>
>>>>>> I'm very open to the idea that I'm missing something here, but I'd
>>>>>> like to know that you see the issue - that when we test, we have
>>>>>>
>>>>>> VM + "classlib to be tested" + JUnit + testcases
>>>>>>
>>>>>> where the testcases are testing the classlib the VM is running JUnit
>>>>>> with.
>>>>>>
>>>>>> There never is isolation of the code being tested :
>>>>>>
>>>>>> VM + "known good classlib" + Junit + testcases
>>>>>>
>>>>>> unless we have some framework where
>>>>>>
>>>>>> VM + "known good classlib" + JUnit
>>>>>> + framework("classlib to be tested")
>>>>>> + testcases
>>>>>>
>>>>>> and it's that notion of "framework()" that I'm advocating we explore.
>>>>>>
>>>>> I'm all for exploring it, I just fundamentally don't buy into the
>>>>> "known
>>>>> good" bit. What happens when the 'classlib to be tested' is 'known
>>>>> better' than the 'known good' one? How do you define "known"? How
>>>>> do you
>>>>> define "good"?
>>>>>
>>>> Known? Passed some set of tests. So it could be the Sun JDK for the
>>>> VM + "known good" part.
>>>>
>>>> I think you intuitively understand this. When you find a bug in code
>>>> you are testing, you first assume it's your code, not the framework,
>>>> right? In our case, our framework is actually the code we are
>>>> testing, so we have a bit of a logical conundrum.
>>>>
>>>>
>>> Hi Geir,
>>>
>>> The number of Harmony public API classes that get loaded just to run the
>>> JUnit harness is a little over 200. The majority of these are out of
>>> LUNI with a very low number coming from each of Security, NIO, Archive
>>> and Text.
>>>
>>> Sure there is a circular dependency between what we are building and the
>>> framework we are using to test it but it appears to touch on only a
>>> relatively small part of Harmony....IMHO.
>>>
>>> Best regards,
>>> George
>>>
>>>
>>>
>>>>>>> Further ideas...
>>>>>>>
>>>>>>> -> look at how the native world does testing
>>>>>>> (hint: it usually has #ifdefs, uses perl along the way, and it is
>>>>>>> certainly
>>>>>>> "messy")
>>>>>>> -> emulate that
>>>>>>>
>>>>>>> -> build a bigger, better specification test
>>>>>>> -> and somehow "prove" it is "good enough"
>>>>>>>
>>>>>>> -> build a bigger, better integration test
>>>>>>> -> and somehow "prove" it is "good enough"
>>>>>>>
>>>>>>> I'll admit my primary interest is the last one...
>>>>>>>
>>>>>> The problem I see with the last one is that the "parameter space" is
>>>>>> *huge*.
>>>>>>
>>>>> Yeah, that's one of the things that makes it interesting. Fortunately
>>>>> open source does have many monkeys...
>>>>>
>>>>>
>>>>>> I believe that your preference for the last one comes from the
>>>>>> Monte-Carlo style approach that Gump uses - hope that your test
>>>>>> suite has enough variance that you "push" the thing being tested
>>>>>> through enough of the parameter space that you can be comfortable
>>>>>> you would have exposed the bugs. Maybe.
>>>>>>
>>>>> Ooh, now its becoming rather abstract...
>>>>>
>>>>> Well, perhaps, but more of the gump approache comes from the idea that
>>>>> the parameter space itself is also at some point defined in software,
>>>>> which may have bugs of its own. You circumvent that by making
>>>>> humans the
>>>>> parameter space (don't start about how humans are buggy. We don't
>>>>> want to
>>>>> get into existialism or faith systems when talking about unit
>>>>> testing do
>>>>> we?). The thing that gump enables is "many monkey QA" - a way for
>>>>> thousands
>>>>> of human beings to concurrently make shared assertions about software
>>>>> without actually needing all that much human interaction.
>>>>>
>>>>> More concretely, if harmony can run all known java software, and run
>>>>> it to
>>>>> the asserted satisfaction of all its developers, you can trust that
>>>>> you have
>>>>> covered all the /relevant/ parts of the parameter space you describe.
>>>>>
>>>> Yes. And when you can run all knownn Java software, let me know :)
>>>> That's my point about the parameter space being huge. Even when you
>>>> reduce the definition to "that of all known Java software", you still
>>>> have a huge problem on your hands.
>>>>
>>>>
>>>>> You
>>>>> will never get that level of trust when the assertions are made by
>>>>> software
>>>>> rather than humans. This is how open source leads to software quality.
>>>>>
>>>>> Quoting myself, 'gump is the most misunderstood piece of software,
>>>>> ever'.
>>>>>
>>>>> cheers,
>>>>>
>>>>> Leo
>>>>>
>>>>>
>>>>>
>>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Tim Ellison <t....@gmail.com>.
Geir Magnusson Jr wrote:
> Ok - so this is an aspect of modularization, rather than some deranged
> bending of package implementation? :D
Yes.
> So if a "module" has public packages A, B and C would it be :
.. and therein lies the problem, there is no such thing as a 'public
package' and a 'non-public' package in Java syntax (yet), so we have the
naming convention.
> org.apache.harmony.module.A
> org.apache.harmony.module.B
> org.apache.harmony.module.C
> org.apache.harmony.module.internal
>
> not
>
> org.apache.harmony.module.A
> org.apache.harmony.module.A.internal
> org.apache.harmony.module.B
> org.apache.harmony.module.B.internal
> org.apache.harmony.module.C
> org.apache.harmony.module.C.internal
Yep.
Regards,
Tim
> Tim Ellison wrote:
>> '.internal.' is used to denote packages containing types that are wholly
>> the business of a particular module; whereas non-internal packages
>> contain types that can be called from other modules (e.g. utilities) and
>> are expected to be stable.
>>
>> In OSGi speak, we will export all packages that are *not* marked
>> '.internal.', and all packages that are marked as '.internal.' will not
>> be exported.
>>
>> The naming convention is simply our convention to identify
>> internal-APIs, it is not required by OSGi/Eclipse/...
>>
>> Regards,
>> Tim
>>
>>
>> Geir Magnusson Jr wrote:
>>>
>>> Oliver Deakin wrote:
>>>> George Harley wrote:
>>>>> Mikhail Loenko wrote:
>>>>> Of course, the text module has only "implementation-independent tests
>>>>> that designed to be run from classpath". For modules that have got
>>>>> implementation-specific tests then I suppose we could use something
>>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
>>>>> "org.apache.harmony.[module].tests.internal.[package under test]"
>>>>> etc. I've got no preference.
>>>> I think impl is preferable over internal here, as we already use
>>>> internal in our implementation package names to indicate classes
>>>> totally internal to that bundle. To also use internal to label tests
>>>> that are implementation specific may cause confusion.
>>> I think the whole 'internal' thing is just awful IMO. (Man, it feels
>>> good to stay that...)
>>>
>>> Why do we need it? Eclipse? OSGi?
>>>
>>> Isn't it pre-supposing a packaging system in the source code structur?
>>> (one that I think is pretty unnatural for java programmers....)
>>>
>>> I'm 100% behind offering the Harmony classlibs packaged for OSGi, but
>>> I'm 100% against assuming it's the only way to go...
>>>
>>> geir
>>>
>>>
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
--
Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Geir Magnusson Jr <ge...@pobox.com>.
Ok - so this is an aspect of modularization, rather than some deranged
bending of package implementation? :D
So if a "module" has public packages A, B and C would it be :
org.apache.harmony.module.A
org.apache.harmony.module.B
org.apache.harmony.module.C
org.apache.harmony.module.internal
not
org.apache.harmony.module.A
org.apache.harmony.module.A.internal
org.apache.harmony.module.B
org.apache.harmony.module.B.internal
org.apache.harmony.module.C
org.apache.harmony.module.C.internal
geir
Tim Ellison wrote:
> '.internal.' is used to denote packages containing types that are wholly
> the business of a particular module; whereas non-internal packages
> contain types that can be called from other modules (e.g. utilities) and
> are expected to be stable.
>
> In OSGi speak, we will export all packages that are *not* marked
> '.internal.', and all packages that are marked as '.internal.' will not
> be exported.
>
> The naming convention is simply our convention to identify
> internal-APIs, it is not required by OSGi/Eclipse/...
>
> Regards,
> Tim
>
>
> Geir Magnusson Jr wrote:
>>
>> Oliver Deakin wrote:
>>> George Harley wrote:
>>>> Mikhail Loenko wrote:
>>>> Of course, the text module has only "implementation-independent tests
>>>> that designed to be run from classpath". For modules that have got
>>>> implementation-specific tests then I suppose we could use something
>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
>>>> "org.apache.harmony.[module].tests.internal.[package under test]"
>>>> etc. I've got no preference.
>>> I think impl is preferable over internal here, as we already use
>>> internal in our implementation package names to indicate classes
>>> totally internal to that bundle. To also use internal to label tests
>>> that are implementation specific may cause confusion.
>> I think the whole 'internal' thing is just awful IMO. (Man, it feels
>> good to stay that...)
>>
>> Why do we need it? Eclipse? OSGi?
>>
>> Isn't it pre-supposing a packaging system in the source code structur?
>> (one that I think is pretty unnatural for java programmers....)
>>
>> I'm 100% behind offering the Harmony classlibs packaged for OSGi, but
>> I'm 100% against assuming it's the only way to go...
>>
>> geir
>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Tim Ellison <t....@gmail.com>.
'.internal.' is used to denote packages containing types that are wholly
the business of a particular module; whereas non-internal packages
contain types that can be called from other modules (e.g. utilities) and
are expected to be stable.
In OSGi speak, we will export all packages that are *not* marked
'.internal.', and all packages that are marked as '.internal.' will not
be exported.
The naming convention is simply our convention to identify
internal-APIs, it is not required by OSGi/Eclipse/...
Regards,
Tim
Geir Magnusson Jr wrote:
>
>
> Oliver Deakin wrote:
>> George Harley wrote:
>>> Mikhail Loenko wrote:
>>> Of course, the text module has only "implementation-independent tests
>>> that designed to be run from classpath". For modules that have got
>>> implementation-specific tests then I suppose we could use something
>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
>>> "org.apache.harmony.[module].tests.internal.[package under test]"
>>> etc. I've got no preference.
>>
>> I think impl is preferable over internal here, as we already use
>> internal in our implementation package names to indicate classes
>> totally internal to that bundle. To also use internal to label tests
>> that are implementation specific may cause confusion.
>
> I think the whole 'internal' thing is just awful IMO. (Man, it feels
> good to stay that...)
>
> Why do we need it? Eclipse? OSGi?
>
> Isn't it pre-supposing a packaging system in the source code structur?
> (one that I think is pretty unnatural for java programmers....)
>
> I'm 100% behind offering the Harmony classlibs packaged for OSGi, but
> I'm 100% against assuming it's the only way to go...
>
> geir
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
--
Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Geir Magnusson Jr <ge...@pobox.com>.
Oliver Deakin wrote:
> George Harley wrote:
>> Mikhail Loenko wrote:
>> Of course, the text module has only "implementation-independent tests
>> that designed to be run from classpath". For modules that have got
>> implementation-specific tests then I suppose we could use something
>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
>> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
>> I've got no preference.
>
> I think impl is preferable over internal here, as we already use
> internal in our implementation package names to indicate classes totally
> internal to that bundle. To also use internal to label tests that are
> implementation specific may cause confusion.
I think the whole 'internal' thing is just awful IMO. (Man, it feels
good to stay that...)
Why do we need it? Eclipse? OSGi?
Isn't it pre-supposing a packaging system in the source code structur?
(one that I think is pretty unnatural for java programmers....)
I'm 100% behind offering the Harmony classlibs packaged for OSGi, but
I'm 100% against assuming it's the only way to go...
geir
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Anton Avtamonov <an...@gmail.com>.
On 4/26/06, Mikhail Loenko <ml...@gmail.com> wrote:
> Seems like implemantation-specific tests for java.security package do not
> fit well into the proposed solution
>
> I think we do not have to stick to requirement about
> organization name in the tests: we already have tests in package java.*
> why don't have package tests.*?
+1. The smaller and obvious package names we have the easier to navigate.
"tests.*" looks very obvious and small.
Wishes,
--
Anton Avtamonov,
Intel Middleware Products Division
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Paulex Yang <pa...@gmail.com>.
Mikhail Loenko wrote:
> Seems like implemantation-specific tests for java.security package do not
> fit well into the proposed solution
>
> I think we do not have to stick to requirement about
> organization name in the tests: we already have tests in package java.*
> why don't have package tests.*?
>
Well, this is another issue. If the organization name is not necessary
as prefix, then I'm fine with the tests.<package name>.
> Thanks,
> Mikhail
>
> 2006/4/26, Paulex Yang <pa...@gmail.com>:
>
>> Richard Liang wrote:
>>
>>> Mikhail Loenko wrote:
>>>
>>>> how about 'specific'? impl seems to be not very informative.
>>>>
>>>>
>>>>
>>> +1 to Oliver's impl :-)
>>>
>>>> I have a concern abou proposed package naming guidelines:
>>>> package name
>>>> org.apache.harmony.security.tests.org.apache.harmony.security
>>>> is not much better then 1000-character long test name.
>>>>
>>>>
>>>>
>>> +1. I think the prefix org.apache.harmony.security is unnecessary.
>>> "tests.impl.org.apache.harmony.security" is enough to tell people what
>>> the test cases belong to
>>>
>> I agree it is duplicated and unnecessary, but it is a general convention
>> for Java application to start with organization name, i.e.,
>> "org.apache.harmony". If I understand correctly, our implementation
>> package's name must start with "org.apache.harmony.<module name>", so
>> how about this:
>>
>> org.apache.harmony.<module name>.tests.impl.<rest of package name>
>>
>> as a example, test for org.apache.harmony.security.util will be named as:
>> org.apache.harmony.security.tests.impl.util
>>
>> while the API test still be named as
>> org.apache.harmony.security.tests.api.java.security
>>
>> comments?
>>
>>> Any comments? Thanks a lot.
>>>
>>>> Thanks,
>>>> Mikhail
>>>>
>>>>
>>>> 2006/4/26, Paulex Yang <pa...@gmail.com>:
>>>>
>>>>
>>>>> Oliver Deakin wrote:
>>>>>
>>>>>
>>>>>> George Harley wrote:
>>>>>>
>>>>>>
>>>>>>> Mikhail Loenko wrote:
>>>>>>>
>>>>>>>
>>>>>>>> Hello
>>>>>>>>
>>>>>>>> I'd like to bring this thread back.
>>>>>>>>
>>>>>>>> Number of tests is growing and it is time to put them in order.
>>>>>>>>
>>>>>>>> So far we may have:
>>>>>>>>
>>>>>>>> 1) implementation-specific tests that designed to be run from
>>>>>>>> bootclasspath
>>>>>>>> 2) implementation-specific tests that might be run from classpath
>>>>>>>> 3) implementation-specific tests that designed to be run from
>>>>>>>> classpath
>>>>>>>> 4) implementation-independent tests that designed to be run from
>>>>>>>> bootclasspath
>>>>>>>> 5) implementation-independent tests that might be run from classpath
>>>>>>>> 6) implementation-independent tests that designed to be run from
>>>>>>>> classpath
>>>>>>>>
>>>>>>>> Also we seem to have the following packages, where the tests are:
>>>>>>>>
>>>>>>>> 1) the same package as implementation
>>>>>>>> 2) org.apache.harmony.tests.[the same package as implementation]
>>>>>>>> 3) tests.api.[the same package as implementation]
>>>>>>>>
>>>>>>>> I suggest that we work out step-by-step solution as we could not
>>>>>>>> reach
>>>>>>>> an agreement for the general universal one
>>>>>>>>
>>>>>>>> So for the first step I suggest that we separate i-independent tests
>>>>>>>> that must or may be run from classpath
>>>>>>>>
>>>>>>>> I suggest that we put them into package
>>>>>>>> tests.module.compatible.[package of implementation being tested]
>>>>>>>>
>>>>>>>> Comments?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Mikhail
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> Hi Mikhail,
>>>>>>>
>>>>>>> I've just started working through the modules to merge test packages
>>>>>>> "tests.api.[same package as implementation]" and "tests.api.[same
>>>>>>> package as implementation]" into one package space. Using the class
>>>>>>> library package naming guidelines from off the web site [1], all of
>>>>>>> the tests for the text module have been consolidated under
>>>>>>> org.apache.harmony.text.tests.[package under test].
>>>>>>>
>>>>>>> Of course, the text module has only "implementation-independent tests
>>>>>>> that designed to be run from classpath". For modules that have got
>>>>>>> implementation-specific tests then I suppose we could use something
>>>>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
>>>>>>> "org.apache.harmony.[module].tests.internal.[package under test]"
>>>>>>> etc. I've got no preference.
>>>>>>>
>>>>>>>
>>>>>> I think impl is preferable over internal here, as we already use
>>>>>> internal in our implementation package names to indicate classes
>>>>>> totally internal to that bundle. To also use internal to label tests
>>>>>> that are implementation specific may cause confusion.
>>>>>>
>>>>>>
>>>>>>
>>>>> +1 from me.
>>>>>
>>>>>
>>>>>> Regards,
>>>>>> Oliver
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Best regards,
>>>>>>> George
>>>>>>>
>>>>>>>
>>>>>>> [1]
>>>>>>> http://incubator.apache.org/harmony/subcomponents/classlibrary/pkgnaming.html
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> 2006/3/24, George Harley <ge...@googlemail.com>:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Geir Magnusson Jr wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Leo Simons wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On Wed, Mar 22, 2006 at 08:02:44AM -0500, Geir Magnusson Jr
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Leo Simons wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Mar 22, 2006 at 07:15:28AM -0500, Geir Magnusson Jr
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Pulling out of the various threads where we have been
>>>>>>>>>>>>>> discussing,
>>>>>>>>>>>>>> can we agree on the problem :
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We have unique problems compared to other Java projects
>>>>>>>>>>>>>> because we
>>>>>>>>>>>>>> need to find a way to reliably test the things that are
>>>>>>>>>>>>>> commonly
>>>>>>>>>>>>>> expected to be a solid point of reference - namely the core
>>>>>>>>>>>>>> class
>>>>>>>>>>>>>> library.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Further, we've been implicitly doing "integration testing"
>>>>>>>>>>>>>> because
>>>>>>>>>>>>>> - so far - the only way we've been testing our code has
>>>>>>>>>>>>>> been 'in
>>>>>>>>>>>>>> situ' in the VM - not in an isolated test harness. To me,
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>> turns it into an integration test.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sure, we're using JUnit, but because of the fact we are
>>>>>>>>>>>>>> implmenting core java.* APIs, we aren't testing with a
>>>>>>>>>>>>>> framework
>>>>>>>>>>>>>> that has been independently tested for correctness, like we
>>>>>>>>>>>>>> would
>>>>>>>>>>>>>> when testing any other code.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I hope I got that idea across - I believe that we have to go
>>>>>>>>>>>>>> beyond normal testing approaches because we don't have a
>>>>>>>>>>>>>> normal
>>>>>>>>>>>>>> situation.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Where we define 'normal situation' as "running a test
>>>>>>>>>>>>> framework on
>>>>>>>>>>>>> top of
>>>>>>>>>>>>> the sun jdk and expecting any bugs to not be in that jdk".
>>>>>>>>>>>>> There's
>>>>>>>>>>>>> plenty
>>>>>>>>>>>>> of projects out there that have to test things without having
>>>>>>>>>>>>> such a
>>>>>>>>>>>>> "stable reference JDK" luxury.....I imagine that testing GCC is
>>>>>>>>>>>>> just as
>>>>>>>>>>>>> hard as this problem we have here :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> Is it the same? We need to have a running JVM+classlibarary to
>>>>>>>>>>>> test
>>>>>>>>>>>> the classlibrary code.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> Well you need a working C compiler and standard C library to
>>>>>>>>>>> compile the
>>>>>>>>>>> compiler so you can compile make so you can build bash so you can
>>>>>>>>>>> run
>>>>>>>>>>> perl (which uses the standard C library functions all over the
>>>>>>>>>>> place of
>>>>>>>>>>> course) so you can run the standard C library tests so that
>>>>>>>>>>> you know
>>>>>>>>>>> that
>>>>>>>>>>> the library you used when compiling the compiler were correct so
>>>>>>>>>>> you can
>>>>>>>>>>> run the compiler tests. I don't think they actually do things
>>>>>>>>>>> that
>>>>>>>>>>> way, but
>>>>>>>>>>> it seems like basically the same problem. Having a virtual
>>>>>>>>>>> machine just
>>>>>>>>>>> makes it easier since you still assume "the native world" as a
>>>>>>>>>>> baseline,
>>>>>>>>>>> which is a lot more than "the hardware".
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> There's a difference. You can use a completely separate
>>>>>>>>>> toolchain to
>>>>>>>>>> build, test and verify the output of the C compiler.
>>>>>>>>>>
>>>>>>>>>> In our case, we are using the thing we are testing to test itself.
>>>>>>>>>> There is no "known good" element possible right now.
>>>>>>>>>>
>>>>>>>>>> We use the classlibrary we are trying to test to execute the test
>>>>>>>>>> framework that tests the classlibrary that is running it.
>>>>>>>>>>
>>>>>>>>>> The tool is testing itself.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>>> So I think there are three things we want to do (adopting the
>>>>>>>>>>>>>> terminology that came from the discussion with Tim and Leo ) :
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1) implementation tests
>>>>>>>>>>>>>> 2) spec/API tests (I'll bundle together)
>>>>>>>>>>>>>> 3) integration/functional tests
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I believe that for #1, the issues related to being on the
>>>>>>>>>>>>>> bootclasspath don't matter, because we aren't testing that
>>>>>>>>>>>>>> aspect
>>>>>>>>>>>>>> of the classes (which is how they behave integrated w/ the
>>>>>>>>>>>>>> VM and
>>>>>>>>>>>>>> security system) but rather the basic internal functioning.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm not sure how to approach this, but I'll try. I'd love to
>>>>>>>>>>>>>> hear
>>>>>>>>>>>>>> how Sun, IBM or BEA deals with this, or be told why it
>>>>>>>>>>>>>> isn't an
>>>>>>>>>>>>>> issue :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Implementation tests : I'd like to see us be able to do #1 via
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> standard same-package technique (i.e. testing a.b.C w/
>>>>>>>>>>>>>> a.b.CTest)
>>>>>>>>>>>>>> but we'll run into a tangle of classloader problems, I
>>>>>>>>>>>>>> suspect,
>>>>>>>>>>>>>> becuase we want to be testing java.* code in a system that
>>>>>>>>>>>>>> already
>>>>>>>>>>>>>> has java.* code. Can anyone see a way we can do this - test
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> classlibrary from the integration point of view - using
>>>>>>>>>>>>>> some test
>>>>>>>>>>>>>> harness + any known-good JRE, like Sun's or IBM's?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Ew, that won't work in the end since we should assume our
>>>>>>>>>>>>> own JRE
>>>>>>>>>>>>> is going
>>>>>>>>>>>>> to be "known-better" :-). But it might be a nice way to
>>>>>>>>>>>>> "bootstrap"
>>>>>>>>>>>>> (eg
>>>>>>>>>>>>> we test with an external JRE until we satisfy the tests and
>>>>>>>>>>>>> then we
>>>>>>>>>>>>> switch
>>>>>>>>>>>>> to testing with an earlier build).
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> Lets be clear - even using our own "earlier build" doesn't solve
>>>>>>>>>>>> the
>>>>>>>>>>>> problem I'm describing, because as it stands now, we don't use
>>>>>>>>>>>> "earlier build" classes to test with - we use the code we
>>>>>>>>>>>> want to
>>>>>>>>>>>> test as the clsaslibrary for the JRE that's running the test
>>>>>>>>>>>> framework.
>>>>>>>>>>>>
>>>>>>>>>>>> The classes that we are testing are also the classes used by the
>>>>>>>>>>>> testing framework. IOW, any of the java.* classes that JUnit
>>>>>>>>>>>> itself
>>>>>>>>>>>> needs (ex. java.util.HashMap) are exactly the same
>>>>>>>>>>>> implementation
>>>>>>>>>>>> that it's testing.
>>>>>>>>>>>>
>>>>>>>>>>>> That's why I think it's subtly different than a "bootstrap
>>>>>>>>>>>> and use
>>>>>>>>>>>> version - 1 to test" problem. See what I mean?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> Yeah yeah, I was already way beyond thinking "just" JUnit is
>>>>>>>>>>> usable
>>>>>>>>>>> for the
>>>>>>>>>>> kind of test you're describing. At some point, fundamentally, you
>>>>>>>>>>> either trust
>>>>>>>>>>> something external (whether its the sun jdk or the intel compiler
>>>>>>>>>>> designers,
>>>>>>>>>>> at some point you do draw a line) or you find a way to bootstrap.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Well, we do trust the Sun JDK.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> I'm very open to the idea that I'm missing something here,
>>>>>>>>>>>> but I'd
>>>>>>>>>>>> like to know that you see the issue - that when we test, we have
>>>>>>>>>>>>
>>>>>>>>>>>> VM + "classlib to be tested" + JUnit + testcases
>>>>>>>>>>>>
>>>>>>>>>>>> where the testcases are testing the classlib the VM is running
>>>>>>>>>>>> JUnit
>>>>>>>>>>>> with.
>>>>>>>>>>>>
>>>>>>>>>>>> There never is isolation of the code being tested :
>>>>>>>>>>>>
>>>>>>>>>>>> VM + "known good classlib" + Junit + testcases
>>>>>>>>>>>>
>>>>>>>>>>>> unless we have some framework where
>>>>>>>>>>>>
>>>>>>>>>>>> VM + "known good classlib" + JUnit
>>>>>>>>>>>> + framework("classlib to be tested")
>>>>>>>>>>>> + testcases
>>>>>>>>>>>>
>>>>>>>>>>>> and it's that notion of "framework()" that I'm advocating we
>>>>>>>>>>>> explore.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> I'm all for exploring it, I just fundamentally don't buy into the
>>>>>>>>>>> "known
>>>>>>>>>>> good" bit. What happens when the 'classlib to be tested' is
>>>>>>>>>>> 'known
>>>>>>>>>>> better' than the 'known good' one? How do you define "known"? How
>>>>>>>>>>> do you
>>>>>>>>>>> define "good"?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Known? Passed some set of tests. So it could be the Sun JDK
>>>>>>>>>> for the
>>>>>>>>>> VM + "known good" part.
>>>>>>>>>>
>>>>>>>>>> I think you intuitively understand this. When you find a bug
>>>>>>>>>> in code
>>>>>>>>>> you are testing, you first assume it's your code, not the
>>>>>>>>>> framework,
>>>>>>>>>> right? In our case, our framework is actually the code we are
>>>>>>>>>> testing, so we have a bit of a logical conundrum.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Hi Geir,
>>>>>>>>>
>>>>>>>>> The number of Harmony public API classes that get loaded just to
>>>>>>>>> run the
>>>>>>>>> JUnit harness is a little over 200. The majority of these are
>>>>>>>>> out of
>>>>>>>>> LUNI with a very low number coming from each of Security, NIO,
>>>>>>>>> Archive
>>>>>>>>> and Text.
>>>>>>>>>
>>>>>>>>> Sure there is a circular dependency between what we are building
>>>>>>>>> and the
>>>>>>>>> framework we are using to test it but it appears to touch on only a
>>>>>>>>> relatively small part of Harmony....IMHO.
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>> George
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>> Further ideas...
>>>>>>>>>>>>>
>>>>>>>>>>>>> -> look at how the native world does testing
>>>>>>>>>>>>> (hint: it usually has #ifdefs, uses perl along the way, and
>>>>>>>>>>>>> it is
>>>>>>>>>>>>> certainly
>>>>>>>>>>>>> "messy")
>>>>>>>>>>>>> -> emulate that
>>>>>>>>>>>>>
>>>>>>>>>>>>> -> build a bigger, better specification test
>>>>>>>>>>>>> -> and somehow "prove" it is "good enough"
>>>>>>>>>>>>>
>>>>>>>>>>>>> -> build a bigger, better integration test
>>>>>>>>>>>>> -> and somehow "prove" it is "good enough"
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'll admit my primary interest is the last one...
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> The problem I see with the last one is that the "parameter
>>>>>>>>>>>> space" is
>>>>>>>>>>>> *huge*.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> Yeah, that's one of the things that makes it interesting.
>>>>>>>>>>> Fortunately
>>>>>>>>>>> open source does have many monkeys...
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> I believe that your preference for the last one comes from the
>>>>>>>>>>>> Monte-Carlo style approach that Gump uses - hope that your test
>>>>>>>>>>>> suite has enough variance that you "push" the thing being tested
>>>>>>>>>>>> through enough of the parameter space that you can be
>>>>>>>>>>>> comfortable
>>>>>>>>>>>> you would have exposed the bugs. Maybe.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> Ooh, now its becoming rather abstract...
>>>>>>>>>>>
>>>>>>>>>>> Well, perhaps, but more of the gump approache comes from the idea
>>>>>>>>>>> that
>>>>>>>>>>> the parameter space itself is also at some point defined in
>>>>>>>>>>> software,
>>>>>>>>>>> which may have bugs of its own. You circumvent that by making
>>>>>>>>>>> humans the
>>>>>>>>>>> parameter space (don't start about how humans are buggy. We don't
>>>>>>>>>>> want to
>>>>>>>>>>> get into existialism or faith systems when talking about unit
>>>>>>>>>>> testing do
>>>>>>>>>>> we?). The thing that gump enables is "many monkey QA" - a way for
>>>>>>>>>>> thousands
>>>>>>>>>>> of human beings to concurrently make shared assertions about
>>>>>>>>>>> software
>>>>>>>>>>> without actually needing all that much human interaction.
>>>>>>>>>>>
>>>>>>>>>>> More concretely, if harmony can run all known java software,
>>>>>>>>>>> and run
>>>>>>>>>>> it to
>>>>>>>>>>> the asserted satisfaction of all its developers, you can trust
>>>>>>>>>>> that
>>>>>>>>>>> you have
>>>>>>>>>>> covered all the /relevant/ parts of the parameter space you
>>>>>>>>>>> describe.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Yes. And when you can run all knownn Java software, let me
>>>>>>>>>> know :)
>>>>>>>>>> That's my point about the parameter space being huge. Even
>>>>>>>>>> when you
>>>>>>>>>> reduce the definition to "that of all known Java software", you
>>>>>>>>>> still
>>>>>>>>>> have a huge problem on your hands.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> You
>>>>>>>>>>> will never get that level of trust when the assertions are
>>>>>>>>>>> made by
>>>>>>>>>>> software
>>>>>>>>>>> rather than humans. This is how open source leads to software
>>>>>>>>>>> quality.
>>>>>>>>>>>
>>>>>>>>>>> Quoting myself, 'gump is the most misunderstood piece of
>>>>>>>>>>> software,
>>>>>>>>>>> ever'.
>>>>>>>>>>>
>>>>>>>>>>> cheers,
>>>>>>>>>>>
>>>>>>>>>>> Leo
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>
>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>>> For additional commands, e-mail:
>>>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>>> For additional commands, e-mail:
>>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> --
>>>>> Paulex Yang
>>>>> China Software Development Lab
>>>>> IBM
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>
>>>>
>>>>
>>>>
>>>
>> --
>> Paulex Yang
>> China Software Development Lab
>> IBM
>>
>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
>
--
Paulex Yang
China Software Development Lab
IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Mikhail Loenko <ml...@gmail.com>.
Seems like implemantation-specific tests for java.security package do not
fit well into the proposed solution
I think we do not have to stick to requirement about
organization name in the tests: we already have tests in package java.*
why don't have package tests.*?
Thanks,
Mikhail
2006/4/26, Paulex Yang <pa...@gmail.com>:
> Richard Liang wrote:
> > Mikhail Loenko wrote:
> >> how about 'specific'? impl seems to be not very informative.
> >>
> >>
> > +1 to Oliver's impl :-)
> >> I have a concern abou proposed package naming guidelines:
> >> package name
> >> org.apache.harmony.security.tests.org.apache.harmony.security
> >> is not much better then 1000-character long test name.
> >>
> >>
> > +1. I think the prefix org.apache.harmony.security is unnecessary.
> > "tests.impl.org.apache.harmony.security" is enough to tell people what
> > the test cases belong to
> I agree it is duplicated and unnecessary, but it is a general convention
> for Java application to start with organization name, i.e.,
> "org.apache.harmony". If I understand correctly, our implementation
> package's name must start with "org.apache.harmony.<module name>", so
> how about this:
>
> org.apache.harmony.<module name>.tests.impl.<rest of package name>
>
> as a example, test for org.apache.harmony.security.util will be named as:
> org.apache.harmony.security.tests.impl.util
>
> while the API test still be named as
> org.apache.harmony.security.tests.api.java.security
>
> comments?
> >
> > Any comments? Thanks a lot.
> >> Thanks,
> >> Mikhail
> >>
> >>
> >> 2006/4/26, Paulex Yang <pa...@gmail.com>:
> >>
> >>> Oliver Deakin wrote:
> >>>
> >>>> George Harley wrote:
> >>>>
> >>>>> Mikhail Loenko wrote:
> >>>>>
> >>>>>> Hello
> >>>>>>
> >>>>>> I'd like to bring this thread back.
> >>>>>>
> >>>>>> Number of tests is growing and it is time to put them in order.
> >>>>>>
> >>>>>> So far we may have:
> >>>>>>
> >>>>>> 1) implementation-specific tests that designed to be run from
> >>>>>> bootclasspath
> >>>>>> 2) implementation-specific tests that might be run from classpath
> >>>>>> 3) implementation-specific tests that designed to be run from
> >>>>>> classpath
> >>>>>> 4) implementation-independent tests that designed to be run from
> >>>>>> bootclasspath
> >>>>>> 5) implementation-independent tests that might be run from classpath
> >>>>>> 6) implementation-independent tests that designed to be run from
> >>>>>> classpath
> >>>>>>
> >>>>>> Also we seem to have the following packages, where the tests are:
> >>>>>>
> >>>>>> 1) the same package as implementation
> >>>>>> 2) org.apache.harmony.tests.[the same package as implementation]
> >>>>>> 3) tests.api.[the same package as implementation]
> >>>>>>
> >>>>>> I suggest that we work out step-by-step solution as we could not
> >>>>>> reach
> >>>>>> an agreement for the general universal one
> >>>>>>
> >>>>>> So for the first step I suggest that we separate i-independent tests
> >>>>>> that must or may be run from classpath
> >>>>>>
> >>>>>> I suggest that we put them into package
> >>>>>> tests.module.compatible.[package of implementation being tested]
> >>>>>>
> >>>>>> Comments?
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Mikhail
> >>>>>>
> >>>>>>
> >>>>> Hi Mikhail,
> >>>>>
> >>>>> I've just started working through the modules to merge test packages
> >>>>> "tests.api.[same package as implementation]" and "tests.api.[same
> >>>>> package as implementation]" into one package space. Using the class
> >>>>> library package naming guidelines from off the web site [1], all of
> >>>>> the tests for the text module have been consolidated under
> >>>>> org.apache.harmony.text.tests.[package under test].
> >>>>>
> >>>>> Of course, the text module has only "implementation-independent tests
> >>>>> that designed to be run from classpath". For modules that have got
> >>>>> implementation-specific tests then I suppose we could use something
> >>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
> >>>>> "org.apache.harmony.[module].tests.internal.[package under test]"
> >>>>> etc. I've got no preference.
> >>>>>
> >>>> I think impl is preferable over internal here, as we already use
> >>>> internal in our implementation package names to indicate classes
> >>>> totally internal to that bundle. To also use internal to label tests
> >>>> that are implementation specific may cause confusion.
> >>>>
> >>>>
> >>> +1 from me.
> >>>
> >>>> Regards,
> >>>> Oliver
> >>>>
> >>>>
> >>>>> Best regards,
> >>>>> George
> >>>>>
> >>>>>
> >>>>> [1]
> >>>>> http://incubator.apache.org/harmony/subcomponents/classlibrary/pkgnaming.html
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>> 2006/3/24, George Harley <ge...@googlemail.com>:
> >>>>>>
> >>>>>>
> >>>>>>> Geir Magnusson Jr wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>> Leo Simons wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Wed, Mar 22, 2006 at 08:02:44AM -0500, Geir Magnusson Jr
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> Leo Simons wrote:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> On Wed, Mar 22, 2006 at 07:15:28AM -0500, Geir Magnusson Jr
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> Pulling out of the various threads where we have been
> >>>>>>>>>>>> discussing,
> >>>>>>>>>>>> can we agree on the problem :
> >>>>>>>>>>>>
> >>>>>>>>>>>> We have unique problems compared to other Java projects
> >>>>>>>>>>>> because we
> >>>>>>>>>>>> need to find a way to reliably test the things that are
> >>>>>>>>>>>> commonly
> >>>>>>>>>>>> expected to be a solid point of reference - namely the core
> >>>>>>>>>>>> class
> >>>>>>>>>>>> library.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Further, we've been implicitly doing "integration testing"
> >>>>>>>>>>>> because
> >>>>>>>>>>>> - so far - the only way we've been testing our code has
> >>>>>>>>>>>> been 'in
> >>>>>>>>>>>> situ' in the VM - not in an isolated test harness. To me,
> >>>>>>>>>>>> this
> >>>>>>>>>>>> turns it into an integration test.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Sure, we're using JUnit, but because of the fact we are
> >>>>>>>>>>>> implmenting core java.* APIs, we aren't testing with a
> >>>>>>>>>>>> framework
> >>>>>>>>>>>> that has been independently tested for correctness, like we
> >>>>>>>>>>>> would
> >>>>>>>>>>>> when testing any other code.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I hope I got that idea across - I believe that we have to go
> >>>>>>>>>>>> beyond normal testing approaches because we don't have a
> >>>>>>>>>>>> normal
> >>>>>>>>>>>> situation.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>> Where we define 'normal situation' as "running a test
> >>>>>>>>>>> framework on
> >>>>>>>>>>> top of
> >>>>>>>>>>> the sun jdk and expecting any bugs to not be in that jdk".
> >>>>>>>>>>> There's
> >>>>>>>>>>> plenty
> >>>>>>>>>>> of projects out there that have to test things without having
> >>>>>>>>>>> such a
> >>>>>>>>>>> "stable reference JDK" luxury.....I imagine that testing GCC is
> >>>>>>>>>>> just as
> >>>>>>>>>>> hard as this problem we have here :-)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> Is it the same? We need to have a running JVM+classlibarary to
> >>>>>>>>>> test
> >>>>>>>>>> the classlibrary code.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> Well you need a working C compiler and standard C library to
> >>>>>>>>> compile the
> >>>>>>>>> compiler so you can compile make so you can build bash so you can
> >>>>>>>>> run
> >>>>>>>>> perl (which uses the standard C library functions all over the
> >>>>>>>>> place of
> >>>>>>>>> course) so you can run the standard C library tests so that
> >>>>>>>>> you know
> >>>>>>>>> that
> >>>>>>>>> the library you used when compiling the compiler were correct so
> >>>>>>>>> you can
> >>>>>>>>> run the compiler tests. I don't think they actually do things
> >>>>>>>>> that
> >>>>>>>>> way, but
> >>>>>>>>> it seems like basically the same problem. Having a virtual
> >>>>>>>>> machine just
> >>>>>>>>> makes it easier since you still assume "the native world" as a
> >>>>>>>>> baseline,
> >>>>>>>>> which is a lot more than "the hardware".
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> There's a difference. You can use a completely separate
> >>>>>>>> toolchain to
> >>>>>>>> build, test and verify the output of the C compiler.
> >>>>>>>>
> >>>>>>>> In our case, we are using the thing we are testing to test itself.
> >>>>>>>> There is no "known good" element possible right now.
> >>>>>>>>
> >>>>>>>> We use the classlibrary we are trying to test to execute the test
> >>>>>>>> framework that tests the classlibrary that is running it.
> >>>>>>>>
> >>>>>>>> The tool is testing itself.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>>>> So I think there are three things we want to do (adopting the
> >>>>>>>>>>>> terminology that came from the discussion with Tim and Leo ) :
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1) implementation tests
> >>>>>>>>>>>> 2) spec/API tests (I'll bundle together)
> >>>>>>>>>>>> 3) integration/functional tests
> >>>>>>>>>>>>
> >>>>>>>>>>>> I believe that for #1, the issues related to being on the
> >>>>>>>>>>>> bootclasspath don't matter, because we aren't testing that
> >>>>>>>>>>>> aspect
> >>>>>>>>>>>> of the classes (which is how they behave integrated w/ the
> >>>>>>>>>>>> VM and
> >>>>>>>>>>>> security system) but rather the basic internal functioning.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'm not sure how to approach this, but I'll try. I'd love to
> >>>>>>>>>>>> hear
> >>>>>>>>>>>> how Sun, IBM or BEA deals with this, or be told why it
> >>>>>>>>>>>> isn't an
> >>>>>>>>>>>> issue :)
> >>>>>>>>>>>>
> >>>>>>>>>>>> Implementation tests : I'd like to see us be able to do #1 via
> >>>>>>>>>>>> the
> >>>>>>>>>>>> standard same-package technique (i.e. testing a.b.C w/
> >>>>>>>>>>>> a.b.CTest)
> >>>>>>>>>>>> but we'll run into a tangle of classloader problems, I
> >>>>>>>>>>>> suspect,
> >>>>>>>>>>>> becuase we want to be testing java.* code in a system that
> >>>>>>>>>>>> already
> >>>>>>>>>>>> has java.* code. Can anyone see a way we can do this - test
> >>>>>>>>>>>> the
> >>>>>>>>>>>> classlibrary from the integration point of view - using
> >>>>>>>>>>>> some test
> >>>>>>>>>>>> harness + any known-good JRE, like Sun's or IBM's?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>> Ew, that won't work in the end since we should assume our
> >>>>>>>>>>> own JRE
> >>>>>>>>>>> is going
> >>>>>>>>>>> to be "known-better" :-). But it might be a nice way to
> >>>>>>>>>>> "bootstrap"
> >>>>>>>>>>> (eg
> >>>>>>>>>>> we test with an external JRE until we satisfy the tests and
> >>>>>>>>>>> then we
> >>>>>>>>>>> switch
> >>>>>>>>>>> to testing with an earlier build).
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> Lets be clear - even using our own "earlier build" doesn't solve
> >>>>>>>>>> the
> >>>>>>>>>> problem I'm describing, because as it stands now, we don't use
> >>>>>>>>>> "earlier build" classes to test with - we use the code we
> >>>>>>>>>> want to
> >>>>>>>>>> test as the clsaslibrary for the JRE that's running the test
> >>>>>>>>>> framework.
> >>>>>>>>>>
> >>>>>>>>>> The classes that we are testing are also the classes used by the
> >>>>>>>>>> testing framework. IOW, any of the java.* classes that JUnit
> >>>>>>>>>> itself
> >>>>>>>>>> needs (ex. java.util.HashMap) are exactly the same
> >>>>>>>>>> implementation
> >>>>>>>>>> that it's testing.
> >>>>>>>>>>
> >>>>>>>>>> That's why I think it's subtly different than a "bootstrap
> >>>>>>>>>> and use
> >>>>>>>>>> version - 1 to test" problem. See what I mean?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> Yeah yeah, I was already way beyond thinking "just" JUnit is
> >>>>>>>>> usable
> >>>>>>>>> for the
> >>>>>>>>> kind of test you're describing. At some point, fundamentally, you
> >>>>>>>>> either trust
> >>>>>>>>> something external (whether its the sun jdk or the intel compiler
> >>>>>>>>> designers,
> >>>>>>>>> at some point you do draw a line) or you find a way to bootstrap.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> Well, we do trust the Sun JDK.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>> I'm very open to the idea that I'm missing something here,
> >>>>>>>>>> but I'd
> >>>>>>>>>> like to know that you see the issue - that when we test, we have
> >>>>>>>>>>
> >>>>>>>>>> VM + "classlib to be tested" + JUnit + testcases
> >>>>>>>>>>
> >>>>>>>>>> where the testcases are testing the classlib the VM is running
> >>>>>>>>>> JUnit
> >>>>>>>>>> with.
> >>>>>>>>>>
> >>>>>>>>>> There never is isolation of the code being tested :
> >>>>>>>>>>
> >>>>>>>>>> VM + "known good classlib" + Junit + testcases
> >>>>>>>>>>
> >>>>>>>>>> unless we have some framework where
> >>>>>>>>>>
> >>>>>>>>>> VM + "known good classlib" + JUnit
> >>>>>>>>>> + framework("classlib to be tested")
> >>>>>>>>>> + testcases
> >>>>>>>>>>
> >>>>>>>>>> and it's that notion of "framework()" that I'm advocating we
> >>>>>>>>>> explore.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> I'm all for exploring it, I just fundamentally don't buy into the
> >>>>>>>>> "known
> >>>>>>>>> good" bit. What happens when the 'classlib to be tested' is
> >>>>>>>>> 'known
> >>>>>>>>> better' than the 'known good' one? How do you define "known"? How
> >>>>>>>>> do you
> >>>>>>>>> define "good"?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> Known? Passed some set of tests. So it could be the Sun JDK
> >>>>>>>> for the
> >>>>>>>> VM + "known good" part.
> >>>>>>>>
> >>>>>>>> I think you intuitively understand this. When you find a bug
> >>>>>>>> in code
> >>>>>>>> you are testing, you first assume it's your code, not the
> >>>>>>>> framework,
> >>>>>>>> right? In our case, our framework is actually the code we are
> >>>>>>>> testing, so we have a bit of a logical conundrum.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>> Hi Geir,
> >>>>>>>
> >>>>>>> The number of Harmony public API classes that get loaded just to
> >>>>>>> run the
> >>>>>>> JUnit harness is a little over 200. The majority of these are
> >>>>>>> out of
> >>>>>>> LUNI with a very low number coming from each of Security, NIO,
> >>>>>>> Archive
> >>>>>>> and Text.
> >>>>>>>
> >>>>>>> Sure there is a circular dependency between what we are building
> >>>>>>> and the
> >>>>>>> framework we are using to test it but it appears to touch on only a
> >>>>>>> relatively small part of Harmony....IMHO.
> >>>>>>>
> >>>>>>> Best regards,
> >>>>>>> George
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>>>> Further ideas...
> >>>>>>>>>>>
> >>>>>>>>>>> -> look at how the native world does testing
> >>>>>>>>>>> (hint: it usually has #ifdefs, uses perl along the way, and
> >>>>>>>>>>> it is
> >>>>>>>>>>> certainly
> >>>>>>>>>>> "messy")
> >>>>>>>>>>> -> emulate that
> >>>>>>>>>>>
> >>>>>>>>>>> -> build a bigger, better specification test
> >>>>>>>>>>> -> and somehow "prove" it is "good enough"
> >>>>>>>>>>>
> >>>>>>>>>>> -> build a bigger, better integration test
> >>>>>>>>>>> -> and somehow "prove" it is "good enough"
> >>>>>>>>>>>
> >>>>>>>>>>> I'll admit my primary interest is the last one...
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> The problem I see with the last one is that the "parameter
> >>>>>>>>>> space" is
> >>>>>>>>>> *huge*.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> Yeah, that's one of the things that makes it interesting.
> >>>>>>>>> Fortunately
> >>>>>>>>> open source does have many monkeys...
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> I believe that your preference for the last one comes from the
> >>>>>>>>>> Monte-Carlo style approach that Gump uses - hope that your test
> >>>>>>>>>> suite has enough variance that you "push" the thing being tested
> >>>>>>>>>> through enough of the parameter space that you can be
> >>>>>>>>>> comfortable
> >>>>>>>>>> you would have exposed the bugs. Maybe.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> Ooh, now its becoming rather abstract...
> >>>>>>>>>
> >>>>>>>>> Well, perhaps, but more of the gump approache comes from the idea
> >>>>>>>>> that
> >>>>>>>>> the parameter space itself is also at some point defined in
> >>>>>>>>> software,
> >>>>>>>>> which may have bugs of its own. You circumvent that by making
> >>>>>>>>> humans the
> >>>>>>>>> parameter space (don't start about how humans are buggy. We don't
> >>>>>>>>> want to
> >>>>>>>>> get into existialism or faith systems when talking about unit
> >>>>>>>>> testing do
> >>>>>>>>> we?). The thing that gump enables is "many monkey QA" - a way for
> >>>>>>>>> thousands
> >>>>>>>>> of human beings to concurrently make shared assertions about
> >>>>>>>>> software
> >>>>>>>>> without actually needing all that much human interaction.
> >>>>>>>>>
> >>>>>>>>> More concretely, if harmony can run all known java software,
> >>>>>>>>> and run
> >>>>>>>>> it to
> >>>>>>>>> the asserted satisfaction of all its developers, you can trust
> >>>>>>>>> that
> >>>>>>>>> you have
> >>>>>>>>> covered all the /relevant/ parts of the parameter space you
> >>>>>>>>> describe.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> Yes. And when you can run all knownn Java software, let me
> >>>>>>>> know :)
> >>>>>>>> That's my point about the parameter space being huge. Even
> >>>>>>>> when you
> >>>>>>>> reduce the definition to "that of all known Java software", you
> >>>>>>>> still
> >>>>>>>> have a huge problem on your hands.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> You
> >>>>>>>>> will never get that level of trust when the assertions are
> >>>>>>>>> made by
> >>>>>>>>> software
> >>>>>>>>> rather than humans. This is how open source leads to software
> >>>>>>>>> quality.
> >>>>>>>>>
> >>>>>>>>> Quoting myself, 'gump is the most misunderstood piece of
> >>>>>>>>> software,
> >>>>>>>>> ever'.
> >>>>>>>>>
> >>>>>>>>> cheers,
> >>>>>>>>>
> >>>>>>>>> Leo
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>> ---------------------------------------------------------------------
> >>>>>>
> >>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>>> For additional commands, e-mail:
> >>>>>> harmony-dev-help@incubator.apache.org
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>>>> For additional commands, e-mail:
> >>>>> harmony-dev-help@incubator.apache.org
> >>>>>
> >>>>>
> >>>>>
> >>> --
> >>> Paulex Yang
> >>> China Software Development Lab
> >>> IBM
> >>>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>
> >>>
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>
> >>
> >>
> >
> >
>
>
> --
> Paulex Yang
> China Software Development Lab
> IBM
>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Geir Magnusson Jr <ge...@pobox.com>.
Richard Liang wrote:
> Mikhail Loenko wrote:
>> how about 'specific'? impl seems to be not very informative.
>>
>>
> +1 to Oliver's impl :-)
>> I have a concern abou proposed package naming guidelines:
>> package name
>> org.apache.harmony.security.tests.org.apache.harmony.security
>> is not much better then 1000-character long test name.
>>
>>
> +1. I think the prefix org.apache.harmony.security is unnecessary.
> "tests.impl.org.apache.harmony.security" is enough to tell people what
> the test cases belong to
>
> Any comments? Thanks a lot.
Our package namespace starts with "org.apache.harmony".
geir
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Paulex Yang <pa...@gmail.com>.
Richard Liang wrote:
> Mikhail Loenko wrote:
>> how about 'specific'? impl seems to be not very informative.
>>
>>
> +1 to Oliver's impl :-)
>> I have a concern abou proposed package naming guidelines:
>> package name
>> org.apache.harmony.security.tests.org.apache.harmony.security
>> is not much better then 1000-character long test name.
>>
>>
> +1. I think the prefix org.apache.harmony.security is unnecessary.
> "tests.impl.org.apache.harmony.security" is enough to tell people what
> the test cases belong to
I agree it is duplicated and unnecessary, but it is a general convention
for Java application to start with organization name, i.e.,
"org.apache.harmony". If I understand correctly, our implementation
package's name must start with "org.apache.harmony.<module name>", so
how about this:
org.apache.harmony.<module name>.tests.impl.<rest of package name>
as a example, test for org.apache.harmony.security.util will be named as:
org.apache.harmony.security.tests.impl.util
while the API test still be named as
org.apache.harmony.security.tests.api.java.security
comments?
>
> Any comments? Thanks a lot.
>> Thanks,
>> Mikhail
>>
>>
>> 2006/4/26, Paulex Yang <pa...@gmail.com>:
>>
>>> Oliver Deakin wrote:
>>>
>>>> George Harley wrote:
>>>>
>>>>> Mikhail Loenko wrote:
>>>>>
>>>>>> Hello
>>>>>>
>>>>>> I'd like to bring this thread back.
>>>>>>
>>>>>> Number of tests is growing and it is time to put them in order.
>>>>>>
>>>>>> So far we may have:
>>>>>>
>>>>>> 1) implementation-specific tests that designed to be run from
>>>>>> bootclasspath
>>>>>> 2) implementation-specific tests that might be run from classpath
>>>>>> 3) implementation-specific tests that designed to be run from
>>>>>> classpath
>>>>>> 4) implementation-independent tests that designed to be run from
>>>>>> bootclasspath
>>>>>> 5) implementation-independent tests that might be run from classpath
>>>>>> 6) implementation-independent tests that designed to be run from
>>>>>> classpath
>>>>>>
>>>>>> Also we seem to have the following packages, where the tests are:
>>>>>>
>>>>>> 1) the same package as implementation
>>>>>> 2) org.apache.harmony.tests.[the same package as implementation]
>>>>>> 3) tests.api.[the same package as implementation]
>>>>>>
>>>>>> I suggest that we work out step-by-step solution as we could not
>>>>>> reach
>>>>>> an agreement for the general universal one
>>>>>>
>>>>>> So for the first step I suggest that we separate i-independent tests
>>>>>> that must or may be run from classpath
>>>>>>
>>>>>> I suggest that we put them into package
>>>>>> tests.module.compatible.[package of implementation being tested]
>>>>>>
>>>>>> Comments?
>>>>>>
>>>>>> Thanks,
>>>>>> Mikhail
>>>>>>
>>>>>>
>>>>> Hi Mikhail,
>>>>>
>>>>> I've just started working through the modules to merge test packages
>>>>> "tests.api.[same package as implementation]" and "tests.api.[same
>>>>> package as implementation]" into one package space. Using the class
>>>>> library package naming guidelines from off the web site [1], all of
>>>>> the tests for the text module have been consolidated under
>>>>> org.apache.harmony.text.tests.[package under test].
>>>>>
>>>>> Of course, the text module has only "implementation-independent tests
>>>>> that designed to be run from classpath". For modules that have got
>>>>> implementation-specific tests then I suppose we could use something
>>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
>>>>> "org.apache.harmony.[module].tests.internal.[package under test]"
>>>>> etc. I've got no preference.
>>>>>
>>>> I think impl is preferable over internal here, as we already use
>>>> internal in our implementation package names to indicate classes
>>>> totally internal to that bundle. To also use internal to label tests
>>>> that are implementation specific may cause confusion.
>>>>
>>>>
>>> +1 from me.
>>>
>>>> Regards,
>>>> Oliver
>>>>
>>>>
>>>>> Best regards,
>>>>> George
>>>>>
>>>>>
>>>>> [1]
>>>>> http://incubator.apache.org/harmony/subcomponents/classlibrary/pkgnaming.html
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> 2006/3/24, George Harley <ge...@googlemail.com>:
>>>>>>
>>>>>>
>>>>>>> Geir Magnusson Jr wrote:
>>>>>>>
>>>>>>>
>>>>>>>> Leo Simons wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Wed, Mar 22, 2006 at 08:02:44AM -0500, Geir Magnusson Jr
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Leo Simons wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On Wed, Mar 22, 2006 at 07:15:28AM -0500, Geir Magnusson Jr
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Pulling out of the various threads where we have been
>>>>>>>>>>>> discussing,
>>>>>>>>>>>> can we agree on the problem :
>>>>>>>>>>>>
>>>>>>>>>>>> We have unique problems compared to other Java projects
>>>>>>>>>>>> because we
>>>>>>>>>>>> need to find a way to reliably test the things that are
>>>>>>>>>>>> commonly
>>>>>>>>>>>> expected to be a solid point of reference - namely the core
>>>>>>>>>>>> class
>>>>>>>>>>>> library.
>>>>>>>>>>>>
>>>>>>>>>>>> Further, we've been implicitly doing "integration testing"
>>>>>>>>>>>> because
>>>>>>>>>>>> - so far - the only way we've been testing our code has
>>>>>>>>>>>> been 'in
>>>>>>>>>>>> situ' in the VM - not in an isolated test harness. To me,
>>>>>>>>>>>> this
>>>>>>>>>>>> turns it into an integration test.
>>>>>>>>>>>>
>>>>>>>>>>>> Sure, we're using JUnit, but because of the fact we are
>>>>>>>>>>>> implmenting core java.* APIs, we aren't testing with a
>>>>>>>>>>>> framework
>>>>>>>>>>>> that has been independently tested for correctness, like we
>>>>>>>>>>>> would
>>>>>>>>>>>> when testing any other code.
>>>>>>>>>>>>
>>>>>>>>>>>> I hope I got that idea across - I believe that we have to go
>>>>>>>>>>>> beyond normal testing approaches because we don't have a
>>>>>>>>>>>> normal
>>>>>>>>>>>> situation.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> Where we define 'normal situation' as "running a test
>>>>>>>>>>> framework on
>>>>>>>>>>> top of
>>>>>>>>>>> the sun jdk and expecting any bugs to not be in that jdk".
>>>>>>>>>>> There's
>>>>>>>>>>> plenty
>>>>>>>>>>> of projects out there that have to test things without having
>>>>>>>>>>> such a
>>>>>>>>>>> "stable reference JDK" luxury.....I imagine that testing GCC is
>>>>>>>>>>> just as
>>>>>>>>>>> hard as this problem we have here :-)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Is it the same? We need to have a running JVM+classlibarary to
>>>>>>>>>> test
>>>>>>>>>> the classlibrary code.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Well you need a working C compiler and standard C library to
>>>>>>>>> compile the
>>>>>>>>> compiler so you can compile make so you can build bash so you can
>>>>>>>>> run
>>>>>>>>> perl (which uses the standard C library functions all over the
>>>>>>>>> place of
>>>>>>>>> course) so you can run the standard C library tests so that
>>>>>>>>> you know
>>>>>>>>> that
>>>>>>>>> the library you used when compiling the compiler were correct so
>>>>>>>>> you can
>>>>>>>>> run the compiler tests. I don't think they actually do things
>>>>>>>>> that
>>>>>>>>> way, but
>>>>>>>>> it seems like basically the same problem. Having a virtual
>>>>>>>>> machine just
>>>>>>>>> makes it easier since you still assume "the native world" as a
>>>>>>>>> baseline,
>>>>>>>>> which is a lot more than "the hardware".
>>>>>>>>>
>>>>>>>>>
>>>>>>>> There's a difference. You can use a completely separate
>>>>>>>> toolchain to
>>>>>>>> build, test and verify the output of the C compiler.
>>>>>>>>
>>>>>>>> In our case, we are using the thing we are testing to test itself.
>>>>>>>> There is no "known good" element possible right now.
>>>>>>>>
>>>>>>>> We use the classlibrary we are trying to test to execute the test
>>>>>>>> framework that tests the classlibrary that is running it.
>>>>>>>>
>>>>>>>> The tool is testing itself.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>>>> So I think there are three things we want to do (adopting the
>>>>>>>>>>>> terminology that came from the discussion with Tim and Leo ) :
>>>>>>>>>>>>
>>>>>>>>>>>> 1) implementation tests
>>>>>>>>>>>> 2) spec/API tests (I'll bundle together)
>>>>>>>>>>>> 3) integration/functional tests
>>>>>>>>>>>>
>>>>>>>>>>>> I believe that for #1, the issues related to being on the
>>>>>>>>>>>> bootclasspath don't matter, because we aren't testing that
>>>>>>>>>>>> aspect
>>>>>>>>>>>> of the classes (which is how they behave integrated w/ the
>>>>>>>>>>>> VM and
>>>>>>>>>>>> security system) but rather the basic internal functioning.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm not sure how to approach this, but I'll try. I'd love to
>>>>>>>>>>>> hear
>>>>>>>>>>>> how Sun, IBM or BEA deals with this, or be told why it
>>>>>>>>>>>> isn't an
>>>>>>>>>>>> issue :)
>>>>>>>>>>>>
>>>>>>>>>>>> Implementation tests : I'd like to see us be able to do #1 via
>>>>>>>>>>>> the
>>>>>>>>>>>> standard same-package technique (i.e. testing a.b.C w/
>>>>>>>>>>>> a.b.CTest)
>>>>>>>>>>>> but we'll run into a tangle of classloader problems, I
>>>>>>>>>>>> suspect,
>>>>>>>>>>>> becuase we want to be testing java.* code in a system that
>>>>>>>>>>>> already
>>>>>>>>>>>> has java.* code. Can anyone see a way we can do this - test
>>>>>>>>>>>> the
>>>>>>>>>>>> classlibrary from the integration point of view - using
>>>>>>>>>>>> some test
>>>>>>>>>>>> harness + any known-good JRE, like Sun's or IBM's?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> Ew, that won't work in the end since we should assume our
>>>>>>>>>>> own JRE
>>>>>>>>>>> is going
>>>>>>>>>>> to be "known-better" :-). But it might be a nice way to
>>>>>>>>>>> "bootstrap"
>>>>>>>>>>> (eg
>>>>>>>>>>> we test with an external JRE until we satisfy the tests and
>>>>>>>>>>> then we
>>>>>>>>>>> switch
>>>>>>>>>>> to testing with an earlier build).
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Lets be clear - even using our own "earlier build" doesn't solve
>>>>>>>>>> the
>>>>>>>>>> problem I'm describing, because as it stands now, we don't use
>>>>>>>>>> "earlier build" classes to test with - we use the code we
>>>>>>>>>> want to
>>>>>>>>>> test as the clsaslibrary for the JRE that's running the test
>>>>>>>>>> framework.
>>>>>>>>>>
>>>>>>>>>> The classes that we are testing are also the classes used by the
>>>>>>>>>> testing framework. IOW, any of the java.* classes that JUnit
>>>>>>>>>> itself
>>>>>>>>>> needs (ex. java.util.HashMap) are exactly the same
>>>>>>>>>> implementation
>>>>>>>>>> that it's testing.
>>>>>>>>>>
>>>>>>>>>> That's why I think it's subtly different than a "bootstrap
>>>>>>>>>> and use
>>>>>>>>>> version - 1 to test" problem. See what I mean?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Yeah yeah, I was already way beyond thinking "just" JUnit is
>>>>>>>>> usable
>>>>>>>>> for the
>>>>>>>>> kind of test you're describing. At some point, fundamentally, you
>>>>>>>>> either trust
>>>>>>>>> something external (whether its the sun jdk or the intel compiler
>>>>>>>>> designers,
>>>>>>>>> at some point you do draw a line) or you find a way to bootstrap.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Well, we do trust the Sun JDK.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>> I'm very open to the idea that I'm missing something here,
>>>>>>>>>> but I'd
>>>>>>>>>> like to know that you see the issue - that when we test, we have
>>>>>>>>>>
>>>>>>>>>> VM + "classlib to be tested" + JUnit + testcases
>>>>>>>>>>
>>>>>>>>>> where the testcases are testing the classlib the VM is running
>>>>>>>>>> JUnit
>>>>>>>>>> with.
>>>>>>>>>>
>>>>>>>>>> There never is isolation of the code being tested :
>>>>>>>>>>
>>>>>>>>>> VM + "known good classlib" + Junit + testcases
>>>>>>>>>>
>>>>>>>>>> unless we have some framework where
>>>>>>>>>>
>>>>>>>>>> VM + "known good classlib" + JUnit
>>>>>>>>>> + framework("classlib to be tested")
>>>>>>>>>> + testcases
>>>>>>>>>>
>>>>>>>>>> and it's that notion of "framework()" that I'm advocating we
>>>>>>>>>> explore.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> I'm all for exploring it, I just fundamentally don't buy into the
>>>>>>>>> "known
>>>>>>>>> good" bit. What happens when the 'classlib to be tested' is
>>>>>>>>> 'known
>>>>>>>>> better' than the 'known good' one? How do you define "known"? How
>>>>>>>>> do you
>>>>>>>>> define "good"?
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Known? Passed some set of tests. So it could be the Sun JDK
>>>>>>>> for the
>>>>>>>> VM + "known good" part.
>>>>>>>>
>>>>>>>> I think you intuitively understand this. When you find a bug
>>>>>>>> in code
>>>>>>>> you are testing, you first assume it's your code, not the
>>>>>>>> framework,
>>>>>>>> right? In our case, our framework is actually the code we are
>>>>>>>> testing, so we have a bit of a logical conundrum.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> Hi Geir,
>>>>>>>
>>>>>>> The number of Harmony public API classes that get loaded just to
>>>>>>> run the
>>>>>>> JUnit harness is a little over 200. The majority of these are
>>>>>>> out of
>>>>>>> LUNI with a very low number coming from each of Security, NIO,
>>>>>>> Archive
>>>>>>> and Text.
>>>>>>>
>>>>>>> Sure there is a circular dependency between what we are building
>>>>>>> and the
>>>>>>> framework we are using to test it but it appears to touch on only a
>>>>>>> relatively small part of Harmony....IMHO.
>>>>>>>
>>>>>>> Best regards,
>>>>>>> George
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>>>> Further ideas...
>>>>>>>>>>>
>>>>>>>>>>> -> look at how the native world does testing
>>>>>>>>>>> (hint: it usually has #ifdefs, uses perl along the way, and
>>>>>>>>>>> it is
>>>>>>>>>>> certainly
>>>>>>>>>>> "messy")
>>>>>>>>>>> -> emulate that
>>>>>>>>>>>
>>>>>>>>>>> -> build a bigger, better specification test
>>>>>>>>>>> -> and somehow "prove" it is "good enough"
>>>>>>>>>>>
>>>>>>>>>>> -> build a bigger, better integration test
>>>>>>>>>>> -> and somehow "prove" it is "good enough"
>>>>>>>>>>>
>>>>>>>>>>> I'll admit my primary interest is the last one...
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> The problem I see with the last one is that the "parameter
>>>>>>>>>> space" is
>>>>>>>>>> *huge*.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Yeah, that's one of the things that makes it interesting.
>>>>>>>>> Fortunately
>>>>>>>>> open source does have many monkeys...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I believe that your preference for the last one comes from the
>>>>>>>>>> Monte-Carlo style approach that Gump uses - hope that your test
>>>>>>>>>> suite has enough variance that you "push" the thing being tested
>>>>>>>>>> through enough of the parameter space that you can be
>>>>>>>>>> comfortable
>>>>>>>>>> you would have exposed the bugs. Maybe.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Ooh, now its becoming rather abstract...
>>>>>>>>>
>>>>>>>>> Well, perhaps, but more of the gump approache comes from the idea
>>>>>>>>> that
>>>>>>>>> the parameter space itself is also at some point defined in
>>>>>>>>> software,
>>>>>>>>> which may have bugs of its own. You circumvent that by making
>>>>>>>>> humans the
>>>>>>>>> parameter space (don't start about how humans are buggy. We don't
>>>>>>>>> want to
>>>>>>>>> get into existialism or faith systems when talking about unit
>>>>>>>>> testing do
>>>>>>>>> we?). The thing that gump enables is "many monkey QA" - a way for
>>>>>>>>> thousands
>>>>>>>>> of human beings to concurrently make shared assertions about
>>>>>>>>> software
>>>>>>>>> without actually needing all that much human interaction.
>>>>>>>>>
>>>>>>>>> More concretely, if harmony can run all known java software,
>>>>>>>>> and run
>>>>>>>>> it to
>>>>>>>>> the asserted satisfaction of all its developers, you can trust
>>>>>>>>> that
>>>>>>>>> you have
>>>>>>>>> covered all the /relevant/ parts of the parameter space you
>>>>>>>>> describe.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Yes. And when you can run all knownn Java software, let me
>>>>>>>> know :)
>>>>>>>> That's my point about the parameter space being huge. Even
>>>>>>>> when you
>>>>>>>> reduce the definition to "that of all known Java software", you
>>>>>>>> still
>>>>>>>> have a huge problem on your hands.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> You
>>>>>>>>> will never get that level of trust when the assertions are
>>>>>>>>> made by
>>>>>>>>> software
>>>>>>>>> rather than humans. This is how open source leads to software
>>>>>>>>> quality.
>>>>>>>>>
>>>>>>>>> Quoting myself, 'gump is the most misunderstood piece of
>>>>>>>>> software,
>>>>>>>>> ever'.
>>>>>>>>>
>>>>>>>>> cheers,
>>>>>>>>>
>>>>>>>>> Leo
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>>
>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail:
>>>>>> harmony-dev-help@incubator.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>> For additional commands, e-mail:
>>>>> harmony-dev-help@incubator.apache.org
>>>>>
>>>>>
>>>>>
>>> --
>>> Paulex Yang
>>> China Software Development Lab
>>> IBM
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>>
>
>
--
Paulex Yang
China Software Development Lab
IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Richard Liang <ri...@gmail.com>.
Mikhail Loenko wrote:
> how about 'specific'? impl seems to be not very informative.
>
>
+1 to Oliver's impl :-)
> I have a concern abou proposed package naming guidelines:
> package name
> org.apache.harmony.security.tests.org.apache.harmony.security
> is not much better then 1000-character long test name.
>
>
+1. I think the prefix org.apache.harmony.security is unnecessary.
"tests.impl.org.apache.harmony.security" is enough to tell people what
the test cases belong to
Any comments? Thanks a lot.
> Thanks,
> Mikhail
>
>
> 2006/4/26, Paulex Yang <pa...@gmail.com>:
>
>> Oliver Deakin wrote:
>>
>>> George Harley wrote:
>>>
>>>> Mikhail Loenko wrote:
>>>>
>>>>> Hello
>>>>>
>>>>> I'd like to bring this thread back.
>>>>>
>>>>> Number of tests is growing and it is time to put them in order.
>>>>>
>>>>> So far we may have:
>>>>>
>>>>> 1) implementation-specific tests that designed to be run from
>>>>> bootclasspath
>>>>> 2) implementation-specific tests that might be run from classpath
>>>>> 3) implementation-specific tests that designed to be run from classpath
>>>>> 4) implementation-independent tests that designed to be run from
>>>>> bootclasspath
>>>>> 5) implementation-independent tests that might be run from classpath
>>>>> 6) implementation-independent tests that designed to be run from
>>>>> classpath
>>>>>
>>>>> Also we seem to have the following packages, where the tests are:
>>>>>
>>>>> 1) the same package as implementation
>>>>> 2) org.apache.harmony.tests.[the same package as implementation]
>>>>> 3) tests.api.[the same package as implementation]
>>>>>
>>>>> I suggest that we work out step-by-step solution as we could not reach
>>>>> an agreement for the general universal one
>>>>>
>>>>> So for the first step I suggest that we separate i-independent tests
>>>>> that must or may be run from classpath
>>>>>
>>>>> I suggest that we put them into package
>>>>> tests.module.compatible.[package of implementation being tested]
>>>>>
>>>>> Comments?
>>>>>
>>>>> Thanks,
>>>>> Mikhail
>>>>>
>>>>>
>>>> Hi Mikhail,
>>>>
>>>> I've just started working through the modules to merge test packages
>>>> "tests.api.[same package as implementation]" and "tests.api.[same
>>>> package as implementation]" into one package space. Using the class
>>>> library package naming guidelines from off the web site [1], all of
>>>> the tests for the text module have been consolidated under
>>>> org.apache.harmony.text.tests.[package under test].
>>>>
>>>> Of course, the text module has only "implementation-independent tests
>>>> that designed to be run from classpath". For modules that have got
>>>> implementation-specific tests then I suppose we could use something
>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
>>>> "org.apache.harmony.[module].tests.internal.[package under test]"
>>>> etc. I've got no preference.
>>>>
>>> I think impl is preferable over internal here, as we already use
>>> internal in our implementation package names to indicate classes
>>> totally internal to that bundle. To also use internal to label tests
>>> that are implementation specific may cause confusion.
>>>
>>>
>> +1 from me.
>>
>>> Regards,
>>> Oliver
>>>
>>>
>>>> Best regards,
>>>> George
>>>>
>>>>
>>>> [1]
>>>> http://incubator.apache.org/harmony/subcomponents/classlibrary/pkgnaming.html
>>>>
>>>>
>>>>
>>>>
>>>>> 2006/3/24, George Harley <ge...@googlemail.com>:
>>>>>
>>>>>
>>>>>> Geir Magnusson Jr wrote:
>>>>>>
>>>>>>
>>>>>>> Leo Simons wrote:
>>>>>>>
>>>>>>>
>>>>>>>> On Wed, Mar 22, 2006 at 08:02:44AM -0500, Geir Magnusson Jr wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> Leo Simons wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Wed, Mar 22, 2006 at 07:15:28AM -0500, Geir Magnusson Jr wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Pulling out of the various threads where we have been discussing,
>>>>>>>>>>> can we agree on the problem :
>>>>>>>>>>>
>>>>>>>>>>> We have unique problems compared to other Java projects
>>>>>>>>>>> because we
>>>>>>>>>>> need to find a way to reliably test the things that are commonly
>>>>>>>>>>> expected to be a solid point of reference - namely the core class
>>>>>>>>>>> library.
>>>>>>>>>>>
>>>>>>>>>>> Further, we've been implicitly doing "integration testing"
>>>>>>>>>>> because
>>>>>>>>>>> - so far - the only way we've been testing our code has been 'in
>>>>>>>>>>> situ' in the VM - not in an isolated test harness. To me, this
>>>>>>>>>>> turns it into an integration test.
>>>>>>>>>>>
>>>>>>>>>>> Sure, we're using JUnit, but because of the fact we are
>>>>>>>>>>> implmenting core java.* APIs, we aren't testing with a framework
>>>>>>>>>>> that has been independently tested for correctness, like we would
>>>>>>>>>>> when testing any other code.
>>>>>>>>>>>
>>>>>>>>>>> I hope I got that idea across - I believe that we have to go
>>>>>>>>>>> beyond normal testing approaches because we don't have a normal
>>>>>>>>>>> situation.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Where we define 'normal situation' as "running a test framework on
>>>>>>>>>> top of
>>>>>>>>>> the sun jdk and expecting any bugs to not be in that jdk". There's
>>>>>>>>>> plenty
>>>>>>>>>> of projects out there that have to test things without having
>>>>>>>>>> such a
>>>>>>>>>> "stable reference JDK" luxury.....I imagine that testing GCC is
>>>>>>>>>> just as
>>>>>>>>>> hard as this problem we have here :-)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Is it the same? We need to have a running JVM+classlibarary to
>>>>>>>>> test
>>>>>>>>> the classlibrary code.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Well you need a working C compiler and standard C library to
>>>>>>>> compile the
>>>>>>>> compiler so you can compile make so you can build bash so you can
>>>>>>>> run
>>>>>>>> perl (which uses the standard C library functions all over the
>>>>>>>> place of
>>>>>>>> course) so you can run the standard C library tests so that you know
>>>>>>>> that
>>>>>>>> the library you used when compiling the compiler were correct so
>>>>>>>> you can
>>>>>>>> run the compiler tests. I don't think they actually do things that
>>>>>>>> way, but
>>>>>>>> it seems like basically the same problem. Having a virtual
>>>>>>>> machine just
>>>>>>>> makes it easier since you still assume "the native world" as a
>>>>>>>> baseline,
>>>>>>>> which is a lot more than "the hardware".
>>>>>>>>
>>>>>>>>
>>>>>>> There's a difference. You can use a completely separate toolchain to
>>>>>>> build, test and verify the output of the C compiler.
>>>>>>>
>>>>>>> In our case, we are using the thing we are testing to test itself.
>>>>>>> There is no "known good" element possible right now.
>>>>>>>
>>>>>>> We use the classlibrary we are trying to test to execute the test
>>>>>>> framework that tests the classlibrary that is running it.
>>>>>>>
>>>>>>> The tool is testing itself.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>>>> So I think there are three things we want to do (adopting the
>>>>>>>>>>> terminology that came from the discussion with Tim and Leo ) :
>>>>>>>>>>>
>>>>>>>>>>> 1) implementation tests
>>>>>>>>>>> 2) spec/API tests (I'll bundle together)
>>>>>>>>>>> 3) integration/functional tests
>>>>>>>>>>>
>>>>>>>>>>> I believe that for #1, the issues related to being on the
>>>>>>>>>>> bootclasspath don't matter, because we aren't testing that aspect
>>>>>>>>>>> of the classes (which is how they behave integrated w/ the VM and
>>>>>>>>>>> security system) but rather the basic internal functioning.
>>>>>>>>>>>
>>>>>>>>>>> I'm not sure how to approach this, but I'll try. I'd love to
>>>>>>>>>>> hear
>>>>>>>>>>> how Sun, IBM or BEA deals with this, or be told why it isn't an
>>>>>>>>>>> issue :)
>>>>>>>>>>>
>>>>>>>>>>> Implementation tests : I'd like to see us be able to do #1 via
>>>>>>>>>>> the
>>>>>>>>>>> standard same-package technique (i.e. testing a.b.C w/ a.b.CTest)
>>>>>>>>>>> but we'll run into a tangle of classloader problems, I suspect,
>>>>>>>>>>> becuase we want to be testing java.* code in a system that
>>>>>>>>>>> already
>>>>>>>>>>> has java.* code. Can anyone see a way we can do this - test the
>>>>>>>>>>> classlibrary from the integration point of view - using some test
>>>>>>>>>>> harness + any known-good JRE, like Sun's or IBM's?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Ew, that won't work in the end since we should assume our own JRE
>>>>>>>>>> is going
>>>>>>>>>> to be "known-better" :-). But it might be a nice way to
>>>>>>>>>> "bootstrap"
>>>>>>>>>> (eg
>>>>>>>>>> we test with an external JRE until we satisfy the tests and
>>>>>>>>>> then we
>>>>>>>>>> switch
>>>>>>>>>> to testing with an earlier build).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Lets be clear - even using our own "earlier build" doesn't solve
>>>>>>>>> the
>>>>>>>>> problem I'm describing, because as it stands now, we don't use
>>>>>>>>> "earlier build" classes to test with - we use the code we want to
>>>>>>>>> test as the clsaslibrary for the JRE that's running the test
>>>>>>>>> framework.
>>>>>>>>>
>>>>>>>>> The classes that we are testing are also the classes used by the
>>>>>>>>> testing framework. IOW, any of the java.* classes that JUnit
>>>>>>>>> itself
>>>>>>>>> needs (ex. java.util.HashMap) are exactly the same implementation
>>>>>>>>> that it's testing.
>>>>>>>>>
>>>>>>>>> That's why I think it's subtly different than a "bootstrap and use
>>>>>>>>> version - 1 to test" problem. See what I mean?
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Yeah yeah, I was already way beyond thinking "just" JUnit is usable
>>>>>>>> for the
>>>>>>>> kind of test you're describing. At some point, fundamentally, you
>>>>>>>> either trust
>>>>>>>> something external (whether its the sun jdk or the intel compiler
>>>>>>>> designers,
>>>>>>>> at some point you do draw a line) or you find a way to bootstrap.
>>>>>>>>
>>>>>>>>
>>>>>>> Well, we do trust the Sun JDK.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>> I'm very open to the idea that I'm missing something here, but I'd
>>>>>>>>> like to know that you see the issue - that when we test, we have
>>>>>>>>>
>>>>>>>>> VM + "classlib to be tested" + JUnit + testcases
>>>>>>>>>
>>>>>>>>> where the testcases are testing the classlib the VM is running
>>>>>>>>> JUnit
>>>>>>>>> with.
>>>>>>>>>
>>>>>>>>> There never is isolation of the code being tested :
>>>>>>>>>
>>>>>>>>> VM + "known good classlib" + Junit + testcases
>>>>>>>>>
>>>>>>>>> unless we have some framework where
>>>>>>>>>
>>>>>>>>> VM + "known good classlib" + JUnit
>>>>>>>>> + framework("classlib to be tested")
>>>>>>>>> + testcases
>>>>>>>>>
>>>>>>>>> and it's that notion of "framework()" that I'm advocating we
>>>>>>>>> explore.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> I'm all for exploring it, I just fundamentally don't buy into the
>>>>>>>> "known
>>>>>>>> good" bit. What happens when the 'classlib to be tested' is 'known
>>>>>>>> better' than the 'known good' one? How do you define "known"? How
>>>>>>>> do you
>>>>>>>> define "good"?
>>>>>>>>
>>>>>>>>
>>>>>>> Known? Passed some set of tests. So it could be the Sun JDK for the
>>>>>>> VM + "known good" part.
>>>>>>>
>>>>>>> I think you intuitively understand this. When you find a bug in code
>>>>>>> you are testing, you first assume it's your code, not the framework,
>>>>>>> right? In our case, our framework is actually the code we are
>>>>>>> testing, so we have a bit of a logical conundrum.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Hi Geir,
>>>>>>
>>>>>> The number of Harmony public API classes that get loaded just to
>>>>>> run the
>>>>>> JUnit harness is a little over 200. The majority of these are out of
>>>>>> LUNI with a very low number coming from each of Security, NIO, Archive
>>>>>> and Text.
>>>>>>
>>>>>> Sure there is a circular dependency between what we are building
>>>>>> and the
>>>>>> framework we are using to test it but it appears to touch on only a
>>>>>> relatively small part of Harmony....IMHO.
>>>>>>
>>>>>> Best regards,
>>>>>> George
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>>>> Further ideas...
>>>>>>>>>>
>>>>>>>>>> -> look at how the native world does testing
>>>>>>>>>> (hint: it usually has #ifdefs, uses perl along the way, and
>>>>>>>>>> it is
>>>>>>>>>> certainly
>>>>>>>>>> "messy")
>>>>>>>>>> -> emulate that
>>>>>>>>>>
>>>>>>>>>> -> build a bigger, better specification test
>>>>>>>>>> -> and somehow "prove" it is "good enough"
>>>>>>>>>>
>>>>>>>>>> -> build a bigger, better integration test
>>>>>>>>>> -> and somehow "prove" it is "good enough"
>>>>>>>>>>
>>>>>>>>>> I'll admit my primary interest is the last one...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> The problem I see with the last one is that the "parameter
>>>>>>>>> space" is
>>>>>>>>> *huge*.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Yeah, that's one of the things that makes it interesting.
>>>>>>>> Fortunately
>>>>>>>> open source does have many monkeys...
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> I believe that your preference for the last one comes from the
>>>>>>>>> Monte-Carlo style approach that Gump uses - hope that your test
>>>>>>>>> suite has enough variance that you "push" the thing being tested
>>>>>>>>> through enough of the parameter space that you can be comfortable
>>>>>>>>> you would have exposed the bugs. Maybe.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Ooh, now its becoming rather abstract...
>>>>>>>>
>>>>>>>> Well, perhaps, but more of the gump approache comes from the idea
>>>>>>>> that
>>>>>>>> the parameter space itself is also at some point defined in
>>>>>>>> software,
>>>>>>>> which may have bugs of its own. You circumvent that by making
>>>>>>>> humans the
>>>>>>>> parameter space (don't start about how humans are buggy. We don't
>>>>>>>> want to
>>>>>>>> get into existialism or faith systems when talking about unit
>>>>>>>> testing do
>>>>>>>> we?). The thing that gump enables is "many monkey QA" - a way for
>>>>>>>> thousands
>>>>>>>> of human beings to concurrently make shared assertions about
>>>>>>>> software
>>>>>>>> without actually needing all that much human interaction.
>>>>>>>>
>>>>>>>> More concretely, if harmony can run all known java software, and run
>>>>>>>> it to
>>>>>>>> the asserted satisfaction of all its developers, you can trust that
>>>>>>>> you have
>>>>>>>> covered all the /relevant/ parts of the parameter space you
>>>>>>>> describe.
>>>>>>>>
>>>>>>>>
>>>>>>> Yes. And when you can run all knownn Java software, let me know :)
>>>>>>> That's my point about the parameter space being huge. Even when you
>>>>>>> reduce the definition to "that of all known Java software", you still
>>>>>>> have a huge problem on your hands.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> You
>>>>>>>> will never get that level of trust when the assertions are made by
>>>>>>>> software
>>>>>>>> rather than humans. This is how open source leads to software
>>>>>>>> quality.
>>>>>>>>
>>>>>>>> Quoting myself, 'gump is the most misunderstood piece of software,
>>>>>>>> ever'.
>>>>>>>>
>>>>>>>> cheers,
>>>>>>>>
>>>>>>>> Leo
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>>
>>>>
>>>>
>> --
>> Paulex Yang
>> China Software Development Lab
>> IBM
>>
>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
>
--
Richard Liang
China Software Development Lab, IBM
Re: [classlib] Testing
Posted by Stepan Mishura <st...@gmail.com>.
I agree with Mikhail here: package names should be short and informative.
Thanks,
Stepan.
On 4/26/06, Mikhail Loenko wrote:
>
> how about 'specific'? impl seems to be not very informative.
>
> I have a concern abou proposed package naming guidelines:
> package name
> org.apache.harmony.security.tests.org.apache.harmony.security
> is not much better then 1000-character long test name.
>
> Thanks,
> Mikhail
>
>
> 2006/4/26, Paulex Yang <pa...@gmail.com>:
> > Oliver Deakin wrote:
> > > George Harley wrote:
> > >> Mikhail Loenko wrote:
> > >>> Hello
> > >>>
> > >>> I'd like to bring this thread back.
> > >>>
> > >>> Number of tests is growing and it is time to put them in order.
> > >>>
> > >>> So far we may have:
> > >>>
> > >>> 1) implementation-specific tests that designed to be run from
> > >>> bootclasspath
> > >>> 2) implementation-specific tests that might be run from classpath
> > >>> 3) implementation-specific tests that designed to be run from
> classpath
> > >>> 4) implementation-independent tests that designed to be run from
> > >>> bootclasspath
> > >>> 5) implementation-independent tests that might be run from classpath
> > >>> 6) implementation-independent tests that designed to be run from
> > >>> classpath
> > >>>
> > >>> Also we seem to have the following packages, where the tests are:
> > >>>
> > >>> 1) the same package as implementation
> > >>> 2) org.apache.harmony.tests.[the same package as implementation]
> > >>> 3) tests.api.[the same package as implementation]
> > >>>
> > >>> I suggest that we work out step-by-step solution as we could not
> reach
> > >>> an agreement for the general universal one
> > >>>
> > >>> So for the first step I suggest that we separate i-independent tests
> > >>> that must or may be run from classpath
> > >>>
> > >>> I suggest that we put them into package
> > >>> tests.module.compatible.[package of implementation being tested]
> > >>>
> > >>> Comments?
> > >>>
> > >>> Thanks,
> > >>> Mikhail
> > >>>
> > >>
> > >> Hi Mikhail,
> > >>
> > >> I've just started working through the modules to merge test packages
> > >> "tests.api.[same package as implementation]" and "tests.api.[same
> > >> package as implementation]" into one package space. Using the class
> > >> library package naming guidelines from off the web site [1], all of
> > >> the tests for the text module have been consolidated under
> > >> org.apache.harmony.text.tests.[package under test].
> > >>
> > >> Of course, the text module has only "implementation-independent tests
> > >> that designed to be run from classpath". For modules that have got
> > >> implementation-specific tests then I suppose we could use something
> > >> like "org.apache.harmony.[module].tests.impl.[package under test]" or
> > >> "org.apache.harmony.[module].tests.internal.[package under test]"
> > >> etc. I've got no preference.
> > >
> > > I think impl is preferable over internal here, as we already use
> > > internal in our implementation package names to indicate classes
> > > totally internal to that bundle. To also use internal to label tests
> > > that are implementation specific may cause confusion.
> > >
> > +1 from me.
> > > Regards,
> > > Oliver
> > >
> > >>
> > >> Best regards,
> > >> George
> > >>
> > >>
> > >> [1]
> > >>
> http://incubator.apache.org/harmony/subcomponents/classlibrary/pkgnaming.html
> > >>
> > >>
> > >>
> > >>>
> > >>> 2006/3/24, George Harley <ge...@googlemail.com>:
> > >>>
> > >>>> Geir Magnusson Jr wrote:
> > >>>>
> > >>>>> Leo Simons wrote:
> > >>>>>
> > >>>>>> On Wed, Mar 22, 2006 at 08:02:44AM -0500, Geir Magnusson Jr
> wrote:
> > >>>>>>
> > >>>>>>> Leo Simons wrote:
> > >>>>>>>
> > >>>>>>>> On Wed, Mar 22, 2006 at 07:15:28AM -0500, Geir Magnusson Jr
> wrote:
> > >>>>>>>>
> > >>>>>>>>> Pulling out of the various threads where we have been
> discussing,
> > >>>>>>>>> can we agree on the problem :
> > >>>>>>>>>
> > >>>>>>>>> We have unique problems compared to other Java projects
> > >>>>>>>>> because we
> > >>>>>>>>> need to find a way to reliably test the things that are
> commonly
> > >>>>>>>>> expected to be a solid point of reference - namely the core
> class
> > >>>>>>>>> library.
> > >>>>>>>>>
> > >>>>>>>>> Further, we've been implicitly doing "integration testing"
> > >>>>>>>>> because
> > >>>>>>>>> - so far - the only way we've been testing our code has been
> 'in
> > >>>>>>>>> situ' in the VM - not in an isolated test harness. To me,
> this
> > >>>>>>>>> turns it into an integration test.
> > >>>>>>>>>
> > >>>>>>>>> Sure, we're using JUnit, but because of the fact we are
> > >>>>>>>>> implmenting core java.* APIs, we aren't testing with a
> framework
> > >>>>>>>>> that has been independently tested for correctness, like we
> would
> > >>>>>>>>> when testing any other code.
> > >>>>>>>>>
> > >>>>>>>>> I hope I got that idea across - I believe that we have to go
> > >>>>>>>>> beyond normal testing approaches because we don't have a
> normal
> > >>>>>>>>> situation.
> > >>>>>>>>>
> > >>>>>>>> Where we define 'normal situation' as "running a test framework
> on
> > >>>>>>>> top of
> > >>>>>>>> the sun jdk and expecting any bugs to not be in that jdk".
> There's
> > >>>>>>>> plenty
> > >>>>>>>> of projects out there that have to test things without having
> > >>>>>>>> such a
> > >>>>>>>> "stable reference JDK" luxury.....I imagine that testing GCC is
> > >>>>>>>> just as
> > >>>>>>>> hard as this problem we have here :-)
> > >>>>>>>>
> > >>>>>>> Is it the same? We need to have a running JVM+classlibarary to
> > >>>>>>> test
> > >>>>>>> the classlibrary code.
> > >>>>>>>
> > >>>>>> Well you need a working C compiler and standard C library to
> > >>>>>> compile the
> > >>>>>> compiler so you can compile make so you can build bash so you can
> > >>>>>> run
> > >>>>>> perl (which uses the standard C library functions all over the
> > >>>>>> place of
> > >>>>>> course) so you can run the standard C library tests so that you
> know
> > >>>>>> that
> > >>>>>> the library you used when compiling the compiler were correct so
> > >>>>>> you can
> > >>>>>> run the compiler tests. I don't think they actually do things
> that
> > >>>>>> way, but
> > >>>>>> it seems like basically the same problem. Having a virtual
> > >>>>>> machine just
> > >>>>>> makes it easier since you still assume "the native world" as a
> > >>>>>> baseline,
> > >>>>>> which is a lot more than "the hardware".
> > >>>>>>
> > >>>>> There's a difference. You can use a completely separate toolchain
> to
> > >>>>> build, test and verify the output of the C compiler.
> > >>>>>
> > >>>>> In our case, we are using the thing we are testing to test itself.
> > >>>>> There is no "known good" element possible right now.
> > >>>>>
> > >>>>> We use the classlibrary we are trying to test to execute the test
> > >>>>> framework that tests the classlibrary that is running it.
> > >>>>>
> > >>>>> The tool is testing itself.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>>>>> So I think there are three things we want to do (adopting the
> > >>>>>>>>> terminology that came from the discussion with Tim and Leo ) :
> > >>>>>>>>>
> > >>>>>>>>> 1) implementation tests
> > >>>>>>>>> 2) spec/API tests (I'll bundle together)
> > >>>>>>>>> 3) integration/functional tests
> > >>>>>>>>>
> > >>>>>>>>> I believe that for #1, the issues related to being on the
> > >>>>>>>>> bootclasspath don't matter, because we aren't testing that
> aspect
> > >>>>>>>>> of the classes (which is how they behave integrated w/ the VM
> and
> > >>>>>>>>> security system) but rather the basic internal functioning.
> > >>>>>>>>>
> > >>>>>>>>> I'm not sure how to approach this, but I'll try. I'd love to
> > >>>>>>>>> hear
> > >>>>>>>>> how Sun, IBM or BEA deals with this, or be told why it isn't
> an
> > >>>>>>>>> issue :)
> > >>>>>>>>>
> > >>>>>>>>> Implementation tests : I'd like to see us be able to do #1 via
> > >>>>>>>>> the
> > >>>>>>>>> standard same-package technique (i.e. testing a.b.C w/
> a.b.CTest)
> > >>>>>>>>> but we'll run into a tangle of classloader problems, I
> suspect,
> > >>>>>>>>> becuase we want to be testing java.* code in a system that
> > >>>>>>>>> already
> > >>>>>>>>> has java.* code. Can anyone see a way we can do this - test
> the
> > >>>>>>>>> classlibrary from the integration point of view - using some
> test
> > >>>>>>>>> harness + any known-good JRE, like Sun's or IBM's?
> > >>>>>>>>>
> > >>>>>>>> Ew, that won't work in the end since we should assume our own
> JRE
> > >>>>>>>> is going
> > >>>>>>>> to be "known-better" :-). But it might be a nice way to
> > >>>>>>>> "bootstrap"
> > >>>>>>>> (eg
> > >>>>>>>> we test with an external JRE until we satisfy the tests and
> > >>>>>>>> then we
> > >>>>>>>> switch
> > >>>>>>>> to testing with an earlier build).
> > >>>>>>>>
> > >>>>>>> Lets be clear - even using our own "earlier build" doesn't solve
> > >>>>>>> the
> > >>>>>>> problem I'm describing, because as it stands now, we don't use
> > >>>>>>> "earlier build" classes to test with - we use the code we want
> to
> > >>>>>>> test as the clsaslibrary for the JRE that's running the test
> > >>>>>>> framework.
> > >>>>>>>
> > >>>>>>> The classes that we are testing are also the classes used by the
> > >>>>>>> testing framework. IOW, any of the java.* classes that JUnit
> > >>>>>>> itself
> > >>>>>>> needs (ex. java.util.HashMap) are exactly the same
> implementation
> > >>>>>>> that it's testing.
> > >>>>>>>
> > >>>>>>> That's why I think it's subtly different than a "bootstrap and
> use
> > >>>>>>> version - 1 to test" problem. See what I mean?
> > >>>>>>>
> > >>>>>> Yeah yeah, I was already way beyond thinking "just" JUnit is
> usable
> > >>>>>> for the
> > >>>>>> kind of test you're describing. At some point, fundamentally, you
> > >>>>>> either trust
> > >>>>>> something external (whether its the sun jdk or the intel compiler
> > >>>>>> designers,
> > >>>>>> at some point you do draw a line) or you find a way to bootstrap.
> > >>>>>>
> > >>>>> Well, we do trust the Sun JDK.
> > >>>>>
> > >>>>>
> > >>>>>>> I'm very open to the idea that I'm missing something here, but
> I'd
> > >>>>>>> like to know that you see the issue - that when we test, we have
> > >>>>>>>
> > >>>>>>> VM + "classlib to be tested" + JUnit + testcases
> > >>>>>>>
> > >>>>>>> where the testcases are testing the classlib the VM is running
> > >>>>>>> JUnit
> > >>>>>>> with.
> > >>>>>>>
> > >>>>>>> There never is isolation of the code being tested :
> > >>>>>>>
> > >>>>>>> VM + "known good classlib" + Junit + testcases
> > >>>>>>>
> > >>>>>>> unless we have some framework where
> > >>>>>>>
> > >>>>>>> VM + "known good classlib" + JUnit
> > >>>>>>> + framework("classlib to be tested")
> > >>>>>>> + testcases
> > >>>>>>>
> > >>>>>>> and it's that notion of "framework()" that I'm advocating we
> > >>>>>>> explore.
> > >>>>>>>
> > >>>>>> I'm all for exploring it, I just fundamentally don't buy into the
> > >>>>>> "known
> > >>>>>> good" bit. What happens when the 'classlib to be tested' is
> 'known
> > >>>>>> better' than the 'known good' one? How do you define "known"? How
> > >>>>>> do you
> > >>>>>> define "good"?
> > >>>>>>
> > >>>>> Known? Passed some set of tests. So it could be the Sun JDK for
> the
> > >>>>> VM + "known good" part.
> > >>>>>
> > >>>>> I think you intuitively understand this. When you find a bug in
> code
> > >>>>> you are testing, you first assume it's your code, not the
> framework,
> > >>>>> right? In our case, our framework is actually the code we are
> > >>>>> testing, so we have a bit of a logical conundrum.
> > >>>>>
> > >>>>>
> > >>>> Hi Geir,
> > >>>>
> > >>>> The number of Harmony public API classes that get loaded just to
> > >>>> run the
> > >>>> JUnit harness is a little over 200. The majority of these are out
> of
> > >>>> LUNI with a very low number coming from each of Security, NIO,
> Archive
> > >>>> and Text.
> > >>>>
> > >>>> Sure there is a circular dependency between what we are building
> > >>>> and the
> > >>>> framework we are using to test it but it appears to touch on only a
> > >>>> relatively small part of Harmony....IMHO.
> > >>>>
> > >>>> Best regards,
> > >>>> George
> > >>>>
> > >>>>
> > >>>>
> > >>>>>>>> Further ideas...
> > >>>>>>>>
> > >>>>>>>> -> look at how the native world does testing
> > >>>>>>>> (hint: it usually has #ifdefs, uses perl along the way, and
> > >>>>>>>> it is
> > >>>>>>>> certainly
> > >>>>>>>> "messy")
> > >>>>>>>> -> emulate that
> > >>>>>>>>
> > >>>>>>>> -> build a bigger, better specification test
> > >>>>>>>> -> and somehow "prove" it is "good enough"
> > >>>>>>>>
> > >>>>>>>> -> build a bigger, better integration test
> > >>>>>>>> -> and somehow "prove" it is "good enough"
> > >>>>>>>>
> > >>>>>>>> I'll admit my primary interest is the last one...
> > >>>>>>>>
> > >>>>>>> The problem I see with the last one is that the "parameter
> > >>>>>>> space" is
> > >>>>>>> *huge*.
> > >>>>>>>
> > >>>>>> Yeah, that's one of the things that makes it interesting.
> > >>>>>> Fortunately
> > >>>>>> open source does have many monkeys...
> > >>>>>>
> > >>>>>>
> > >>>>>>> I believe that your preference for the last one comes from the
> > >>>>>>> Monte-Carlo style approach that Gump uses - hope that your test
> > >>>>>>> suite has enough variance that you "push" the thing being tested
> > >>>>>>> through enough of the parameter space that you can be
> comfortable
> > >>>>>>> you would have exposed the bugs. Maybe.
> > >>>>>>>
> > >>>>>> Ooh, now its becoming rather abstract...
> > >>>>>>
> > >>>>>> Well, perhaps, but more of the gump approache comes from the idea
> > >>>>>> that
> > >>>>>> the parameter space itself is also at some point defined in
> > >>>>>> software,
> > >>>>>> which may have bugs of its own. You circumvent that by making
> > >>>>>> humans the
> > >>>>>> parameter space (don't start about how humans are buggy. We don't
> > >>>>>> want to
> > >>>>>> get into existialism or faith systems when talking about unit
> > >>>>>> testing do
> > >>>>>> we?). The thing that gump enables is "many monkey QA" - a way for
> > >>>>>> thousands
> > >>>>>> of human beings to concurrently make shared assertions about
> > >>>>>> software
> > >>>>>> without actually needing all that much human interaction.
> > >>>>>>
> > >>>>>> More concretely, if harmony can run all known java software, and
> run
> > >>>>>> it to
> > >>>>>> the asserted satisfaction of all its developers, you can trust
> that
> > >>>>>> you have
> > >>>>>> covered all the /relevant/ parts of the parameter space you
> > >>>>>> describe.
> > >>>>>>
> > >>>>> Yes. And when you can run all knownn Java software, let me know
> :)
> > >>>>> That's my point about the parameter space being huge. Even when
> you
> > >>>>> reduce the definition to "that of all known Java software", you
> still
> > >>>>> have a huge problem on your hands.
> > >>>>>
> > >>>>>
> > >>>>>> You
> > >>>>>> will never get that level of trust when the assertions are made
> by
> > >>>>>> software
> > >>>>>> rather than humans. This is how open source leads to software
> > >>>>>> quality.
> > >>>>>>
> > >>>>>> Quoting myself, 'gump is the most misunderstood piece of
> software,
> > >>>>>> ever'.
> > >>>>>>
> > >>>>>> cheers,
> > >>>>>>
> > >>>>>> Leo
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>
> > >>>
> > >>>
> ---------------------------------------------------------------------
> > >>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> > >>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > >>> For additional commands, e-mail:
> harmony-dev-help@incubator.apache.org
> > >>>
> > >>>
> > >>>
> > >>
> > >>
> > >> ---------------------------------------------------------------------
> > >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> > >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > >> For additional commands, e-mail:
> harmony-dev-help@incubator.apache.org
> > >>
> > >>
> > >
> >
> >
> > --
> > Paulex Yang
> > China Software Development Lab
> > IBM
> >
> >
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
--
Thanks,
Stepan Mishura
Intel Middleware Products Division
------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Mikhail Loenko <ml...@gmail.com>.
how about 'specific'? impl seems to be not very informative.
I have a concern abou proposed package naming guidelines:
package name
org.apache.harmony.security.tests.org.apache.harmony.security
is not much better then 1000-character long test name.
Thanks,
Mikhail
2006/4/26, Paulex Yang <pa...@gmail.com>:
> Oliver Deakin wrote:
> > George Harley wrote:
> >> Mikhail Loenko wrote:
> >>> Hello
> >>>
> >>> I'd like to bring this thread back.
> >>>
> >>> Number of tests is growing and it is time to put them in order.
> >>>
> >>> So far we may have:
> >>>
> >>> 1) implementation-specific tests that designed to be run from
> >>> bootclasspath
> >>> 2) implementation-specific tests that might be run from classpath
> >>> 3) implementation-specific tests that designed to be run from classpath
> >>> 4) implementation-independent tests that designed to be run from
> >>> bootclasspath
> >>> 5) implementation-independent tests that might be run from classpath
> >>> 6) implementation-independent tests that designed to be run from
> >>> classpath
> >>>
> >>> Also we seem to have the following packages, where the tests are:
> >>>
> >>> 1) the same package as implementation
> >>> 2) org.apache.harmony.tests.[the same package as implementation]
> >>> 3) tests.api.[the same package as implementation]
> >>>
> >>> I suggest that we work out step-by-step solution as we could not reach
> >>> an agreement for the general universal one
> >>>
> >>> So for the first step I suggest that we separate i-independent tests
> >>> that must or may be run from classpath
> >>>
> >>> I suggest that we put them into package
> >>> tests.module.compatible.[package of implementation being tested]
> >>>
> >>> Comments?
> >>>
> >>> Thanks,
> >>> Mikhail
> >>>
> >>
> >> Hi Mikhail,
> >>
> >> I've just started working through the modules to merge test packages
> >> "tests.api.[same package as implementation]" and "tests.api.[same
> >> package as implementation]" into one package space. Using the class
> >> library package naming guidelines from off the web site [1], all of
> >> the tests for the text module have been consolidated under
> >> org.apache.harmony.text.tests.[package under test].
> >>
> >> Of course, the text module has only "implementation-independent tests
> >> that designed to be run from classpath". For modules that have got
> >> implementation-specific tests then I suppose we could use something
> >> like "org.apache.harmony.[module].tests.impl.[package under test]" or
> >> "org.apache.harmony.[module].tests.internal.[package under test]"
> >> etc. I've got no preference.
> >
> > I think impl is preferable over internal here, as we already use
> > internal in our implementation package names to indicate classes
> > totally internal to that bundle. To also use internal to label tests
> > that are implementation specific may cause confusion.
> >
> +1 from me.
> > Regards,
> > Oliver
> >
> >>
> >> Best regards,
> >> George
> >>
> >>
> >> [1]
> >> http://incubator.apache.org/harmony/subcomponents/classlibrary/pkgnaming.html
> >>
> >>
> >>
> >>>
> >>> 2006/3/24, George Harley <ge...@googlemail.com>:
> >>>
> >>>> Geir Magnusson Jr wrote:
> >>>>
> >>>>> Leo Simons wrote:
> >>>>>
> >>>>>> On Wed, Mar 22, 2006 at 08:02:44AM -0500, Geir Magnusson Jr wrote:
> >>>>>>
> >>>>>>> Leo Simons wrote:
> >>>>>>>
> >>>>>>>> On Wed, Mar 22, 2006 at 07:15:28AM -0500, Geir Magnusson Jr wrote:
> >>>>>>>>
> >>>>>>>>> Pulling out of the various threads where we have been discussing,
> >>>>>>>>> can we agree on the problem :
> >>>>>>>>>
> >>>>>>>>> We have unique problems compared to other Java projects
> >>>>>>>>> because we
> >>>>>>>>> need to find a way to reliably test the things that are commonly
> >>>>>>>>> expected to be a solid point of reference - namely the core class
> >>>>>>>>> library.
> >>>>>>>>>
> >>>>>>>>> Further, we've been implicitly doing "integration testing"
> >>>>>>>>> because
> >>>>>>>>> - so far - the only way we've been testing our code has been 'in
> >>>>>>>>> situ' in the VM - not in an isolated test harness. To me, this
> >>>>>>>>> turns it into an integration test.
> >>>>>>>>>
> >>>>>>>>> Sure, we're using JUnit, but because of the fact we are
> >>>>>>>>> implmenting core java.* APIs, we aren't testing with a framework
> >>>>>>>>> that has been independently tested for correctness, like we would
> >>>>>>>>> when testing any other code.
> >>>>>>>>>
> >>>>>>>>> I hope I got that idea across - I believe that we have to go
> >>>>>>>>> beyond normal testing approaches because we don't have a normal
> >>>>>>>>> situation.
> >>>>>>>>>
> >>>>>>>> Where we define 'normal situation' as "running a test framework on
> >>>>>>>> top of
> >>>>>>>> the sun jdk and expecting any bugs to not be in that jdk". There's
> >>>>>>>> plenty
> >>>>>>>> of projects out there that have to test things without having
> >>>>>>>> such a
> >>>>>>>> "stable reference JDK" luxury.....I imagine that testing GCC is
> >>>>>>>> just as
> >>>>>>>> hard as this problem we have here :-)
> >>>>>>>>
> >>>>>>> Is it the same? We need to have a running JVM+classlibarary to
> >>>>>>> test
> >>>>>>> the classlibrary code.
> >>>>>>>
> >>>>>> Well you need a working C compiler and standard C library to
> >>>>>> compile the
> >>>>>> compiler so you can compile make so you can build bash so you can
> >>>>>> run
> >>>>>> perl (which uses the standard C library functions all over the
> >>>>>> place of
> >>>>>> course) so you can run the standard C library tests so that you know
> >>>>>> that
> >>>>>> the library you used when compiling the compiler were correct so
> >>>>>> you can
> >>>>>> run the compiler tests. I don't think they actually do things that
> >>>>>> way, but
> >>>>>> it seems like basically the same problem. Having a virtual
> >>>>>> machine just
> >>>>>> makes it easier since you still assume "the native world" as a
> >>>>>> baseline,
> >>>>>> which is a lot more than "the hardware".
> >>>>>>
> >>>>> There's a difference. You can use a completely separate toolchain to
> >>>>> build, test and verify the output of the C compiler.
> >>>>>
> >>>>> In our case, we are using the thing we are testing to test itself.
> >>>>> There is no "known good" element possible right now.
> >>>>>
> >>>>> We use the classlibrary we are trying to test to execute the test
> >>>>> framework that tests the classlibrary that is running it.
> >>>>>
> >>>>> The tool is testing itself.
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>> So I think there are three things we want to do (adopting the
> >>>>>>>>> terminology that came from the discussion with Tim and Leo ) :
> >>>>>>>>>
> >>>>>>>>> 1) implementation tests
> >>>>>>>>> 2) spec/API tests (I'll bundle together)
> >>>>>>>>> 3) integration/functional tests
> >>>>>>>>>
> >>>>>>>>> I believe that for #1, the issues related to being on the
> >>>>>>>>> bootclasspath don't matter, because we aren't testing that aspect
> >>>>>>>>> of the classes (which is how they behave integrated w/ the VM and
> >>>>>>>>> security system) but rather the basic internal functioning.
> >>>>>>>>>
> >>>>>>>>> I'm not sure how to approach this, but I'll try. I'd love to
> >>>>>>>>> hear
> >>>>>>>>> how Sun, IBM or BEA deals with this, or be told why it isn't an
> >>>>>>>>> issue :)
> >>>>>>>>>
> >>>>>>>>> Implementation tests : I'd like to see us be able to do #1 via
> >>>>>>>>> the
> >>>>>>>>> standard same-package technique (i.e. testing a.b.C w/ a.b.CTest)
> >>>>>>>>> but we'll run into a tangle of classloader problems, I suspect,
> >>>>>>>>> becuase we want to be testing java.* code in a system that
> >>>>>>>>> already
> >>>>>>>>> has java.* code. Can anyone see a way we can do this - test the
> >>>>>>>>> classlibrary from the integration point of view - using some test
> >>>>>>>>> harness + any known-good JRE, like Sun's or IBM's?
> >>>>>>>>>
> >>>>>>>> Ew, that won't work in the end since we should assume our own JRE
> >>>>>>>> is going
> >>>>>>>> to be "known-better" :-). But it might be a nice way to
> >>>>>>>> "bootstrap"
> >>>>>>>> (eg
> >>>>>>>> we test with an external JRE until we satisfy the tests and
> >>>>>>>> then we
> >>>>>>>> switch
> >>>>>>>> to testing with an earlier build).
> >>>>>>>>
> >>>>>>> Lets be clear - even using our own "earlier build" doesn't solve
> >>>>>>> the
> >>>>>>> problem I'm describing, because as it stands now, we don't use
> >>>>>>> "earlier build" classes to test with - we use the code we want to
> >>>>>>> test as the clsaslibrary for the JRE that's running the test
> >>>>>>> framework.
> >>>>>>>
> >>>>>>> The classes that we are testing are also the classes used by the
> >>>>>>> testing framework. IOW, any of the java.* classes that JUnit
> >>>>>>> itself
> >>>>>>> needs (ex. java.util.HashMap) are exactly the same implementation
> >>>>>>> that it's testing.
> >>>>>>>
> >>>>>>> That's why I think it's subtly different than a "bootstrap and use
> >>>>>>> version - 1 to test" problem. See what I mean?
> >>>>>>>
> >>>>>> Yeah yeah, I was already way beyond thinking "just" JUnit is usable
> >>>>>> for the
> >>>>>> kind of test you're describing. At some point, fundamentally, you
> >>>>>> either trust
> >>>>>> something external (whether its the sun jdk or the intel compiler
> >>>>>> designers,
> >>>>>> at some point you do draw a line) or you find a way to bootstrap.
> >>>>>>
> >>>>> Well, we do trust the Sun JDK.
> >>>>>
> >>>>>
> >>>>>>> I'm very open to the idea that I'm missing something here, but I'd
> >>>>>>> like to know that you see the issue - that when we test, we have
> >>>>>>>
> >>>>>>> VM + "classlib to be tested" + JUnit + testcases
> >>>>>>>
> >>>>>>> where the testcases are testing the classlib the VM is running
> >>>>>>> JUnit
> >>>>>>> with.
> >>>>>>>
> >>>>>>> There never is isolation of the code being tested :
> >>>>>>>
> >>>>>>> VM + "known good classlib" + Junit + testcases
> >>>>>>>
> >>>>>>> unless we have some framework where
> >>>>>>>
> >>>>>>> VM + "known good classlib" + JUnit
> >>>>>>> + framework("classlib to be tested")
> >>>>>>> + testcases
> >>>>>>>
> >>>>>>> and it's that notion of "framework()" that I'm advocating we
> >>>>>>> explore.
> >>>>>>>
> >>>>>> I'm all for exploring it, I just fundamentally don't buy into the
> >>>>>> "known
> >>>>>> good" bit. What happens when the 'classlib to be tested' is 'known
> >>>>>> better' than the 'known good' one? How do you define "known"? How
> >>>>>> do you
> >>>>>> define "good"?
> >>>>>>
> >>>>> Known? Passed some set of tests. So it could be the Sun JDK for the
> >>>>> VM + "known good" part.
> >>>>>
> >>>>> I think you intuitively understand this. When you find a bug in code
> >>>>> you are testing, you first assume it's your code, not the framework,
> >>>>> right? In our case, our framework is actually the code we are
> >>>>> testing, so we have a bit of a logical conundrum.
> >>>>>
> >>>>>
> >>>> Hi Geir,
> >>>>
> >>>> The number of Harmony public API classes that get loaded just to
> >>>> run the
> >>>> JUnit harness is a little over 200. The majority of these are out of
> >>>> LUNI with a very low number coming from each of Security, NIO, Archive
> >>>> and Text.
> >>>>
> >>>> Sure there is a circular dependency between what we are building
> >>>> and the
> >>>> framework we are using to test it but it appears to touch on only a
> >>>> relatively small part of Harmony....IMHO.
> >>>>
> >>>> Best regards,
> >>>> George
> >>>>
> >>>>
> >>>>
> >>>>>>>> Further ideas...
> >>>>>>>>
> >>>>>>>> -> look at how the native world does testing
> >>>>>>>> (hint: it usually has #ifdefs, uses perl along the way, and
> >>>>>>>> it is
> >>>>>>>> certainly
> >>>>>>>> "messy")
> >>>>>>>> -> emulate that
> >>>>>>>>
> >>>>>>>> -> build a bigger, better specification test
> >>>>>>>> -> and somehow "prove" it is "good enough"
> >>>>>>>>
> >>>>>>>> -> build a bigger, better integration test
> >>>>>>>> -> and somehow "prove" it is "good enough"
> >>>>>>>>
> >>>>>>>> I'll admit my primary interest is the last one...
> >>>>>>>>
> >>>>>>> The problem I see with the last one is that the "parameter
> >>>>>>> space" is
> >>>>>>> *huge*.
> >>>>>>>
> >>>>>> Yeah, that's one of the things that makes it interesting.
> >>>>>> Fortunately
> >>>>>> open source does have many monkeys...
> >>>>>>
> >>>>>>
> >>>>>>> I believe that your preference for the last one comes from the
> >>>>>>> Monte-Carlo style approach that Gump uses - hope that your test
> >>>>>>> suite has enough variance that you "push" the thing being tested
> >>>>>>> through enough of the parameter space that you can be comfortable
> >>>>>>> you would have exposed the bugs. Maybe.
> >>>>>>>
> >>>>>> Ooh, now its becoming rather abstract...
> >>>>>>
> >>>>>> Well, perhaps, but more of the gump approache comes from the idea
> >>>>>> that
> >>>>>> the parameter space itself is also at some point defined in
> >>>>>> software,
> >>>>>> which may have bugs of its own. You circumvent that by making
> >>>>>> humans the
> >>>>>> parameter space (don't start about how humans are buggy. We don't
> >>>>>> want to
> >>>>>> get into existialism or faith systems when talking about unit
> >>>>>> testing do
> >>>>>> we?). The thing that gump enables is "many monkey QA" - a way for
> >>>>>> thousands
> >>>>>> of human beings to concurrently make shared assertions about
> >>>>>> software
> >>>>>> without actually needing all that much human interaction.
> >>>>>>
> >>>>>> More concretely, if harmony can run all known java software, and run
> >>>>>> it to
> >>>>>> the asserted satisfaction of all its developers, you can trust that
> >>>>>> you have
> >>>>>> covered all the /relevant/ parts of the parameter space you
> >>>>>> describe.
> >>>>>>
> >>>>> Yes. And when you can run all knownn Java software, let me know :)
> >>>>> That's my point about the parameter space being huge. Even when you
> >>>>> reduce the definition to "that of all known Java software", you still
> >>>>> have a huge problem on your hands.
> >>>>>
> >>>>>
> >>>>>> You
> >>>>>> will never get that level of trust when the assertions are made by
> >>>>>> software
> >>>>>> rather than humans. This is how open source leads to software
> >>>>>> quality.
> >>>>>>
> >>>>>> Quoting myself, 'gump is the most misunderstood piece of software,
> >>>>>> ever'.
> >>>>>>
> >>>>>> cheers,
> >>>>>>
> >>>>>> Leo
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>>
> >>>
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>
> >>
> >
>
>
> --
> Paulex Yang
> China Software Development Lab
> IBM
>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Paulex Yang <pa...@gmail.com>.
Oliver Deakin wrote:
> George Harley wrote:
>> Mikhail Loenko wrote:
>>> Hello
>>>
>>> I'd like to bring this thread back.
>>>
>>> Number of tests is growing and it is time to put them in order.
>>>
>>> So far we may have:
>>>
>>> 1) implementation-specific tests that designed to be run from
>>> bootclasspath
>>> 2) implementation-specific tests that might be run from classpath
>>> 3) implementation-specific tests that designed to be run from classpath
>>> 4) implementation-independent tests that designed to be run from
>>> bootclasspath
>>> 5) implementation-independent tests that might be run from classpath
>>> 6) implementation-independent tests that designed to be run from
>>> classpath
>>>
>>> Also we seem to have the following packages, where the tests are:
>>>
>>> 1) the same package as implementation
>>> 2) org.apache.harmony.tests.[the same package as implementation]
>>> 3) tests.api.[the same package as implementation]
>>>
>>> I suggest that we work out step-by-step solution as we could not reach
>>> an agreement for the general universal one
>>>
>>> So for the first step I suggest that we separate i-independent tests
>>> that must or may be run from classpath
>>>
>>> I suggest that we put them into package
>>> tests.module.compatible.[package of implementation being tested]
>>>
>>> Comments?
>>>
>>> Thanks,
>>> Mikhail
>>>
>>
>> Hi Mikhail,
>>
>> I've just started working through the modules to merge test packages
>> "tests.api.[same package as implementation]" and "tests.api.[same
>> package as implementation]" into one package space. Using the class
>> library package naming guidelines from off the web site [1], all of
>> the tests for the text module have been consolidated under
>> org.apache.harmony.text.tests.[package under test].
>>
>> Of course, the text module has only "implementation-independent tests
>> that designed to be run from classpath". For modules that have got
>> implementation-specific tests then I suppose we could use something
>> like "org.apache.harmony.[module].tests.impl.[package under test]" or
>> "org.apache.harmony.[module].tests.internal.[package under test]"
>> etc. I've got no preference.
>
> I think impl is preferable over internal here, as we already use
> internal in our implementation package names to indicate classes
> totally internal to that bundle. To also use internal to label tests
> that are implementation specific may cause confusion.
>
+1 from me.
> Regards,
> Oliver
>
>>
>> Best regards,
>> George
>>
>>
>> [1]
>> http://incubator.apache.org/harmony/subcomponents/classlibrary/pkgnaming.html
>>
>>
>>
>>>
>>> 2006/3/24, George Harley <ge...@googlemail.com>:
>>>
>>>> Geir Magnusson Jr wrote:
>>>>
>>>>> Leo Simons wrote:
>>>>>
>>>>>> On Wed, Mar 22, 2006 at 08:02:44AM -0500, Geir Magnusson Jr wrote:
>>>>>>
>>>>>>> Leo Simons wrote:
>>>>>>>
>>>>>>>> On Wed, Mar 22, 2006 at 07:15:28AM -0500, Geir Magnusson Jr wrote:
>>>>>>>>
>>>>>>>>> Pulling out of the various threads where we have been discussing,
>>>>>>>>> can we agree on the problem :
>>>>>>>>>
>>>>>>>>> We have unique problems compared to other Java projects
>>>>>>>>> because we
>>>>>>>>> need to find a way to reliably test the things that are commonly
>>>>>>>>> expected to be a solid point of reference - namely the core class
>>>>>>>>> library.
>>>>>>>>>
>>>>>>>>> Further, we've been implicitly doing "integration testing"
>>>>>>>>> because
>>>>>>>>> - so far - the only way we've been testing our code has been 'in
>>>>>>>>> situ' in the VM - not in an isolated test harness. To me, this
>>>>>>>>> turns it into an integration test.
>>>>>>>>>
>>>>>>>>> Sure, we're using JUnit, but because of the fact we are
>>>>>>>>> implmenting core java.* APIs, we aren't testing with a framework
>>>>>>>>> that has been independently tested for correctness, like we would
>>>>>>>>> when testing any other code.
>>>>>>>>>
>>>>>>>>> I hope I got that idea across - I believe that we have to go
>>>>>>>>> beyond normal testing approaches because we don't have a normal
>>>>>>>>> situation.
>>>>>>>>>
>>>>>>>> Where we define 'normal situation' as "running a test framework on
>>>>>>>> top of
>>>>>>>> the sun jdk and expecting any bugs to not be in that jdk". There's
>>>>>>>> plenty
>>>>>>>> of projects out there that have to test things without having
>>>>>>>> such a
>>>>>>>> "stable reference JDK" luxury.....I imagine that testing GCC is
>>>>>>>> just as
>>>>>>>> hard as this problem we have here :-)
>>>>>>>>
>>>>>>> Is it the same? We need to have a running JVM+classlibarary to
>>>>>>> test
>>>>>>> the classlibrary code.
>>>>>>>
>>>>>> Well you need a working C compiler and standard C library to
>>>>>> compile the
>>>>>> compiler so you can compile make so you can build bash so you can
>>>>>> run
>>>>>> perl (which uses the standard C library functions all over the
>>>>>> place of
>>>>>> course) so you can run the standard C library tests so that you know
>>>>>> that
>>>>>> the library you used when compiling the compiler were correct so
>>>>>> you can
>>>>>> run the compiler tests. I don't think they actually do things that
>>>>>> way, but
>>>>>> it seems like basically the same problem. Having a virtual
>>>>>> machine just
>>>>>> makes it easier since you still assume "the native world" as a
>>>>>> baseline,
>>>>>> which is a lot more than "the hardware".
>>>>>>
>>>>> There's a difference. You can use a completely separate toolchain to
>>>>> build, test and verify the output of the C compiler.
>>>>>
>>>>> In our case, we are using the thing we are testing to test itself.
>>>>> There is no "known good" element possible right now.
>>>>>
>>>>> We use the classlibrary we are trying to test to execute the test
>>>>> framework that tests the classlibrary that is running it.
>>>>>
>>>>> The tool is testing itself.
>>>>>
>>>>>
>>>>>
>>>>>>>>> So I think there are three things we want to do (adopting the
>>>>>>>>> terminology that came from the discussion with Tim and Leo ) :
>>>>>>>>>
>>>>>>>>> 1) implementation tests
>>>>>>>>> 2) spec/API tests (I'll bundle together)
>>>>>>>>> 3) integration/functional tests
>>>>>>>>>
>>>>>>>>> I believe that for #1, the issues related to being on the
>>>>>>>>> bootclasspath don't matter, because we aren't testing that aspect
>>>>>>>>> of the classes (which is how they behave integrated w/ the VM and
>>>>>>>>> security system) but rather the basic internal functioning.
>>>>>>>>>
>>>>>>>>> I'm not sure how to approach this, but I'll try. I'd love to
>>>>>>>>> hear
>>>>>>>>> how Sun, IBM or BEA deals with this, or be told why it isn't an
>>>>>>>>> issue :)
>>>>>>>>>
>>>>>>>>> Implementation tests : I'd like to see us be able to do #1 via
>>>>>>>>> the
>>>>>>>>> standard same-package technique (i.e. testing a.b.C w/ a.b.CTest)
>>>>>>>>> but we'll run into a tangle of classloader problems, I suspect,
>>>>>>>>> becuase we want to be testing java.* code in a system that
>>>>>>>>> already
>>>>>>>>> has java.* code. Can anyone see a way we can do this - test the
>>>>>>>>> classlibrary from the integration point of view - using some test
>>>>>>>>> harness + any known-good JRE, like Sun's or IBM's?
>>>>>>>>>
>>>>>>>> Ew, that won't work in the end since we should assume our own JRE
>>>>>>>> is going
>>>>>>>> to be "known-better" :-). But it might be a nice way to
>>>>>>>> "bootstrap"
>>>>>>>> (eg
>>>>>>>> we test with an external JRE until we satisfy the tests and
>>>>>>>> then we
>>>>>>>> switch
>>>>>>>> to testing with an earlier build).
>>>>>>>>
>>>>>>> Lets be clear - even using our own "earlier build" doesn't solve
>>>>>>> the
>>>>>>> problem I'm describing, because as it stands now, we don't use
>>>>>>> "earlier build" classes to test with - we use the code we want to
>>>>>>> test as the clsaslibrary for the JRE that's running the test
>>>>>>> framework.
>>>>>>>
>>>>>>> The classes that we are testing are also the classes used by the
>>>>>>> testing framework. IOW, any of the java.* classes that JUnit
>>>>>>> itself
>>>>>>> needs (ex. java.util.HashMap) are exactly the same implementation
>>>>>>> that it's testing.
>>>>>>>
>>>>>>> That's why I think it's subtly different than a "bootstrap and use
>>>>>>> version - 1 to test" problem. See what I mean?
>>>>>>>
>>>>>> Yeah yeah, I was already way beyond thinking "just" JUnit is usable
>>>>>> for the
>>>>>> kind of test you're describing. At some point, fundamentally, you
>>>>>> either trust
>>>>>> something external (whether its the sun jdk or the intel compiler
>>>>>> designers,
>>>>>> at some point you do draw a line) or you find a way to bootstrap.
>>>>>>
>>>>> Well, we do trust the Sun JDK.
>>>>>
>>>>>
>>>>>>> I'm very open to the idea that I'm missing something here, but I'd
>>>>>>> like to know that you see the issue - that when we test, we have
>>>>>>>
>>>>>>> VM + "classlib to be tested" + JUnit + testcases
>>>>>>>
>>>>>>> where the testcases are testing the classlib the VM is running
>>>>>>> JUnit
>>>>>>> with.
>>>>>>>
>>>>>>> There never is isolation of the code being tested :
>>>>>>>
>>>>>>> VM + "known good classlib" + Junit + testcases
>>>>>>>
>>>>>>> unless we have some framework where
>>>>>>>
>>>>>>> VM + "known good classlib" + JUnit
>>>>>>> + framework("classlib to be tested")
>>>>>>> + testcases
>>>>>>>
>>>>>>> and it's that notion of "framework()" that I'm advocating we
>>>>>>> explore.
>>>>>>>
>>>>>> I'm all for exploring it, I just fundamentally don't buy into the
>>>>>> "known
>>>>>> good" bit. What happens when the 'classlib to be tested' is 'known
>>>>>> better' than the 'known good' one? How do you define "known"? How
>>>>>> do you
>>>>>> define "good"?
>>>>>>
>>>>> Known? Passed some set of tests. So it could be the Sun JDK for the
>>>>> VM + "known good" part.
>>>>>
>>>>> I think you intuitively understand this. When you find a bug in code
>>>>> you are testing, you first assume it's your code, not the framework,
>>>>> right? In our case, our framework is actually the code we are
>>>>> testing, so we have a bit of a logical conundrum.
>>>>>
>>>>>
>>>> Hi Geir,
>>>>
>>>> The number of Harmony public API classes that get loaded just to
>>>> run the
>>>> JUnit harness is a little over 200. The majority of these are out of
>>>> LUNI with a very low number coming from each of Security, NIO, Archive
>>>> and Text.
>>>>
>>>> Sure there is a circular dependency between what we are building
>>>> and the
>>>> framework we are using to test it but it appears to touch on only a
>>>> relatively small part of Harmony....IMHO.
>>>>
>>>> Best regards,
>>>> George
>>>>
>>>>
>>>>
>>>>>>>> Further ideas...
>>>>>>>>
>>>>>>>> -> look at how the native world does testing
>>>>>>>> (hint: it usually has #ifdefs, uses perl along the way, and
>>>>>>>> it is
>>>>>>>> certainly
>>>>>>>> "messy")
>>>>>>>> -> emulate that
>>>>>>>>
>>>>>>>> -> build a bigger, better specification test
>>>>>>>> -> and somehow "prove" it is "good enough"
>>>>>>>>
>>>>>>>> -> build a bigger, better integration test
>>>>>>>> -> and somehow "prove" it is "good enough"
>>>>>>>>
>>>>>>>> I'll admit my primary interest is the last one...
>>>>>>>>
>>>>>>> The problem I see with the last one is that the "parameter
>>>>>>> space" is
>>>>>>> *huge*.
>>>>>>>
>>>>>> Yeah, that's one of the things that makes it interesting.
>>>>>> Fortunately
>>>>>> open source does have many monkeys...
>>>>>>
>>>>>>
>>>>>>> I believe that your preference for the last one comes from the
>>>>>>> Monte-Carlo style approach that Gump uses - hope that your test
>>>>>>> suite has enough variance that you "push" the thing being tested
>>>>>>> through enough of the parameter space that you can be comfortable
>>>>>>> you would have exposed the bugs. Maybe.
>>>>>>>
>>>>>> Ooh, now its becoming rather abstract...
>>>>>>
>>>>>> Well, perhaps, but more of the gump approache comes from the idea
>>>>>> that
>>>>>> the parameter space itself is also at some point defined in
>>>>>> software,
>>>>>> which may have bugs of its own. You circumvent that by making
>>>>>> humans the
>>>>>> parameter space (don't start about how humans are buggy. We don't
>>>>>> want to
>>>>>> get into existialism or faith systems when talking about unit
>>>>>> testing do
>>>>>> we?). The thing that gump enables is "many monkey QA" - a way for
>>>>>> thousands
>>>>>> of human beings to concurrently make shared assertions about
>>>>>> software
>>>>>> without actually needing all that much human interaction.
>>>>>>
>>>>>> More concretely, if harmony can run all known java software, and run
>>>>>> it to
>>>>>> the asserted satisfaction of all its developers, you can trust that
>>>>>> you have
>>>>>> covered all the /relevant/ parts of the parameter space you
>>>>>> describe.
>>>>>>
>>>>> Yes. And when you can run all knownn Java software, let me know :)
>>>>> That's my point about the parameter space being huge. Even when you
>>>>> reduce the definition to "that of all known Java software", you still
>>>>> have a huge problem on your hands.
>>>>>
>>>>>
>>>>>> You
>>>>>> will never get that level of trust when the assertions are made by
>>>>>> software
>>>>>> rather than humans. This is how open source leads to software
>>>>>> quality.
>>>>>>
>>>>>> Quoting myself, 'gump is the most misunderstood piece of software,
>>>>>> ever'.
>>>>>>
>>>>>> cheers,
>>>>>>
>>>>>> Leo
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>
--
Paulex Yang
China Software Development Lab
IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Oliver Deakin <ol...@googlemail.com>.
George Harley wrote:
> Mikhail Loenko wrote:
>> Hello
>>
>> I'd like to bring this thread back.
>>
>> Number of tests is growing and it is time to put them in order.
>>
>> So far we may have:
>>
>> 1) implementation-specific tests that designed to be run from
>> bootclasspath
>> 2) implementation-specific tests that might be run from classpath
>> 3) implementation-specific tests that designed to be run from classpath
>> 4) implementation-independent tests that designed to be run from
>> bootclasspath
>> 5) implementation-independent tests that might be run from classpath
>> 6) implementation-independent tests that designed to be run from
>> classpath
>>
>> Also we seem to have the following packages, where the tests are:
>>
>> 1) the same package as implementation
>> 2) org.apache.harmony.tests.[the same package as implementation]
>> 3) tests.api.[the same package as implementation]
>>
>> I suggest that we work out step-by-step solution as we could not reach
>> an agreement for the general universal one
>>
>> So for the first step I suggest that we separate i-independent tests
>> that must or may be run from classpath
>>
>> I suggest that we put them into package
>> tests.module.compatible.[package of implementation being tested]
>>
>> Comments?
>>
>> Thanks,
>> Mikhail
>>
>
> Hi Mikhail,
>
> I've just started working through the modules to merge test packages
> "tests.api.[same package as implementation]" and "tests.api.[same
> package as implementation]" into one package space. Using the class
> library package naming guidelines from off the web site [1], all of
> the tests for the text module have been consolidated under
> org.apache.harmony.text.tests.[package under test].
>
> Of course, the text module has only "implementation-independent tests
> that designed to be run from classpath". For modules that have got
> implementation-specific tests then I suppose we could use something
> like "org.apache.harmony.[module].tests.impl.[package under test]" or
> "org.apache.harmony.[module].tests.internal.[package under test]" etc.
> I've got no preference.
I think impl is preferable over internal here, as we already use
internal in our implementation package names to indicate classes totally
internal to that bundle. To also use internal to label tests that are
implementation specific may cause confusion.
Regards,
Oliver
>
> Best regards,
> George
>
>
> [1]
> http://incubator.apache.org/harmony/subcomponents/classlibrary/pkgnaming.html
>
>
>
>>
>> 2006/3/24, George Harley <ge...@googlemail.com>:
>>
>>> Geir Magnusson Jr wrote:
>>>
>>>> Leo Simons wrote:
>>>>
>>>>> On Wed, Mar 22, 2006 at 08:02:44AM -0500, Geir Magnusson Jr wrote:
>>>>>
>>>>>> Leo Simons wrote:
>>>>>>
>>>>>>> On Wed, Mar 22, 2006 at 07:15:28AM -0500, Geir Magnusson Jr wrote:
>>>>>>>
>>>>>>>> Pulling out of the various threads where we have been discussing,
>>>>>>>> can we agree on the problem :
>>>>>>>>
>>>>>>>> We have unique problems compared to other Java projects because we
>>>>>>>> need to find a way to reliably test the things that are commonly
>>>>>>>> expected to be a solid point of reference - namely the core class
>>>>>>>> library.
>>>>>>>>
>>>>>>>> Further, we've been implicitly doing "integration testing" because
>>>>>>>> - so far - the only way we've been testing our code has been 'in
>>>>>>>> situ' in the VM - not in an isolated test harness. To me, this
>>>>>>>> turns it into an integration test.
>>>>>>>>
>>>>>>>> Sure, we're using JUnit, but because of the fact we are
>>>>>>>> implmenting core java.* APIs, we aren't testing with a framework
>>>>>>>> that has been independently tested for correctness, like we would
>>>>>>>> when testing any other code.
>>>>>>>>
>>>>>>>> I hope I got that idea across - I believe that we have to go
>>>>>>>> beyond normal testing approaches because we don't have a normal
>>>>>>>> situation.
>>>>>>>>
>>>>>>> Where we define 'normal situation' as "running a test framework on
>>>>>>> top of
>>>>>>> the sun jdk and expecting any bugs to not be in that jdk". There's
>>>>>>> plenty
>>>>>>> of projects out there that have to test things without having
>>>>>>> such a
>>>>>>> "stable reference JDK" luxury.....I imagine that testing GCC is
>>>>>>> just as
>>>>>>> hard as this problem we have here :-)
>>>>>>>
>>>>>> Is it the same? We need to have a running JVM+classlibarary to test
>>>>>> the classlibrary code.
>>>>>>
>>>>> Well you need a working C compiler and standard C library to
>>>>> compile the
>>>>> compiler so you can compile make so you can build bash so you can run
>>>>> perl (which uses the standard C library functions all over the
>>>>> place of
>>>>> course) so you can run the standard C library tests so that you know
>>>>> that
>>>>> the library you used when compiling the compiler were correct so
>>>>> you can
>>>>> run the compiler tests. I don't think they actually do things that
>>>>> way, but
>>>>> it seems like basically the same problem. Having a virtual machine
>>>>> just
>>>>> makes it easier since you still assume "the native world" as a
>>>>> baseline,
>>>>> which is a lot more than "the hardware".
>>>>>
>>>> There's a difference. You can use a completely separate toolchain to
>>>> build, test and verify the output of the C compiler.
>>>>
>>>> In our case, we are using the thing we are testing to test itself.
>>>> There is no "known good" element possible right now.
>>>>
>>>> We use the classlibrary we are trying to test to execute the test
>>>> framework that tests the classlibrary that is running it.
>>>>
>>>> The tool is testing itself.
>>>>
>>>>
>>>>
>>>>>>>> So I think there are three things we want to do (adopting the
>>>>>>>> terminology that came from the discussion with Tim and Leo ) :
>>>>>>>>
>>>>>>>> 1) implementation tests
>>>>>>>> 2) spec/API tests (I'll bundle together)
>>>>>>>> 3) integration/functional tests
>>>>>>>>
>>>>>>>> I believe that for #1, the issues related to being on the
>>>>>>>> bootclasspath don't matter, because we aren't testing that aspect
>>>>>>>> of the classes (which is how they behave integrated w/ the VM and
>>>>>>>> security system) but rather the basic internal functioning.
>>>>>>>>
>>>>>>>> I'm not sure how to approach this, but I'll try. I'd love to hear
>>>>>>>> how Sun, IBM or BEA deals with this, or be told why it isn't an
>>>>>>>> issue :)
>>>>>>>>
>>>>>>>> Implementation tests : I'd like to see us be able to do #1 via the
>>>>>>>> standard same-package technique (i.e. testing a.b.C w/ a.b.CTest)
>>>>>>>> but we'll run into a tangle of classloader problems, I suspect,
>>>>>>>> becuase we want to be testing java.* code in a system that already
>>>>>>>> has java.* code. Can anyone see a way we can do this - test the
>>>>>>>> classlibrary from the integration point of view - using some test
>>>>>>>> harness + any known-good JRE, like Sun's or IBM's?
>>>>>>>>
>>>>>>> Ew, that won't work in the end since we should assume our own JRE
>>>>>>> is going
>>>>>>> to be "known-better" :-). But it might be a nice way to "bootstrap"
>>>>>>> (eg
>>>>>>> we test with an external JRE until we satisfy the tests and then we
>>>>>>> switch
>>>>>>> to testing with an earlier build).
>>>>>>>
>>>>>> Lets be clear - even using our own "earlier build" doesn't solve the
>>>>>> problem I'm describing, because as it stands now, we don't use
>>>>>> "earlier build" classes to test with - we use the code we want to
>>>>>> test as the clsaslibrary for the JRE that's running the test
>>>>>> framework.
>>>>>>
>>>>>> The classes that we are testing are also the classes used by the
>>>>>> testing framework. IOW, any of the java.* classes that JUnit itself
>>>>>> needs (ex. java.util.HashMap) are exactly the same implementation
>>>>>> that it's testing.
>>>>>>
>>>>>> That's why I think it's subtly different than a "bootstrap and use
>>>>>> version - 1 to test" problem. See what I mean?
>>>>>>
>>>>> Yeah yeah, I was already way beyond thinking "just" JUnit is usable
>>>>> for the
>>>>> kind of test you're describing. At some point, fundamentally, you
>>>>> either trust
>>>>> something external (whether its the sun jdk or the intel compiler
>>>>> designers,
>>>>> at some point you do draw a line) or you find a way to bootstrap.
>>>>>
>>>> Well, we do trust the Sun JDK.
>>>>
>>>>
>>>>>> I'm very open to the idea that I'm missing something here, but I'd
>>>>>> like to know that you see the issue - that when we test, we have
>>>>>>
>>>>>> VM + "classlib to be tested" + JUnit + testcases
>>>>>>
>>>>>> where the testcases are testing the classlib the VM is running JUnit
>>>>>> with.
>>>>>>
>>>>>> There never is isolation of the code being tested :
>>>>>>
>>>>>> VM + "known good classlib" + Junit + testcases
>>>>>>
>>>>>> unless we have some framework where
>>>>>>
>>>>>> VM + "known good classlib" + JUnit
>>>>>> + framework("classlib to be tested")
>>>>>> + testcases
>>>>>>
>>>>>> and it's that notion of "framework()" that I'm advocating we
>>>>>> explore.
>>>>>>
>>>>> I'm all for exploring it, I just fundamentally don't buy into the
>>>>> "known
>>>>> good" bit. What happens when the 'classlib to be tested' is 'known
>>>>> better' than the 'known good' one? How do you define "known"? How
>>>>> do you
>>>>> define "good"?
>>>>>
>>>> Known? Passed some set of tests. So it could be the Sun JDK for the
>>>> VM + "known good" part.
>>>>
>>>> I think you intuitively understand this. When you find a bug in code
>>>> you are testing, you first assume it's your code, not the framework,
>>>> right? In our case, our framework is actually the code we are
>>>> testing, so we have a bit of a logical conundrum.
>>>>
>>>>
>>> Hi Geir,
>>>
>>> The number of Harmony public API classes that get loaded just to run
>>> the
>>> JUnit harness is a little over 200. The majority of these are out of
>>> LUNI with a very low number coming from each of Security, NIO, Archive
>>> and Text.
>>>
>>> Sure there is a circular dependency between what we are building and
>>> the
>>> framework we are using to test it but it appears to touch on only a
>>> relatively small part of Harmony....IMHO.
>>>
>>> Best regards,
>>> George
>>>
>>>
>>>
>>>>>>> Further ideas...
>>>>>>>
>>>>>>> -> look at how the native world does testing
>>>>>>> (hint: it usually has #ifdefs, uses perl along the way, and it is
>>>>>>> certainly
>>>>>>> "messy")
>>>>>>> -> emulate that
>>>>>>>
>>>>>>> -> build a bigger, better specification test
>>>>>>> -> and somehow "prove" it is "good enough"
>>>>>>>
>>>>>>> -> build a bigger, better integration test
>>>>>>> -> and somehow "prove" it is "good enough"
>>>>>>>
>>>>>>> I'll admit my primary interest is the last one...
>>>>>>>
>>>>>> The problem I see with the last one is that the "parameter space" is
>>>>>> *huge*.
>>>>>>
>>>>> Yeah, that's one of the things that makes it interesting. Fortunately
>>>>> open source does have many monkeys...
>>>>>
>>>>>
>>>>>> I believe that your preference for the last one comes from the
>>>>>> Monte-Carlo style approach that Gump uses - hope that your test
>>>>>> suite has enough variance that you "push" the thing being tested
>>>>>> through enough of the parameter space that you can be comfortable
>>>>>> you would have exposed the bugs. Maybe.
>>>>>>
>>>>> Ooh, now its becoming rather abstract...
>>>>>
>>>>> Well, perhaps, but more of the gump approache comes from the idea
>>>>> that
>>>>> the parameter space itself is also at some point defined in software,
>>>>> which may have bugs of its own. You circumvent that by making
>>>>> humans the
>>>>> parameter space (don't start about how humans are buggy. We don't
>>>>> want to
>>>>> get into existialism or faith systems when talking about unit
>>>>> testing do
>>>>> we?). The thing that gump enables is "many monkey QA" - a way for
>>>>> thousands
>>>>> of human beings to concurrently make shared assertions about software
>>>>> without actually needing all that much human interaction.
>>>>>
>>>>> More concretely, if harmony can run all known java software, and run
>>>>> it to
>>>>> the asserted satisfaction of all its developers, you can trust that
>>>>> you have
>>>>> covered all the /relevant/ parts of the parameter space you describe.
>>>>>
>>>> Yes. And when you can run all knownn Java software, let me know :)
>>>> That's my point about the parameter space being huge. Even when you
>>>> reduce the definition to "that of all known Java software", you still
>>>> have a huge problem on your hands.
>>>>
>>>>
>>>>> You
>>>>> will never get that level of trust when the assertions are made by
>>>>> software
>>>>> rather than humans. This is how open source leads to software
>>>>> quality.
>>>>>
>>>>> Quoting myself, 'gump is the most misunderstood piece of software,
>>>>> ever'.
>>>>>
>>>>> cheers,
>>>>>
>>>>> Leo
>>>>>
>>>>>
>>>>>
>>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
--
Oliver Deakin
IBM United Kingdom Limited
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by George Harley <ge...@googlemail.com>.
Mikhail Loenko wrote:
> Hello
>
> I'd like to bring this thread back.
>
> Number of tests is growing and it is time to put them in order.
>
> So far we may have:
>
> 1) implementation-specific tests that designed to be run from bootclasspath
> 2) implementation-specific tests that might be run from classpath
> 3) implementation-specific tests that designed to be run from classpath
> 4) implementation-independent tests that designed to be run from bootclasspath
> 5) implementation-independent tests that might be run from classpath
> 6) implementation-independent tests that designed to be run from classpath
>
> Also we seem to have the following packages, where the tests are:
>
> 1) the same package as implementation
> 2) org.apache.harmony.tests.[the same package as implementation]
> 3) tests.api.[the same package as implementation]
>
> I suggest that we work out step-by-step solution as we could not reach
> an agreement for the general universal one
>
> So for the first step I suggest that we separate i-independent tests
> that must or may be run from classpath
>
> I suggest that we put them into package
> tests.module.compatible.[package of implementation being tested]
>
> Comments?
>
> Thanks,
> Mikhail
>
Hi Mikhail,
I've just started working through the modules to merge test packages
"tests.api.[same package as implementation]" and "tests.api.[same
package as implementation]" into one package space. Using the class
library package naming guidelines from off the web site [1], all of the
tests for the text module have been consolidated under
org.apache.harmony.text.tests.[package under test].
Of course, the text module has only "implementation-independent tests
that designed to be run from classpath". For modules that have got
implementation-specific tests then I suppose we could use something like
"org.apache.harmony.[module].tests.impl.[package under test]" or
"org.apache.harmony.[module].tests.internal.[package under test]" etc.
I've got no preference.
Best regards,
George
[1]
http://incubator.apache.org/harmony/subcomponents/classlibrary/pkgnaming.html
>
> 2006/3/24, George Harley <ge...@googlemail.com>:
>
>> Geir Magnusson Jr wrote:
>>
>>> Leo Simons wrote:
>>>
>>>> On Wed, Mar 22, 2006 at 08:02:44AM -0500, Geir Magnusson Jr wrote:
>>>>
>>>>> Leo Simons wrote:
>>>>>
>>>>>> On Wed, Mar 22, 2006 at 07:15:28AM -0500, Geir Magnusson Jr wrote:
>>>>>>
>>>>>>> Pulling out of the various threads where we have been discussing,
>>>>>>> can we agree on the problem :
>>>>>>>
>>>>>>> We have unique problems compared to other Java projects because we
>>>>>>> need to find a way to reliably test the things that are commonly
>>>>>>> expected to be a solid point of reference - namely the core class
>>>>>>> library.
>>>>>>>
>>>>>>> Further, we've been implicitly doing "integration testing" because
>>>>>>> - so far - the only way we've been testing our code has been 'in
>>>>>>> situ' in the VM - not in an isolated test harness. To me, this
>>>>>>> turns it into an integration test.
>>>>>>>
>>>>>>> Sure, we're using JUnit, but because of the fact we are
>>>>>>> implmenting core java.* APIs, we aren't testing with a framework
>>>>>>> that has been independently tested for correctness, like we would
>>>>>>> when testing any other code.
>>>>>>>
>>>>>>> I hope I got that idea across - I believe that we have to go
>>>>>>> beyond normal testing approaches because we don't have a normal
>>>>>>> situation.
>>>>>>>
>>>>>> Where we define 'normal situation' as "running a test framework on
>>>>>> top of
>>>>>> the sun jdk and expecting any bugs to not be in that jdk". There's
>>>>>> plenty
>>>>>> of projects out there that have to test things without having such a
>>>>>> "stable reference JDK" luxury.....I imagine that testing GCC is
>>>>>> just as
>>>>>> hard as this problem we have here :-)
>>>>>>
>>>>> Is it the same? We need to have a running JVM+classlibarary to test
>>>>> the classlibrary code.
>>>>>
>>>> Well you need a working C compiler and standard C library to compile the
>>>> compiler so you can compile make so you can build bash so you can run
>>>> perl (which uses the standard C library functions all over the place of
>>>> course) so you can run the standard C library tests so that you know
>>>> that
>>>> the library you used when compiling the compiler were correct so you can
>>>> run the compiler tests. I don't think they actually do things that
>>>> way, but
>>>> it seems like basically the same problem. Having a virtual machine just
>>>> makes it easier since you still assume "the native world" as a baseline,
>>>> which is a lot more than "the hardware".
>>>>
>>> There's a difference. You can use a completely separate toolchain to
>>> build, test and verify the output of the C compiler.
>>>
>>> In our case, we are using the thing we are testing to test itself.
>>> There is no "known good" element possible right now.
>>>
>>> We use the classlibrary we are trying to test to execute the test
>>> framework that tests the classlibrary that is running it.
>>>
>>> The tool is testing itself.
>>>
>>>
>>>
>>>>>>> So I think there are three things we want to do (adopting the
>>>>>>> terminology that came from the discussion with Tim and Leo ) :
>>>>>>>
>>>>>>> 1) implementation tests
>>>>>>> 2) spec/API tests (I'll bundle together)
>>>>>>> 3) integration/functional tests
>>>>>>>
>>>>>>> I believe that for #1, the issues related to being on the
>>>>>>> bootclasspath don't matter, because we aren't testing that aspect
>>>>>>> of the classes (which is how they behave integrated w/ the VM and
>>>>>>> security system) but rather the basic internal functioning.
>>>>>>>
>>>>>>> I'm not sure how to approach this, but I'll try. I'd love to hear
>>>>>>> how Sun, IBM or BEA deals with this, or be told why it isn't an
>>>>>>> issue :)
>>>>>>>
>>>>>>> Implementation tests : I'd like to see us be able to do #1 via the
>>>>>>> standard same-package technique (i.e. testing a.b.C w/ a.b.CTest)
>>>>>>> but we'll run into a tangle of classloader problems, I suspect,
>>>>>>> becuase we want to be testing java.* code in a system that already
>>>>>>> has java.* code. Can anyone see a way we can do this - test the
>>>>>>> classlibrary from the integration point of view - using some test
>>>>>>> harness + any known-good JRE, like Sun's or IBM's?
>>>>>>>
>>>>>> Ew, that won't work in the end since we should assume our own JRE
>>>>>> is going
>>>>>> to be "known-better" :-). But it might be a nice way to "bootstrap"
>>>>>> (eg
>>>>>> we test with an external JRE until we satisfy the tests and then we
>>>>>> switch
>>>>>> to testing with an earlier build).
>>>>>>
>>>>> Lets be clear - even using our own "earlier build" doesn't solve the
>>>>> problem I'm describing, because as it stands now, we don't use
>>>>> "earlier build" classes to test with - we use the code we want to
>>>>> test as the clsaslibrary for the JRE that's running the test framework.
>>>>>
>>>>> The classes that we are testing are also the classes used by the
>>>>> testing framework. IOW, any of the java.* classes that JUnit itself
>>>>> needs (ex. java.util.HashMap) are exactly the same implementation
>>>>> that it's testing.
>>>>>
>>>>> That's why I think it's subtly different than a "bootstrap and use
>>>>> version - 1 to test" problem. See what I mean?
>>>>>
>>>> Yeah yeah, I was already way beyond thinking "just" JUnit is usable
>>>> for the
>>>> kind of test you're describing. At some point, fundamentally, you
>>>> either trust
>>>> something external (whether its the sun jdk or the intel compiler
>>>> designers,
>>>> at some point you do draw a line) or you find a way to bootstrap.
>>>>
>>> Well, we do trust the Sun JDK.
>>>
>>>
>>>>> I'm very open to the idea that I'm missing something here, but I'd
>>>>> like to know that you see the issue - that when we test, we have
>>>>>
>>>>> VM + "classlib to be tested" + JUnit + testcases
>>>>>
>>>>> where the testcases are testing the classlib the VM is running JUnit
>>>>> with.
>>>>>
>>>>> There never is isolation of the code being tested :
>>>>>
>>>>> VM + "known good classlib" + Junit + testcases
>>>>>
>>>>> unless we have some framework where
>>>>>
>>>>> VM + "known good classlib" + JUnit
>>>>> + framework("classlib to be tested")
>>>>> + testcases
>>>>>
>>>>> and it's that notion of "framework()" that I'm advocating we explore.
>>>>>
>>>> I'm all for exploring it, I just fundamentally don't buy into the "known
>>>> good" bit. What happens when the 'classlib to be tested' is 'known
>>>> better' than the 'known good' one? How do you define "known"? How do you
>>>> define "good"?
>>>>
>>> Known? Passed some set of tests. So it could be the Sun JDK for the
>>> VM + "known good" part.
>>>
>>> I think you intuitively understand this. When you find a bug in code
>>> you are testing, you first assume it's your code, not the framework,
>>> right? In our case, our framework is actually the code we are
>>> testing, so we have a bit of a logical conundrum.
>>>
>>>
>> Hi Geir,
>>
>> The number of Harmony public API classes that get loaded just to run the
>> JUnit harness is a little over 200. The majority of these are out of
>> LUNI with a very low number coming from each of Security, NIO, Archive
>> and Text.
>>
>> Sure there is a circular dependency between what we are building and the
>> framework we are using to test it but it appears to touch on only a
>> relatively small part of Harmony....IMHO.
>>
>> Best regards,
>> George
>>
>>
>>
>>>>>> Further ideas...
>>>>>>
>>>>>> -> look at how the native world does testing
>>>>>> (hint: it usually has #ifdefs, uses perl along the way, and it is
>>>>>> certainly
>>>>>> "messy")
>>>>>> -> emulate that
>>>>>>
>>>>>> -> build a bigger, better specification test
>>>>>> -> and somehow "prove" it is "good enough"
>>>>>>
>>>>>> -> build a bigger, better integration test
>>>>>> -> and somehow "prove" it is "good enough"
>>>>>>
>>>>>> I'll admit my primary interest is the last one...
>>>>>>
>>>>> The problem I see with the last one is that the "parameter space" is
>>>>> *huge*.
>>>>>
>>>> Yeah, that's one of the things that makes it interesting. Fortunately
>>>> open source does have many monkeys...
>>>>
>>>>
>>>>> I believe that your preference for the last one comes from the
>>>>> Monte-Carlo style approach that Gump uses - hope that your test
>>>>> suite has enough variance that you "push" the thing being tested
>>>>> through enough of the parameter space that you can be comfortable
>>>>> you would have exposed the bugs. Maybe.
>>>>>
>>>> Ooh, now its becoming rather abstract...
>>>>
>>>> Well, perhaps, but more of the gump approache comes from the idea that
>>>> the parameter space itself is also at some point defined in software,
>>>> which may have bugs of its own. You circumvent that by making humans the
>>>> parameter space (don't start about how humans are buggy. We don't
>>>> want to
>>>> get into existialism or faith systems when talking about unit testing do
>>>> we?). The thing that gump enables is "many monkey QA" - a way for
>>>> thousands
>>>> of human beings to concurrently make shared assertions about software
>>>> without actually needing all that much human interaction.
>>>>
>>>> More concretely, if harmony can run all known java software, and run
>>>> it to
>>>> the asserted satisfaction of all its developers, you can trust that
>>>> you have
>>>> covered all the /relevant/ parts of the parameter space you describe.
>>>>
>>> Yes. And when you can run all knownn Java software, let me know :)
>>> That's my point about the parameter space being huge. Even when you
>>> reduce the definition to "that of all known Java software", you still
>>> have a huge problem on your hands.
>>>
>>>
>>>> You
>>>> will never get that level of trust when the assertions are made by
>>>> software
>>>> rather than humans. This is how open source leads to software quality.
>>>>
>>>> Quoting myself, 'gump is the most misunderstood piece of software,
>>>> ever'.
>>>>
>>>> cheers,
>>>>
>>>> Leo
>>>>
>>>>
>>>>
>>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Mikhail Loenko <ml...@gmail.com>.
2006/4/27, Geir Magnusson Jr <ge...@pobox.com>:
>
>
> Mikhail Loenko wrote:
> > Hello
> >
> > I'd like to bring this thread back.
> >
> > Number of tests is growing and it is time to put them in order.
> >
> > So far we may have:
> >
> > 1) implementation-specific tests that designed to be run from bootclasspath
> > 2) implementation-specific tests that might be run from classpath
> > 3) implementation-specific tests that designed to be run from classpath
> > 4) implementation-independent tests that designed to be run from bootclasspath
> > 5) implementation-independent tests that might be run from classpath
> > 6) implementation-independent tests that designed to be run from classpath
>
> I think there is no "might".
"might" means that they will work well both from classpath and from
bootclasspath
Thanks,
Mikhail
>
> >
> > Also we seem to have the following packages, where the tests are:
> >
> > 1) the same package as implementation
> > 2) org.apache.harmony.tests.[the same package as implementation]
> > 3) tests.api.[the same package as implementation]
> >
> > I suggest that we work out step-by-step solution as we could not reach
> > an agreement for the general universal one
>
> yes!
>
> >
> > So for the first step I suggest that we separate i-independent tests
> > that must or may be run from classpath
>
> yes!
>
> >
> > I suggest that we put them into package
> > tests.module.compatible.[package of implementation being tested]
>
> No!
>
> I think we should keep org.apache.harmony as the prefix for the package
> namespace so we don't collide or get confused with others...
>
> geir
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Geir Magnusson Jr <ge...@pobox.com>.
Mikhail Loenko wrote:
> Hello
>
> I'd like to bring this thread back.
>
> Number of tests is growing and it is time to put them in order.
>
> So far we may have:
>
> 1) implementation-specific tests that designed to be run from bootclasspath
> 2) implementation-specific tests that might be run from classpath
> 3) implementation-specific tests that designed to be run from classpath
> 4) implementation-independent tests that designed to be run from bootclasspath
> 5) implementation-independent tests that might be run from classpath
> 6) implementation-independent tests that designed to be run from classpath
I think there is no "might".
>
> Also we seem to have the following packages, where the tests are:
>
> 1) the same package as implementation
> 2) org.apache.harmony.tests.[the same package as implementation]
> 3) tests.api.[the same package as implementation]
>
> I suggest that we work out step-by-step solution as we could not reach
> an agreement for the general universal one
yes!
>
> So for the first step I suggest that we separate i-independent tests
> that must or may be run from classpath
yes!
>
> I suggest that we put them into package
> tests.module.compatible.[package of implementation being tested]
No!
I think we should keep org.apache.harmony as the prefix for the package
namespace so we don't collide or get confused with others...
geir
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by Mikhail Loenko <ml...@gmail.com>.
Hello
I'd like to bring this thread back.
Number of tests is growing and it is time to put them in order.
So far we may have:
1) implementation-specific tests that designed to be run from bootclasspath
2) implementation-specific tests that might be run from classpath
3) implementation-specific tests that designed to be run from classpath
4) implementation-independent tests that designed to be run from bootclasspath
5) implementation-independent tests that might be run from classpath
6) implementation-independent tests that designed to be run from classpath
Also we seem to have the following packages, where the tests are:
1) the same package as implementation
2) org.apache.harmony.tests.[the same package as implementation]
3) tests.api.[the same package as implementation]
I suggest that we work out step-by-step solution as we could not reach
an agreement for the general universal one
So for the first step I suggest that we separate i-independent tests
that must or may be run from classpath
I suggest that we put them into package
tests.module.compatible.[package of implementation being tested]
Comments?
Thanks,
Mikhail
2006/3/24, George Harley <ge...@googlemail.com>:
> Geir Magnusson Jr wrote:
> >
> >
> > Leo Simons wrote:
> >> On Wed, Mar 22, 2006 at 08:02:44AM -0500, Geir Magnusson Jr wrote:
> >>> Leo Simons wrote:
> >>>> On Wed, Mar 22, 2006 at 07:15:28AM -0500, Geir Magnusson Jr wrote:
> >>>>> Pulling out of the various threads where we have been discussing,
> >>>>> can we agree on the problem :
> >>>>>
> >>>>> We have unique problems compared to other Java projects because we
> >>>>> need to find a way to reliably test the things that are commonly
> >>>>> expected to be a solid point of reference - namely the core class
> >>>>> library.
> >>>>>
> >>>>> Further, we've been implicitly doing "integration testing" because
> >>>>> - so far - the only way we've been testing our code has been 'in
> >>>>> situ' in the VM - not in an isolated test harness. To me, this
> >>>>> turns it into an integration test.
> >>>>>
> >>>>> Sure, we're using JUnit, but because of the fact we are
> >>>>> implmenting core java.* APIs, we aren't testing with a framework
> >>>>> that has been independently tested for correctness, like we would
> >>>>> when testing any other code.
> >>>>>
> >>>>> I hope I got that idea across - I believe that we have to go
> >>>>> beyond normal testing approaches because we don't have a normal
> >>>>> situation.
> >>>> Where we define 'normal situation' as "running a test framework on
> >>>> top of
> >>>> the sun jdk and expecting any bugs to not be in that jdk". There's
> >>>> plenty
> >>>> of projects out there that have to test things without having such a
> >>>> "stable reference JDK" luxury.....I imagine that testing GCC is
> >>>> just as
> >>>> hard as this problem we have here :-)
> >>> Is it the same? We need to have a running JVM+classlibarary to test
> >>> the classlibrary code.
> >>
> >> Well you need a working C compiler and standard C library to compile the
> >> compiler so you can compile make so you can build bash so you can run
> >> perl (which uses the standard C library functions all over the place of
> >> course) so you can run the standard C library tests so that you know
> >> that
> >> the library you used when compiling the compiler were correct so you can
> >> run the compiler tests. I don't think they actually do things that
> >> way, but
> >> it seems like basically the same problem. Having a virtual machine just
> >> makes it easier since you still assume "the native world" as a baseline,
> >> which is a lot more than "the hardware".
> >
> > There's a difference. You can use a completely separate toolchain to
> > build, test and verify the output of the C compiler.
> >
> > In our case, we are using the thing we are testing to test itself.
> > There is no "known good" element possible right now.
> >
> > We use the classlibrary we are trying to test to execute the test
> > framework that tests the classlibrary that is running it.
> >
> > The tool is testing itself.
> >
> >
> >>
> >>>>> So I think there are three things we want to do (adopting the
> >>>>> terminology that came from the discussion with Tim and Leo ) :
> >>>>>
> >>>>> 1) implementation tests
> >>>>> 2) spec/API tests (I'll bundle together)
> >>>>> 3) integration/functional tests
> >>>>>
> >>>>> I believe that for #1, the issues related to being on the
> >>>>> bootclasspath don't matter, because we aren't testing that aspect
> >>>>> of the classes (which is how they behave integrated w/ the VM and
> >>>>> security system) but rather the basic internal functioning.
> >>>>>
> >>>>> I'm not sure how to approach this, but I'll try. I'd love to hear
> >>>>> how Sun, IBM or BEA deals with this, or be told why it isn't an
> >>>>> issue :)
> >>>>>
> >>>>> Implementation tests : I'd like to see us be able to do #1 via the
> >>>>> standard same-package technique (i.e. testing a.b.C w/ a.b.CTest)
> >>>>> but we'll run into a tangle of classloader problems, I suspect,
> >>>>> becuase we want to be testing java.* code in a system that already
> >>>>> has java.* code. Can anyone see a way we can do this - test the
> >>>>> classlibrary from the integration point of view - using some test
> >>>>> harness + any known-good JRE, like Sun's or IBM's?
> >>>> Ew, that won't work in the end since we should assume our own JRE
> >>>> is going
> >>>> to be "known-better" :-). But it might be a nice way to "bootstrap"
> >>>> (eg
> >>>> we test with an external JRE until we satisfy the tests and then we
> >>>> switch
> >>>> to testing with an earlier build).
> >>> Lets be clear - even using our own "earlier build" doesn't solve the
> >>> problem I'm describing, because as it stands now, we don't use
> >>> "earlier build" classes to test with - we use the code we want to
> >>> test as the clsaslibrary for the JRE that's running the test framework.
> >>>
> >>> The classes that we are testing are also the classes used by the
> >>> testing framework. IOW, any of the java.* classes that JUnit itself
> >>> needs (ex. java.util.HashMap) are exactly the same implementation
> >>> that it's testing.
> >>>
> >>> That's why I think it's subtly different than a "bootstrap and use
> >>> version - 1 to test" problem. See what I mean?
> >>
> >> Yeah yeah, I was already way beyond thinking "just" JUnit is usable
> >> for the
> >> kind of test you're describing. At some point, fundamentally, you
> >> either trust
> >> something external (whether its the sun jdk or the intel compiler
> >> designers,
> >> at some point you do draw a line) or you find a way to bootstrap.
> >
> > Well, we do trust the Sun JDK.
> >
> >>
> >>> I'm very open to the idea that I'm missing something here, but I'd
> >>> like to know that you see the issue - that when we test, we have
> >>>
> >>> VM + "classlib to be tested" + JUnit + testcases
> >>>
> >>> where the testcases are testing the classlib the VM is running JUnit
> >>> with.
> >>>
> >>> There never is isolation of the code being tested :
> >>>
> >>> VM + "known good classlib" + Junit + testcases
> >>>
> >>> unless we have some framework where
> >>>
> >>> VM + "known good classlib" + JUnit
> >>> + framework("classlib to be tested")
> >>> + testcases
> >>>
> >>> and it's that notion of "framework()" that I'm advocating we explore.
> >>
> >> I'm all for exploring it, I just fundamentally don't buy into the "known
> >> good" bit. What happens when the 'classlib to be tested' is 'known
> >> better' than the 'known good' one? How do you define "known"? How do you
> >> define "good"?
> >
> > Known? Passed some set of tests. So it could be the Sun JDK for the
> > VM + "known good" part.
> >
> > I think you intuitively understand this. When you find a bug in code
> > you are testing, you first assume it's your code, not the framework,
> > right? In our case, our framework is actually the code we are
> > testing, so we have a bit of a logical conundrum.
> >
>
> Hi Geir,
>
> The number of Harmony public API classes that get loaded just to run the
> JUnit harness is a little over 200. The majority of these are out of
> LUNI with a very low number coming from each of Security, NIO, Archive
> and Text.
>
> Sure there is a circular dependency between what we are building and the
> framework we are using to test it but it appears to touch on only a
> relatively small part of Harmony....IMHO.
>
> Best regards,
> George
>
>
> >>
> >>>> Further ideas...
> >>>>
> >>>> -> look at how the native world does testing
> >>>> (hint: it usually has #ifdefs, uses perl along the way, and it is
> >>>> certainly
> >>>> "messy")
> >>>> -> emulate that
> >>>>
> >>>> -> build a bigger, better specification test
> >>>> -> and somehow "prove" it is "good enough"
> >>>>
> >>>> -> build a bigger, better integration test
> >>>> -> and somehow "prove" it is "good enough"
> >>>>
> >>>> I'll admit my primary interest is the last one...
> >>> The problem I see with the last one is that the "parameter space" is
> >>> *huge*.
> >>
> >> Yeah, that's one of the things that makes it interesting. Fortunately
> >> open source does have many monkeys...
> >>
> >>> I believe that your preference for the last one comes from the
> >>> Monte-Carlo style approach that Gump uses - hope that your test
> >>> suite has enough variance that you "push" the thing being tested
> >>> through enough of the parameter space that you can be comfortable
> >>> you would have exposed the bugs. Maybe.
> >>
> >> Ooh, now its becoming rather abstract...
> >>
> >> Well, perhaps, but more of the gump approache comes from the idea that
> >> the parameter space itself is also at some point defined in software,
> >> which may have bugs of its own. You circumvent that by making humans the
> >> parameter space (don't start about how humans are buggy. We don't
> >> want to
> >> get into existialism or faith systems when talking about unit testing do
> >> we?). The thing that gump enables is "many monkey QA" - a way for
> >> thousands
> >> of human beings to concurrently make shared assertions about software
> >> without actually needing all that much human interaction.
> >>
> >> More concretely, if harmony can run all known java software, and run
> >> it to
> >> the asserted satisfaction of all its developers, you can trust that
> >> you have
> >> covered all the /relevant/ parts of the parameter space you describe.
> >
> > Yes. And when you can run all knownn Java software, let me know :)
> > That's my point about the parameter space being huge. Even when you
> > reduce the definition to "that of all known Java software", you still
> > have a huge problem on your hands.
> >
> >> You
> >> will never get that level of trust when the assertions are made by
> >> software
> >> rather than humans. This is how open source leads to software quality.
> >>
> >> Quoting myself, 'gump is the most misunderstood piece of software,
> >> ever'.
> >>
> >> cheers,
> >>
> >> Leo
> >>
> >>
> >
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib] Testing
Posted by George Harley <ge...@googlemail.com>.
Geir Magnusson Jr wrote:
>
>
> Leo Simons wrote:
>> On Wed, Mar 22, 2006 at 08:02:44AM -0500, Geir Magnusson Jr wrote:
>>> Leo Simons wrote:
>>>> On Wed, Mar 22, 2006 at 07:15:28AM -0500, Geir Magnusson Jr wrote:
>>>>> Pulling out of the various threads where we have been discussing,
>>>>> can we agree on the problem :
>>>>>
>>>>> We have unique problems compared to other Java projects because we
>>>>> need to find a way to reliably test the things that are commonly
>>>>> expected to be a solid point of reference - namely the core class
>>>>> library.
>>>>>
>>>>> Further, we've been implicitly doing "integration testing" because
>>>>> - so far - the only way we've been testing our code has been 'in
>>>>> situ' in the VM - not in an isolated test harness. To me, this
>>>>> turns it into an integration test.
>>>>>
>>>>> Sure, we're using JUnit, but because of the fact we are
>>>>> implmenting core java.* APIs, we aren't testing with a framework
>>>>> that has been independently tested for correctness, like we would
>>>>> when testing any other code.
>>>>>
>>>>> I hope I got that idea across - I believe that we have to go
>>>>> beyond normal testing approaches because we don't have a normal
>>>>> situation.
>>>> Where we define 'normal situation' as "running a test framework on
>>>> top of
>>>> the sun jdk and expecting any bugs to not be in that jdk". There's
>>>> plenty
>>>> of projects out there that have to test things without having such a
>>>> "stable reference JDK" luxury.....I imagine that testing GCC is
>>>> just as
>>>> hard as this problem we have here :-)
>>> Is it the same? We need to have a running JVM+classlibarary to test
>>> the classlibrary code.
>>
>> Well you need a working C compiler and standard C library to compile the
>> compiler so you can compile make so you can build bash so you can run
>> perl (which uses the standard C library functions all over the place of
>> course) so you can run the standard C library tests so that you know
>> that
>> the library you used when compiling the compiler were correct so you can
>> run the compiler tests. I don't think they actually do things that
>> way, but
>> it seems like basically the same problem. Having a virtual machine just
>> makes it easier since you still assume "the native world" as a baseline,
>> which is a lot more than "the hardware".
>
> There's a difference. You can use a completely separate toolchain to
> build, test and verify the output of the C compiler.
>
> In our case, we are using the thing we are testing to test itself.
> There is no "known good" element possible right now.
>
> We use the classlibrary we are trying to test to execute the test
> framework that tests the classlibrary that is running it.
>
> The tool is testing itself.
>
>
>>
>>>>> So I think there are three things we want to do (adopting the
>>>>> terminology that came from the discussion with Tim and Leo ) :
>>>>>
>>>>> 1) implementation tests
>>>>> 2) spec/API tests (I'll bundle together)
>>>>> 3) integration/functional tests
>>>>>
>>>>> I believe that for #1, the issues related to being on the
>>>>> bootclasspath don't matter, because we aren't testing that aspect
>>>>> of the classes (which is how they behave integrated w/ the VM and
>>>>> security system) but rather the basic internal functioning.
>>>>>
>>>>> I'm not sure how to approach this, but I'll try. I'd love to hear
>>>>> how Sun, IBM or BEA deals with this, or be told why it isn't an
>>>>> issue :)
>>>>>
>>>>> Implementation tests : I'd like to see us be able to do #1 via the
>>>>> standard same-package technique (i.e. testing a.b.C w/ a.b.CTest)
>>>>> but we'll run into a tangle of classloader problems, I suspect,
>>>>> becuase we want to be testing java.* code in a system that already
>>>>> has java.* code. Can anyone see a way we can do this - test the
>>>>> classlibrary from the integration point of view - using some test
>>>>> harness + any known-good JRE, like Sun's or IBM's?
>>>> Ew, that won't work in the end since we should assume our own JRE
>>>> is going
>>>> to be "known-better" :-). But it might be a nice way to "bootstrap"
>>>> (eg
>>>> we test with an external JRE until we satisfy the tests and then we
>>>> switch
>>>> to testing with an earlier build).
>>> Lets be clear - even using our own "earlier build" doesn't solve the
>>> problem I'm describing, because as it stands now, we don't use
>>> "earlier build" classes to test with - we use the code we want to
>>> test as the clsaslibrary for the JRE that's running the test framework.
>>>
>>> The classes that we are testing are also the classes used by the
>>> testing framework. IOW, any of the java.* classes that JUnit itself
>>> needs (ex. java.util.HashMap) are exactly the same implementation
>>> that it's testing.
>>>
>>> That's why I think it's subtly different than a "bootstrap and use
>>> version - 1 to test" problem. See what I mean?
>>
>> Yeah yeah, I was already way beyond thinking "just" JUnit is usable
>> for the
>> kind of test you're describing. At some point, fundamentally, you
>> either trust
>> something external (whether its the sun jdk or the intel compiler
>> designers,
>> at some point you do draw a line) or you find a way to bootstrap.
>
> Well, we do trust the Sun JDK.
>
>>
>>> I'm very open to the idea that I'm missing something here, but I'd
>>> like to know that you see the issue - that when we test, we have
>>>
>>> VM + "classlib to be tested" + JUnit + testcases
>>>
>>> where the testcases are testing the classlib the VM is running JUnit
>>> with.
>>>
>>> There never is isolation of the code being tested :
>>>
>>> VM + "known good classlib" + Junit + testcases
>>>
>>> unless we have some framework where
>>>
>>> VM + "known good classlib" + JUnit
>>> + framework("classlib to be tested")
>>> + testcases
>>>
>>> and it's that notion of "framework()" that I'm advocating we explore.
>>
>> I'm all for exploring it, I just fundamentally don't buy into the "known
>> good" bit. What happens when the 'classlib to be tested' is 'known
>> better' than the 'known good' one? How do you define "known"? How do you
>> define "good"?
>
> Known? Passed some set of tests. So it could be the Sun JDK for the
> VM + "known good" part.
>
> I think you intuitively understand this. When you find a bug in code
> you are testing, you first assume it's your code, not the framework,
> right? In our case, our framework is actually the code we are
> testing, so we have a bit of a logical conundrum.
>
Hi Geir,
The number of Harmony public API classes that get loaded just to run the
JUnit harness is a little over 200. The majority of these are out of
LUNI with a very low number coming from each of Security, NIO, Archive
and Text.
Sure there is a circular dependency between what we are building and the
framework we are using to test it but it appears to touch on only a
relatively small part of Harmony....IMHO.
Best regards,
George
>>
>>>> Further ideas...
>>>>
>>>> -> look at how the native world does testing
>>>> (hint: it usually has #ifdefs, uses perl along the way, and it is
>>>> certainly
>>>> "messy")
>>>> -> emulate that
>>>>
>>>> -> build a bigger, better specification test
>>>> -> and somehow "prove" it is "good enough"
>>>>
>>>> -> build a bigger, better integration test
>>>> -> and somehow "prove" it is "good enough"
>>>>
>>>> I'll admit my primary interest is the last one...
>>> The problem I see with the last one is that the "parameter space" is
>>> *huge*.
>>
>> Yeah, that's one of the things that makes it interesting. Fortunately
>> open source does have many monkeys...
>>
>>> I believe that your preference for the last one comes from the
>>> Monte-Carlo style approach that Gump uses - hope that your test
>>> suite has enough variance that you "push" the thing being tested
>>> through enough of the parameter space that you can be comfortable
>>> you would have exposed the bugs. Maybe.
>>
>> Ooh, now its becoming rather abstract...
>>
>> Well, perhaps, but more of the gump approache comes from the idea that
>> the parameter space itself is also at some point defined in software,
>> which may have bugs of its own. You circumvent that by making humans the
>> parameter space (don't start about how humans are buggy. We don't
>> want to
>> get into existialism or faith systems when talking about unit testing do
>> we?). The thing that gump enables is "many monkey QA" - a way for
>> thousands
>> of human beings to concurrently make shared assertions about software
>> without actually needing all that much human interaction.
>>
>> More concretely, if harmony can run all known java software, and run
>> it to
>> the asserted satisfaction of all its developers, you can trust that
>> you have
>> covered all the /relevant/ parts of the parameter space you describe.
>
> Yes. And when you can run all knownn Java software, let me know :)
> That's my point about the parameter space being huge. Even when you
> reduce the definition to "that of all known Java software", you still
> have a huge problem on your hands.
>
>> You
>> will never get that level of trust when the assertions are made by
>> software
>> rather than humans. This is how open source leads to software quality.
>>
>> Quoting myself, 'gump is the most misunderstood piece of software,
>> ever'.
>>
>> cheers,
>>
>> Leo
>>
>>
>
Re: [classlib] Testing
Posted by Geir Magnusson Jr <ge...@pobox.com>.
Leo Simons wrote:
> On Wed, Mar 22, 2006 at 08:02:44AM -0500, Geir Magnusson Jr wrote:
>> Leo Simons wrote:
>>> On Wed, Mar 22, 2006 at 07:15:28AM -0500, Geir Magnusson Jr wrote:
>>>> Pulling out of the various threads where we have been discussing, can we
>>>> agree on the problem :
>>>>
>>>> We have unique problems compared to other Java projects because we need
>>>> to find a way to reliably test the things that are commonly expected to
>>>> be a solid point of reference - namely the core class library.
>>>>
>>>> Further, we've been implicitly doing "integration testing" because - so
>>>> far - the only way we've been testing our code has been 'in situ' in the
>>>> VM - not in an isolated test harness. To me, this turns it into an
>>>> integration test.
>>>>
>>>> Sure, we're using JUnit, but because of the fact we are implmenting core
>>>> java.* APIs, we aren't testing with a framework that has been
>>>> independently tested for correctness, like we would when testing any
>>>> other code.
>>>>
>>>> I hope I got that idea across - I believe that we have to go beyond
>>>> normal testing approaches because we don't have a normal situation.
>>> Where we define 'normal situation' as "running a test framework on top of
>>> the sun jdk and expecting any bugs to not be in that jdk". There's plenty
>>> of projects out there that have to test things without having such a
>>> "stable reference JDK" luxury.....I imagine that testing GCC is just as
>>> hard as this problem we have here :-)
>> Is it the same? We need to have a running JVM+classlibarary to test the
>> classlibrary code.
>
> Well you need a working C compiler and standard C library to compile the
> compiler so you can compile make so you can build bash so you can run
> perl (which uses the standard C library functions all over the place of
> course) so you can run the standard C library tests so that you know that
> the library you used when compiling the compiler were correct so you can
> run the compiler tests. I don't think they actually do things that way, but
> it seems like basically the same problem. Having a virtual machine just
> makes it easier since you still assume "the native world" as a baseline,
> which is a lot more than "the hardware".
There's a difference. You can use a completely separate toolchain to
build, test and verify the output of the C compiler.
In our case, we are using the thing we are testing to test itself.
There is no "known good" element possible right now.
We use the classlibrary we are trying to test to execute the test
framework that tests the classlibrary that is running it.
The tool is testing itself.
>
>>>> So I think there are three things we want to do (adopting the
>>>> terminology that came from the discussion with Tim and Leo ) :
>>>>
>>>> 1) implementation tests
>>>> 2) spec/API tests (I'll bundle together)
>>>> 3) integration/functional tests
>>>>
>>>> I believe that for #1, the issues related to being on the bootclasspath
>>>> don't matter, because we aren't testing that aspect of the classes
>>>> (which is how they behave integrated w/ the VM and security system) but
>>>> rather the basic internal functioning.
>>>>
>>>> I'm not sure how to approach this, but I'll try. I'd love to hear how
>>>> Sun, IBM or BEA deals with this, or be told why it isn't an issue :)
>>>>
>>>> Implementation tests : I'd like to see us be able to do #1 via the
>>>> standard same-package technique (i.e. testing a.b.C w/ a.b.CTest) but
>>>> we'll run into a tangle of classloader problems, I suspect, becuase we
>>>> want to be testing java.* code in a system that already has java.* code.
>>>> Can anyone see a way we can do this - test the classlibrary from the
>>>> integration point of view - using some test harness + any known-good
>>>> JRE, like Sun's or IBM's?
>>> Ew, that won't work in the end since we should assume our own JRE is going
>>> to be "known-better" :-). But it might be a nice way to "bootstrap" (eg
>>> we test with an external JRE until we satisfy the tests and then we switch
>>> to testing with an earlier build).
>> Lets be clear - even using our own "earlier build" doesn't solve the
>> problem I'm describing, because as it stands now, we don't use "earlier
>> build" classes to test with - we use the code we want to test as the
>> clsaslibrary for the JRE that's running the test framework.
>>
>> The classes that we are testing are also the classes used by the testing
>> framework. IOW, any of the java.* classes that JUnit itself needs (ex.
>> java.util.HashMap) are exactly the same implementation that it's testing.
>>
>> That's why I think it's subtly different than a "bootstrap and use
>> version - 1 to test" problem. See what I mean?
>
> Yeah yeah, I was already way beyond thinking "just" JUnit is usable for the
> kind of test you're describing. At some point, fundamentally, you either trust
> something external (whether its the sun jdk or the intel compiler designers,
> at some point you do draw a line) or you find a way to bootstrap.
Well, we do trust the Sun JDK.
>
>> I'm very open to the idea that I'm missing something here, but I'd like
>> to know that you see the issue - that when we test, we have
>>
>> VM + "classlib to be tested" + JUnit + testcases
>>
>> where the testcases are testing the classlib the VM is running JUnit with.
>>
>> There never is isolation of the code being tested :
>>
>> VM + "known good classlib" + Junit + testcases
>>
>> unless we have some framework where
>>
>> VM + "known good classlib" + JUnit
>> + framework("classlib to be tested")
>> + testcases
>>
>> and it's that notion of "framework()" that I'm advocating we explore.
>
> I'm all for exploring it, I just fundamentally don't buy into the "known
> good" bit. What happens when the 'classlib to be tested' is 'known
> better' than the 'known good' one? How do you define "known"? How do you
> define "good"?
Known? Passed some set of tests. So it could be the Sun JDK for the VM
+ "known good" part.
I think you intuitively understand this. When you find a bug in code
you are testing, you first assume it's your code, not the framework,
right? In our case, our framework is actually the code we are testing,
so we have a bit of a logical conundrum.
>
>>> Further ideas...
>>>
>>> -> look at how the native world does testing
>>> (hint: it usually has #ifdefs, uses perl along the way, and it is
>>> certainly
>>> "messy")
>>> -> emulate that
>>>
>>> -> build a bigger, better specification test
>>> -> and somehow "prove" it is "good enough"
>>>
>>> -> build a bigger, better integration test
>>> -> and somehow "prove" it is "good enough"
>>>
>>> I'll admit my primary interest is the last one...
>> The problem I see with the last one is that the "parameter space" is *huge*.
>
> Yeah, that's one of the things that makes it interesting. Fortunately
> open source does have many monkeys...
>
>> I believe that your preference for the last one comes from the
>> Monte-Carlo style approach that Gump uses - hope that your test suite
>> has enough variance that you "push" the thing being tested through
>> enough of the parameter space that you can be comfortable you would have
>> exposed the bugs. Maybe.
>
> Ooh, now its becoming rather abstract...
>
> Well, perhaps, but more of the gump approache comes from the idea that
> the parameter space itself is also at some point defined in software,
> which may have bugs of its own. You circumvent that by making humans the
> parameter space (don't start about how humans are buggy. We don't want to
> get into existialism or faith systems when talking about unit testing do
> we?). The thing that gump enables is "many monkey QA" - a way for thousands
> of human beings to concurrently make shared assertions about software
> without actually needing all that much human interaction.
>
> More concretely, if harmony can run all known java software, and run it to
> the asserted satisfaction of all its developers, you can trust that you have
> covered all the /relevant/ parts of the parameter space you describe.
Yes. And when you can run all knownn Java software, let me know :)
That's my point about the parameter space being huge. Even when you
reduce the definition to "that of all known Java software", you still
have a huge problem on your hands.
> You
> will never get that level of trust when the assertions are made by software
> rather than humans. This is how open source leads to software quality.
>
> Quoting myself, 'gump is the most misunderstood piece of software, ever'.
>
> cheers,
>
> Leo
>
>
Re: [classlib] Testing
Posted by Leo Simons <ma...@leosimons.com>.
On Wed, Mar 22, 2006 at 08:02:44AM -0500, Geir Magnusson Jr wrote:
> Leo Simons wrote:
> >On Wed, Mar 22, 2006 at 07:15:28AM -0500, Geir Magnusson Jr wrote:
> >>Pulling out of the various threads where we have been discussing, can we
> >>agree on the problem :
> >>
> >>We have unique problems compared to other Java projects because we need
> >>to find a way to reliably test the things that are commonly expected to
> >>be a solid point of reference - namely the core class library.
> >>
> >>Further, we've been implicitly doing "integration testing" because - so
> >>far - the only way we've been testing our code has been 'in situ' in the
> >>VM - not in an isolated test harness. To me, this turns it into an
> >>integration test.
> >>
> >>Sure, we're using JUnit, but because of the fact we are implmenting core
> >>java.* APIs, we aren't testing with a framework that has been
> >>independently tested for correctness, like we would when testing any
> >>other code.
> >>
> >>I hope I got that idea across - I believe that we have to go beyond
> >>normal testing approaches because we don't have a normal situation.
> >
> >Where we define 'normal situation' as "running a test framework on top of
> >the sun jdk and expecting any bugs to not be in that jdk". There's plenty
> >of projects out there that have to test things without having such a
> >"stable reference JDK" luxury.....I imagine that testing GCC is just as
> >hard as this problem we have here :-)
>
> Is it the same? We need to have a running JVM+classlibarary to test the
> classlibrary code.
Well you need a working C compiler and standard C library to compile the
compiler so you can compile make so you can build bash so you can run
perl (which uses the standard C library functions all over the place of
course) so you can run the standard C library tests so that you know that
the library you used when compiling the compiler were correct so you can
run the compiler tests. I don't think they actually do things that way, but
it seems like basically the same problem. Having a virtual machine just
makes it easier since you still assume "the native world" as a baseline,
which is a lot more than "the hardware".
> >>So I think there are three things we want to do (adopting the
> >>terminology that came from the discussion with Tim and Leo ) :
> >>
> >>1) implementation tests
> >>2) spec/API tests (I'll bundle together)
> >>3) integration/functional tests
> >>
> >>I believe that for #1, the issues related to being on the bootclasspath
> >>don't matter, because we aren't testing that aspect of the classes
> >>(which is how they behave integrated w/ the VM and security system) but
> >>rather the basic internal functioning.
> >>
> >>I'm not sure how to approach this, but I'll try. I'd love to hear how
> >>Sun, IBM or BEA deals with this, or be told why it isn't an issue :)
> >>
> >>Implementation tests : I'd like to see us be able to do #1 via the
> >>standard same-package technique (i.e. testing a.b.C w/ a.b.CTest) but
> >>we'll run into a tangle of classloader problems, I suspect, becuase we
> >>want to be testing java.* code in a system that already has java.* code.
> >> Can anyone see a way we can do this - test the classlibrary from the
> >>integration point of view - using some test harness + any known-good
> >>JRE, like Sun's or IBM's?
> >
> >Ew, that won't work in the end since we should assume our own JRE is going
> >to be "known-better" :-). But it might be a nice way to "bootstrap" (eg
> >we test with an external JRE until we satisfy the tests and then we switch
> >to testing with an earlier build).
>
> Lets be clear - even using our own "earlier build" doesn't solve the
> problem I'm describing, because as it stands now, we don't use "earlier
> build" classes to test with - we use the code we want to test as the
> clsaslibrary for the JRE that's running the test framework.
>
> The classes that we are testing are also the classes used by the testing
> framework. IOW, any of the java.* classes that JUnit itself needs (ex.
> java.util.HashMap) are exactly the same implementation that it's testing.
>
> That's why I think it's subtly different than a "bootstrap and use
> version - 1 to test" problem. See what I mean?
Yeah yeah, I was already way beyond thinking "just" JUnit is usable for the
kind of test you're describing. At some point, fundamentally, you either trust
something external (whether its the sun jdk or the intel compiler designers,
at some point you do draw a line) or you find a way to bootstrap.
> I'm very open to the idea that I'm missing something here, but I'd like
> to know that you see the issue - that when we test, we have
>
> VM + "classlib to be tested" + JUnit + testcases
>
> where the testcases are testing the classlib the VM is running JUnit with.
>
> There never is isolation of the code being tested :
>
> VM + "known good classlib" + Junit + testcases
>
> unless we have some framework where
>
> VM + "known good classlib" + JUnit
> + framework("classlib to be tested")
> + testcases
>
> and it's that notion of "framework()" that I'm advocating we explore.
I'm all for exploring it, I just fundamentally don't buy into the "known
good" bit. What happens when the 'classlib to be tested' is 'known
better' than the 'known good' one? How do you define "known"? How do you
define "good"?
> >Further ideas...
> >
> >-> look at how the native world does testing
> > (hint: it usually has #ifdefs, uses perl along the way, and it is
> > certainly
> > "messy")
> > -> emulate that
> >
> >-> build a bigger, better specification test
> > -> and somehow "prove" it is "good enough"
> >
> >-> build a bigger, better integration test
> > -> and somehow "prove" it is "good enough"
> >
> >I'll admit my primary interest is the last one...
>
> The problem I see with the last one is that the "parameter space" is *huge*.
Yeah, that's one of the things that makes it interesting. Fortunately
open source does have many monkeys...
> I believe that your preference for the last one comes from the
> Monte-Carlo style approach that Gump uses - hope that your test suite
> has enough variance that you "push" the thing being tested through
> enough of the parameter space that you can be comfortable you would have
> exposed the bugs. Maybe.
Ooh, now its becoming rather abstract...
Well, perhaps, but more of the gump approache comes from the idea that
the parameter space itself is also at some point defined in software,
which may have bugs of its own. You circumvent that by making humans the
parameter space (don't start about how humans are buggy. We don't want to
get into existialism or faith systems when talking about unit testing do
we?). The thing that gump enables is "many monkey QA" - a way for thousands
of human beings to concurrently make shared assertions about software
without actually needing all that much human interaction.
More concretely, if harmony can run all known java software, and run it to
the asserted satisfaction of all its developers, you can trust that you have
covered all the /relevant/ parts of the parameter space you describe. You
will never get that level of trust when the assertions are made by software
rather than humans. This is how open source leads to software quality.
Quoting myself, 'gump is the most misunderstood piece of software, ever'.
cheers,
Leo
Re: [classlib] Testing
Posted by Geir Magnusson Jr <ge...@pobox.com>.
Leo Simons wrote:
> On Wed, Mar 22, 2006 at 07:15:28AM -0500, Geir Magnusson Jr wrote:
>> Pulling out of the various threads where we have been discussing, can we
>> agree on the problem :
>>
>> We have unique problems compared to other Java projects because we need
>> to find a way to reliably test the things that are commonly expected to
>> be a solid point of reference - namely the core class library.
>>
>> Further, we've been implicitly doing "integration testing" because - so
>> far - the only way we've been testing our code has been 'in situ' in the
>> VM - not in an isolated test harness. To me, this turns it into an
>> integration test.
>>
>> Sure, we're using JUnit, but because of the fact we are implmenting core
>> java.* APIs, we aren't testing with a framework that has been
>> independently tested for correctness, like we would when testing any
>> other code.
>>
>> I hope I got that idea across - I believe that we have to go beyond
>> normal testing approaches because we don't have a normal situation.
>
> Where we define 'normal situation' as "running a test framework on top of
> the sun jdk and expecting any bugs to not be in that jdk". There's plenty
> of projects out there that have to test things without having such a
> "stable reference JDK" luxury.....I imagine that testing GCC is just as
> hard as this problem we have here :-)
Is it the same? We need to have a running JVM+classlibarary to test the
classlibrary code.
>
>> So I think there are three things we want to do (adopting the
>> terminology that came from the discussion with Tim and Leo ) :
>>
>> 1) implementation tests
>> 2) spec/API tests (I'll bundle together)
>> 3) integration/functional tests
>>
>> I believe that for #1, the issues related to being on the bootclasspath
>> don't matter, because we aren't testing that aspect of the classes
>> (which is how they behave integrated w/ the VM and security system) but
>> rather the basic internal functioning.
>>
>> I'm not sure how to approach this, but I'll try. I'd love to hear how
>> Sun, IBM or BEA deals with this, or be told why it isn't an issue :)
>>
>> Implementation tests : I'd like to see us be able to do #1 via the
>> standard same-package technique (i.e. testing a.b.C w/ a.b.CTest) but
>> we'll run into a tangle of classloader problems, I suspect, becuase we
>> want to be testing java.* code in a system that already has java.* code.
>> Can anyone see a way we can do this - test the classlibrary from the
>> integration point of view - using some test harness + any known-good
>> JRE, like Sun's or IBM's?
>
> Ew, that won't work in the end since we should assume our own JRE is going
> to be "known-better" :-). But it might be a nice way to "bootstrap" (eg
> we test with an external JRE until we satisfy the tests and then we switch
> to testing with an earlier build).
Lets be clear - even using our own "earlier build" doesn't solve the
problem I'm describing, because as it stands now, we don't use "earlier
build" classes to test with - we use the code we want to test as the
clsaslibrary for the JRE that's running the test framework.
The classes that we are testing are also the classes used by the testing
framework. IOW, any of the java.* classes that JUnit itself needs (ex.
java.util.HashMap) are exactly the same implementation that it's testing.
That's why I think it's subtly different than a "bootstrap and use
version - 1 to test" problem. See what I mean?
I'm very open to the idea that I'm missing something here, but I'd like
to know that you see the issue - that when we test, we have
VM + "classlib to be tested" + JUnit + testcases
where the testcases are testing the classlib the VM is running JUnit with.
There never is isolation of the code being tested :
VM + "known good classlib" + Junit + testcases
unless we have some framework where
VM + "known good classlib" + JUnit
+ framework("classlib to be tested")
+ testcases
and it's that notion of "framework()" that I'm advocating we explore.
>
> For code that has side effects or for which we can conceivably create
> verifiable side effects (where side effect is something outside of the
> whole "java environment") we can try and produce known-good input and
> output. There's a variety of ways to automate things like that, for
> example by using tracing on the relevant bits, manually verifying a
> "known-good" trace, storing it, and comparing future runs.
>
> But I suspect there is a whole lot of code that is either inherently all
> but side-effect-free, or where testing the side effects automatically
> amounts to doing an integration test.
>
>> Spec/API tests : these are, IMO, a kind of integration test, because
>> proper spec/API behavior *is* dependent on factors beyond the actual
>> code itself (like classloader configuration, and security context).
>> Because of this, the *.test.* pattern makes perfect sense. Assuming we
>> could produce something useful for #1 (i.e. a test harness/framework),
>> could we then augment it to simulate the classloader config + security
>> config that we'd get in a real VM? That will give us the ability to
>> test in isolation of the VM, and also let us 'break' the environment to
>> ensure that the code fails in a predictable way.
>>
>> Intgration/functional : this is a whole range of things, from doing the
>> Spec/API tests in an actual VM, to the tests that exercise the code
>> through interaction with external systems (like network, RMI, GUI, etc)
>>
>> ***
>>
>> Now, it might be suggested that we just ignore the implementation
>> testing (#1) and just do #2 and #3 as we are now, and hope we have a
>> good enough test suite. It could be argued that when Sun started, they
>> didn't have a known-good platform to do implementation testing on like
>> we do now. I don't know if that's true.
>>
>> The difference is that we need to produce something of the same quality
>> as Sun's Java 5, not Sun's Java 1.0. We've had 11 years since 1.0 to
>> learn about testing, but they've had 11 years to get things solid.
>>
>> What to do....
>
> No idea! Cool!
>
> We should do #2 and #3 regardless.
We're already doing #2 and #3. The lack of #1 is what bothers me.
> Identifying which-is-which (#1, #2, #3)
> in all the current test suites seems like a good next step. Obviously that
> doesn't really help us get that implementation testing framework you
> describe but it will help more unambigously define the needs.
>
> Further ideas...
>
> -> look at how the native world does testing
> (hint: it usually has #ifdefs, uses perl along the way, and it is certainly
> "messy")
> -> emulate that
>
> -> build a bigger, better specification test
> -> and somehow "prove" it is "good enough"
>
> -> build a bigger, better integration test
> -> and somehow "prove" it is "good enough"
>
> I'll admit my primary interest is the last one...
The problem I see with the last one is that the "parameter space" is *huge*.
I believe that your preference for the last one comes from the
Monte-Carlo style approach that Gump uses - hope that your test suite
has enough variance that you "push" the thing being tested through
enough of the parameter space that you can be comfortable you would have
exposed the bugs. Maybe.
geir
Re: [classlib] Testing
Posted by Leo Simons <ma...@leosimons.com>.
On Wed, Mar 22, 2006 at 07:15:28AM -0500, Geir Magnusson Jr wrote:
> Pulling out of the various threads where we have been discussing, can we
> agree on the problem :
>
> We have unique problems compared to other Java projects because we need
> to find a way to reliably test the things that are commonly expected to
> be a solid point of reference - namely the core class library.
>
> Further, we've been implicitly doing "integration testing" because - so
> far - the only way we've been testing our code has been 'in situ' in the
> VM - not in an isolated test harness. To me, this turns it into an
> integration test.
>
> Sure, we're using JUnit, but because of the fact we are implmenting core
> java.* APIs, we aren't testing with a framework that has been
> independently tested for correctness, like we would when testing any
> other code.
>
> I hope I got that idea across - I believe that we have to go beyond
> normal testing approaches because we don't have a normal situation.
Where we define 'normal situation' as "running a test framework on top of
the sun jdk and expecting any bugs to not be in that jdk". There's plenty
of projects out there that have to test things without having such a
"stable reference JDK" luxury.....I imagine that testing GCC is just as
hard as this problem we have here :-)
> So I think there are three things we want to do (adopting the
> terminology that came from the discussion with Tim and Leo ) :
>
> 1) implementation tests
> 2) spec/API tests (I'll bundle together)
> 3) integration/functional tests
>
> I believe that for #1, the issues related to being on the bootclasspath
> don't matter, because we aren't testing that aspect of the classes
> (which is how they behave integrated w/ the VM and security system) but
> rather the basic internal functioning.
>
> I'm not sure how to approach this, but I'll try. I'd love to hear how
> Sun, IBM or BEA deals with this, or be told why it isn't an issue :)
>
> Implementation tests : I'd like to see us be able to do #1 via the
> standard same-package technique (i.e. testing a.b.C w/ a.b.CTest) but
> we'll run into a tangle of classloader problems, I suspect, becuase we
> want to be testing java.* code in a system that already has java.* code.
> Can anyone see a way we can do this - test the classlibrary from the
> integration point of view - using some test harness + any known-good
> JRE, like Sun's or IBM's?
Ew, that won't work in the end since we should assume our own JRE is going
to be "known-better" :-). But it might be a nice way to "bootstrap" (eg
we test with an external JRE until we satisfy the tests and then we switch
to testing with an earlier build).
For code that has side effects or for which we can conceivably create
verifiable side effects (where side effect is something outside of the
whole "java environment") we can try and produce known-good input and
output. There's a variety of ways to automate things like that, for
example by using tracing on the relevant bits, manually verifying a
"known-good" trace, storing it, and comparing future runs.
But I suspect there is a whole lot of code that is either inherently all
but side-effect-free, or where testing the side effects automatically
amounts to doing an integration test.
> Spec/API tests : these are, IMO, a kind of integration test, because
> proper spec/API behavior *is* dependent on factors beyond the actual
> code itself (like classloader configuration, and security context).
> Because of this, the *.test.* pattern makes perfect sense. Assuming we
> could produce something useful for #1 (i.e. a test harness/framework),
> could we then augment it to simulate the classloader config + security
> config that we'd get in a real VM? That will give us the ability to
> test in isolation of the VM, and also let us 'break' the environment to
> ensure that the code fails in a predictable way.
>
> Intgration/functional : this is a whole range of things, from doing the
> Spec/API tests in an actual VM, to the tests that exercise the code
> through interaction with external systems (like network, RMI, GUI, etc)
>
> ***
>
> Now, it might be suggested that we just ignore the implementation
> testing (#1) and just do #2 and #3 as we are now, and hope we have a
> good enough test suite. It could be argued that when Sun started, they
> didn't have a known-good platform to do implementation testing on like
> we do now. I don't know if that's true.
>
> The difference is that we need to produce something of the same quality
> as Sun's Java 5, not Sun's Java 1.0. We've had 11 years since 1.0 to
> learn about testing, but they've had 11 years to get things solid.
>
> What to do....
No idea! Cool!
We should do #2 and #3 regardless. Identifying which-is-which (#1, #2, #3)
in all the current test suites seems like a good next step. Obviously that
doesn't really help us get that implementation testing framework you
describe but it will help more unambigously define the needs.
Further ideas...
-> look at how the native world does testing
(hint: it usually has #ifdefs, uses perl along the way, and it is certainly
"messy")
-> emulate that
-> build a bigger, better specification test
-> and somehow "prove" it is "good enough"
-> build a bigger, better integration test
-> and somehow "prove" it is "good enough"
I'll admit my primary interest is the last one...
Leo