Excluded tests in x-net module or back to bootclasspath issue

[Stepan Mishura <st...@gmail.com>]

Hi Tim,

Currently you excluded 12 tests in 'x-net' module. The ant script file
contains the following comments for them: "The following tests are excluded
because they fail for reasons to be determined"
The reason is well known and it was discussed a lot. These tests depend on
package access functionality and BC provider jar is still required to be on
bootclasspath (I don't remember that we solved this issue). I know that you
are allergic to putting all stuff on bootclasspath but I'd prefer keep them
running to track regressions rather then adding them to exclude list and
forgetting about them for a long time.

Thanks,
Stepan Mishura
Intel Middleware Products Division

Re: Excluded tests in x-net module or back to bootclasspath issue

[Stepan Mishura <st...@gmail.com>]

Hi Tim,

On 3/14/06, Tim Ellison <t....@gmail.com> wrote:
>
> I agree that they need to be reinstated Stepan, I'd just like to
> separate out the suite of tests that are pure API tests (and therefore
> expected to run on all compliant implementations, modulo their known
> bugs/departures) and the internal implementation tests that we would
> only expect to pass on Harmony code.
>
> I had hoped that we could separate them using JUnit test suites, but as
> Geir pointed out the Ant support for JUnit suites produces very limited
> reports.  I'd always run suites directly from Eclipse and could see all
> the required info.
>
> So it seems that the options are to (a) figure out why Ant is hiding the
> data, and fix that; or (b) live with it and invent a naming scheme for
> Harmony implementation tests that keep them separate from the API tests.


I'd suggest the following tests separation:
<MODULE_ROOT>/src/test/java
    (java|javax|org)/**/*Test.java (excluding org/apache/harmony/tests)
     org/apache/harmony/tests/**/*Test.java

First folder contains tests for 'package access' functionality and tested
classes must be on bootclasspath. Second folder keeps tests for public API (
i.e. 'compliant' tests). To avoid test code duplication the preferred folder
for placing new tests is the second one - if a test doesn't depend on
package access functionality them it goes to 'compliant' tests. So it is up
to a developer to decide where to place a test. Also it may be makes sense
to compile tests to different 'bin' folders.

So we'll have the next 'testing modes':
1) Compliant mode: 'compliant' tests are run on some java certified
implementation.
2) Cross-compliant mode: all tests run against the following JRE
configuration - a module's jar + compliant java implementation (i.e. a
module's jar and 'package access' tests are on the bootclasspath)
3) Self-testing mode: all tests run against Harmony implementation


As (I think it was) Mikhail pointed out a while ago, using class naming
> schemes is far from ideal, as there may be a significant repetition of
> test cases and the suites they fit into.



I believe that the separation suggested above may solve this issue.

 If we want to support testing
> with 'exotic' configurations then we will need some 'suites' that assume
> basic set-up, and others that can assume a test http server, ldap
> server, database, and so on.


I'd prefer to avoid external dependencies for unit testing as much as
possible. Otherwise specific (some special set-up is required) tests should
be placed to a separate test suite.

Thanks,
Stepan.

So my preference is to figure out why Ant produces its limited report
> for JUnit suites.
>
> Matt: can you shed any light on this?
>
> Regards,
> Tim
>
> Stepan Mishura wrote:
> > Hi Tim,
> >
> > Currently you excluded 12 tests in 'x-net' module. The ant script file
> > contains the following comments for them: "The following tests are
> excluded
> > because they fail for reasons to be determined"
> > The reason is well known and it was discussed a lot. These tests depend
> on
> > package access functionality and BC provider jar is still required to be
> on
> > bootclasspath (I don't remember that we solved this issue). I know that
> you
> > are allergic to putting all stuff on bootclasspath but I'd prefer keep
> them
> > running to track regressions rather then adding them to exclude list and
> > forgetting about them for a long time.
> >
> > Thanks,
> > Stepan Mishura
> > Intel Middleware Products Division
> >
>
> --
>
> Tim Ellison (t.p.ellison@gmail.com)
> IBM Java technology centre, UK.
>



--
Thanks,
Stepan Mishura
Intel Middleware Products Division

Re: Excluded tests in x-net module or back to bootclasspath issue

[Tim Ellison <t....@gmail.com>]

I agree that they need to be reinstated Stepan, I'd just like to
separate out the suite of tests that are pure API tests (and therefore
expected to run on all compliant implementations, modulo their known
bugs/departures) and the internal implementation tests that we would
only expect to pass on Harmony code.

I had hoped that we could separate them using JUnit test suites, but as
Geir pointed out the Ant support for JUnit suites produces very limited
reports.  I'd always run suites directly from Eclipse and could see all
the required info.

So it seems that the options are to (a) figure out why Ant is hiding the
data, and fix that; or (b) live with it and invent a naming scheme for
Harmony implementation tests that keep them separate from the API tests.

As (I think it was) Mikhail pointed out a while ago, using class naming
schemes is far from ideal, as there may be a significant repetition of
test cases and the suites they fit into.  If we want to support testing
with 'exotic' configurations then we will need some 'suites' that assume
basic set-up, and others that can assume a test http server, ldap
server, database, and so on.

So my preference is to figure out why Ant produces its limited report
for JUnit suites.

Matt: can you shed any light on this?

Regards,
Tim

Stepan Mishura wrote:
> Hi Tim,
> 
> Currently you excluded 12 tests in 'x-net' module. The ant script file
> contains the following comments for them: "The following tests are excluded
> because they fail for reasons to be determined"
> The reason is well known and it was discussed a lot. These tests depend on
> package access functionality and BC provider jar is still required to be on
> bootclasspath (I don't remember that we solved this issue). I know that you
> are allergic to putting all stuff on bootclasspath but I'd prefer keep them
> running to track regressions rather then adding them to exclude list and
> forgetting about them for a long time.
> 
> Thanks,
> Stepan Mishura
> Intel Middleware Products Division
> 

-- 

Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.