Re: [junit] Top-level suites added

[Vemund Ostgaard <Ve...@Sun.COM>]

Daniel John Debrunner wrote:

>The current approach has very little magic, one decorator for client and
>probably one decorator for DB2 client. I strongly agree that tests
>should not be overly influenced by the suite setup.
>  
>
I think this is a sound principle to keep in mind as work progresses on 
this new testsuite. It might not be possible to avoid suite influence 
totally, but hopefully it is kept at a level where it is not regarded as 
a problem with the testsuite.

>This is an interesting idea, some thoughts jump out at me:
>
> - A single test would potentially run in many configurations:
>     embedded, client, db2 client, encryption DES, encryption AES, ...
>  
>
If the number of configurations grows large, it might pay off to 
carefully review wether something should be a configuration of its own 
or rather be tested through a set of more specialized tests. When the 
master files are gone and we have a new assertion-based testsuite, the 
tests should only fail on things they are specifically looking for, 
possibly making them less suited for testing of a large number of 
configurations? I guess it will be a tradeoff, having many 
configurations or more tests.

> - The large number of decorators required per test could factor into
>large memory use for an entire suite, not sure of the overhead of each
>TestSetup.
>
> - Embedded tests would be running with the network server booted by
>default, not sure of the impact of this, starting the network server for
>each individual TestCase would slow an entire suite down too much
>  
>
Are you thinking of the impact of the network server running in the same 
vm as junit?

> - It does make running a single TestCase in all configurations easy
>
> - The potential requirement to modify all tests when a new
>configuration is added into the run, e.g. if I add a configuration that
>only works in 10% of the tests (that's my itch) then I have to modify
>the other 90% of tests to disable it from running. Assuming the default
>as you say is to run each test in all possible configurations, I assume
>this would be enforced in the base test class. Maybe that's the case
>today with the current approach where a test is ideally self describing.
>  
>
I guess that the default set of configurations that a TestCase supports 
could be something smaller than the set of all possible configurations. 
The default runtime configuration would be to run all tests in all 
possible configurations, but a TestCase with default settings would 
override this with its smaller (default) set. You would then have to fix 
your 10% tests by adding the new configuration to their supported set, 
instead of removing it from the other 90%. Still, that is work, but I 
would hope we don't end up with a jungle of configurations that anyhow 
would be a pain to maintain.

I do believe, though, that it makes sense that a TestCase itself 
maintains the information about what configurations/frameworks it 
supports or requires, and not suite classes located somewhere else.

> - Maybe there are some configurations that are best handled at the leaf
>level, e.g. embedded and client, but others are best handled at a suite
>level, e.g. db2 client, encryption. So a mix of the two approaches would
>be good.
>  
>
Yes, that may very well be. As I've mentioned above I also think it 
should be considered if all configurations are really needed or wether 
it can be tested as well or better with new tests instead.

>I'm going to continue my current setup approach because it's not opposed
>to what you approach, the majority of the work will be getting tests
>converted to JUnit and running in client & db2 client mode. Ie. most of
>the work overlaps anyway.
>  
>
Yes, that is true. The work you are doing now is making it a lot easier 
to imagine how things may fit together with a testsuite written and run 
only with junit, and to have opinions on that.

Vemund