[junit] different configurations at test level Was - Re: [junit] Top-level suites added

[Daniel John Debrunner <dj...@apache.org>]

Vemund Ostgaard wrote:

> This would mean that when executing a single TestCase class (or suite) 
> with a junit runner, the default behaviour could be to run with all 
> (supported) frameworks if that is what is most common. Running with just 
> a selected framework could be possible with a system property.

I now think this is the correct direction for JUnit tests, so that 
running, say, the lang.TimeHandlingTest, would run it in embedded and 
network client, not just embedded. Thus there would not be a reliance on 
top-level suites to run in different fundamental configurations. I see 
fundamental configurations being embedded, derby client and possible the 
db2 client. Other configurations would be handed at a top-level suite, 
e.g. run a set of tests with an encrypted database.

However, I'm don't see a clear way forward of how to implement this.

Currently in suite() methods the code checks the current configuration 
and adds test cases based upon that information. e.g. only add this 
fixture if it's embedded. Thus the assumption is that the suite() method 
will be called multiple times, once per configuration.

To push the multiple configurations into the suite() I see a number of 
possible options;

A) Have a standard pattern of calling the suite method calling itself 
indirectly through existing decorator, 
TestConfiguration.derbyClientServerDecorator.

      public static Test suite() {
         ...
         // run under other configurations
         if (usingEmbedded()) {
            suite.add(
             TestConfiguration.derbyClientServerDecorator(
                   TimeHandlingTest.class);

            // possible similar code for DB2 client
         }
         ...

Seems a little confusing, because the embedded (base) configuration is 
the one that adds the other configurations, the usingEmbedded() is there 
to stop recursion.

Other issue is that if the test doesn't want to run in client, does it 
not include the standard pattern, or just use the usingXXX() methods to 
set up empty suites for the configuration it doesn't run?

B) Have a test's suite method just set up a suite as it sees fit:

       public static Test suite() {
           TestSuite suite = new TestSuite();

           // Embedded tests
           suite.add(TimeHandlingTest.class);
           suite.add("embeddedTestA");
           suite.add("embeddedTestB");
           suite.add("embeddedTestC");

           // Client tests
           TestSuite client = new TestSuite();
           client.add(TimeHandlingTest.class);
           client.add("clientTestA");

           suite.add(new NetworkServerTestSetup(client));

           return suite;
       }

Downsides are:
     possible lack of consistency, but maybe the consistency is use of 
the NetworkServerTestSetup decorator.
     how to handle remote server testing?

Upside is:
      clarity
      matches standard JUnit pattern
      non-reliance of having the configuration setup at suite setup time.

C) Something in the BaseTestClass that automatically runs the test 
fixtures in multiple configurations. e.g. modifying runBase to call 
super.runBase() for each configuration.

Upside is tests automatically run in all configurations.

Downside is that decision if to run a text fixture or not is based at 
test runtime and not suite setup time, which means cluttering test 
fixture methods with if statements to indicate if the will run or not.
Other downside is that a large number of tests may not need to run in 
multiple configurations, e.g. tests that test SQL functionality may not 
have much benefit running in any client mode.

-----------------------------------------------------

Are there any other ideas?  Having written them out, my preference would 
be B), seems very clear and straight forward. Before I wrote it out I 
was thinking that none of the solutions were good.

Thanks,
Dan.

Re: [junit] different configurations at test level Was - Re: [junit] Top-level suites added

[Vemund Ostgaard <Ve...@Sun.COM>]

Daniel John Debrunner wrote:

> Vemund Ostgaard wrote:
>
>> This would mean that when executing a single TestCase class (or 
>> suite) with a junit runner, the default behaviour could be to run 
>> with all (supported) frameworks if that is what is most common. 
>> Running with just a selected framework could be possible with a 
>> system property.
>
<snip>

> B) Have a test's suite method just set up a suite as it sees fit:
>
>       public static Test suite() {
>           TestSuite suite = new TestSuite();
>
>           // Embedded tests
>           suite.add(TimeHandlingTest.class);
>           suite.add("embeddedTestA");
>           suite.add("embeddedTestB");
>           suite.add("embeddedTestC");
>
>           // Client tests
>           TestSuite client = new TestSuite();
>           client.add(TimeHandlingTest.class);
>           client.add("clientTestA");
>
>           suite.add(new NetworkServerTestSetup(client));
>
>           return suite;
>       }
>
> Downsides are:
>     possible lack of consistency, but maybe the consistency is use of 
> the NetworkServerTestSetup decorator.
>     how to handle remote server testing?

The remote server is a special case of the "client" framework here. You 
want the "client" tests to be run, but you don't want the 
NetworkServerTestSetup to actually start a network server (unless 
someone solves the problem of doing this remotely on all platforms). So 
the NetworkServerTestSetup needs to handle this by just doing nothing if 
the server host is set to a remote machine. The tester will have to 
start the server manually before starting the test.

>
> Upside is:
>      clarity
>      matches standard JUnit pattern
>      non-reliance of having the configuration setup at suite setup time.
>
<snip>

> -----------------------------------------------------
>
> Are there any other ideas?  Having written them out, my preference 
> would be B), seems very clear and straight forward. Before I wrote it 
> out I was thinking that none of the solutions were good.

When I made the suggestion you quoted above I was thinking along the 
lines of B, without thinking in detail or having studied the code. I 
think that is the right place to solve the problem, and your code 
suggestion is how I see the default behaviour. In addition it should be 
possible to override with properties, so that the suite does not add the 
"embedded" tests if the user set a property that he did not want 
embedded tests run, only client. For instance, something like:

Testsuite suite = new TestSuite();
if(runEmbedded) {
   Testsuite embedded = new TestSuite();
   suite.add(embedded);
}
if(runClient) {
   Testsuite client = new TestSuite();
   suite.add(new NetworkServerTestSetup(client));
}
// etc.

return suite;
}

Or something along those lines?

Thanks,
Vemund