[junit] Top-level suites added

[Daniel John Debrunner <dj...@apache.org>]

I added some top-level suites that are intended to be run directly by
JUnit runners, not via the test harness. They are in the package:

org.apache.derbyTesting.functionTests.suites

AllPackages - A suite of all the _Suite suites for the function test
packages

Embedded - All the tests for embedded. Currently just includes
AllPackages.suite(), but future may add more in different
configurations, e.g. encryption.

Client - All the tests for client, runs the AllPackages.suite() with a
decorator that changes the configuration and boots the network server
(using the existing network server decorator)

All - All the tests, currently includes Client.suite() and
Embedded.suite(). (Eventual replacement for derbyall)

It's not all working fine yet, a couple of issues:

  1) The network server decorator in jdbcapi._Suite clashes with the one
in Client

  2) Some tests fail in the Client suite, need to investigate.
Additional tests were failing due to some test using the old checks to
see if it was running embedded or not, rather than the utilities
provided in the JUnit classes.

I've checked these in for others to see the direction I'm proposing,
with the goal being all tests running under JUnit with no harness.

Example of running these:

java -Dderby.system.home=${PWD} junit.textui.TestRunner
          org.apache.derbyTesting.functionTests.suites.Embedded

java -Dderby.system.home=${PWD} junit.textui.TestRunner
          org.apache.derbyTesting.functionTests.suites.Client

java -Dderby.system.home=${PWD} junit.textui.TestRunner
          org.apache.derbyTesting.functionTests.suites.All

Comments?
Dan.

Re: [junit] different configurations at test level Was - Re: [junit] Top-level suites added

[Vemund Ostgaard <Ve...@Sun.COM>]

Daniel John Debrunner wrote:

> Vemund Ostgaard wrote:
>
>> This would mean that when executing a single TestCase class (or 
>> suite) with a junit runner, the default behaviour could be to run 
>> with all (supported) frameworks if that is what is most common. 
>> Running with just a selected framework could be possible with a 
>> system property.
>
<snip>

> B) Have a test's suite method just set up a suite as it sees fit:
>
>       public static Test suite() {
>           TestSuite suite = new TestSuite();
>
>           // Embedded tests
>           suite.add(TimeHandlingTest.class);
>           suite.add("embeddedTestA");
>           suite.add("embeddedTestB");
>           suite.add("embeddedTestC");
>
>           // Client tests
>           TestSuite client = new TestSuite();
>           client.add(TimeHandlingTest.class);
>           client.add("clientTestA");
>
>           suite.add(new NetworkServerTestSetup(client));
>
>           return suite;
>       }
>
> Downsides are:
>     possible lack of consistency, but maybe the consistency is use of 
> the NetworkServerTestSetup decorator.
>     how to handle remote server testing?

The remote server is a special case of the "client" framework here. You 
want the "client" tests to be run, but you don't want the 
NetworkServerTestSetup to actually start a network server (unless 
someone solves the problem of doing this remotely on all platforms). So 
the NetworkServerTestSetup needs to handle this by just doing nothing if 
the server host is set to a remote machine. The tester will have to 
start the server manually before starting the test.

>
> Upside is:
>      clarity
>      matches standard JUnit pattern
>      non-reliance of having the configuration setup at suite setup time.
>
<snip>

> -----------------------------------------------------
>
> Are there any other ideas?  Having written them out, my preference 
> would be B), seems very clear and straight forward. Before I wrote it 
> out I was thinking that none of the solutions were good.

When I made the suggestion you quoted above I was thinking along the 
lines of B, without thinking in detail or having studied the code. I 
think that is the right place to solve the problem, and your code 
suggestion is how I see the default behaviour. In addition it should be 
possible to override with properties, so that the suite does not add the 
"embedded" tests if the user set a property that he did not want 
embedded tests run, only client. For instance, something like:

Testsuite suite = new TestSuite();
if(runEmbedded) {
   Testsuite embedded = new TestSuite();
   suite.add(embedded);
}
if(runClient) {
   Testsuite client = new TestSuite();
   suite.add(new NetworkServerTestSetup(client));
}
// etc.

return suite;
}

Or something along those lines?

Thanks,
Vemund

[junit] different configurations at test level Was - Re: [junit] Top-level suites added

[Daniel John Debrunner <dj...@apache.org>]

Vemund Ostgaard wrote:

> This would mean that when executing a single TestCase class (or suite) 
> with a junit runner, the default behaviour could be to run with all 
> (supported) frameworks if that is what is most common. Running with just 
> a selected framework could be possible with a system property.

I now think this is the correct direction for JUnit tests, so that 
running, say, the lang.TimeHandlingTest, would run it in embedded and 
network client, not just embedded. Thus there would not be a reliance on 
top-level suites to run in different fundamental configurations. I see 
fundamental configurations being embedded, derby client and possible the 
db2 client. Other configurations would be handed at a top-level suite, 
e.g. run a set of tests with an encrypted database.

However, I'm don't see a clear way forward of how to implement this.

Currently in suite() methods the code checks the current configuration 
and adds test cases based upon that information. e.g. only add this 
fixture if it's embedded. Thus the assumption is that the suite() method 
will be called multiple times, once per configuration.

To push the multiple configurations into the suite() I see a number of 
possible options;

A) Have a standard pattern of calling the suite method calling itself 
indirectly through existing decorator, 
TestConfiguration.derbyClientServerDecorator.

      public static Test suite() {
         ...
         // run under other configurations
         if (usingEmbedded()) {
            suite.add(
             TestConfiguration.derbyClientServerDecorator(
                   TimeHandlingTest.class);

            // possible similar code for DB2 client
         }
         ...

Seems a little confusing, because the embedded (base) configuration is 
the one that adds the other configurations, the usingEmbedded() is there 
to stop recursion.

Other issue is that if the test doesn't want to run in client, does it 
not include the standard pattern, or just use the usingXXX() methods to 
set up empty suites for the configuration it doesn't run?

B) Have a test's suite method just set up a suite as it sees fit:

       public static Test suite() {
           TestSuite suite = new TestSuite();

           // Embedded tests
           suite.add(TimeHandlingTest.class);
           suite.add("embeddedTestA");
           suite.add("embeddedTestB");
           suite.add("embeddedTestC");

           // Client tests
           TestSuite client = new TestSuite();
           client.add(TimeHandlingTest.class);
           client.add("clientTestA");

           suite.add(new NetworkServerTestSetup(client));

           return suite;
       }

Downsides are:
     possible lack of consistency, but maybe the consistency is use of 
the NetworkServerTestSetup decorator.
     how to handle remote server testing?

Upside is:
      clarity
      matches standard JUnit pattern
      non-reliance of having the configuration setup at suite setup time.

C) Something in the BaseTestClass that automatically runs the test 
fixtures in multiple configurations. e.g. modifying runBase to call 
super.runBase() for each configuration.

Upside is tests automatically run in all configurations.

Downside is that decision if to run a text fixture or not is based at 
test runtime and not suite setup time, which means cluttering test 
fixture methods with if statements to indicate if the will run or not.
Other downside is that a large number of tests may not need to run in 
multiple configurations, e.g. tests that test SQL functionality may not 
have much benefit running in any client mode.

-----------------------------------------------------

Are there any other ideas?  Having written them out, my preference would 
be B), seems very clear and straight forward. Before I wrote it out I 
was thinking that none of the solutions were good.

Thanks,
Dan.

Re: [junit] Top-level suites added

[Vemund Ostgaard <Ve...@Sun.COM>]

Daniel John Debrunner wrote:

>The current approach has very little magic, one decorator for client and
>probably one decorator for DB2 client. I strongly agree that tests
>should not be overly influenced by the suite setup.
>  
>
I think this is a sound principle to keep in mind as work progresses on 
this new testsuite. It might not be possible to avoid suite influence 
totally, but hopefully it is kept at a level where it is not regarded as 
a problem with the testsuite.

>This is an interesting idea, some thoughts jump out at me:
>
> - A single test would potentially run in many configurations:
>     embedded, client, db2 client, encryption DES, encryption AES, ...
>  
>
If the number of configurations grows large, it might pay off to 
carefully review wether something should be a configuration of its own 
or rather be tested through a set of more specialized tests. When the 
master files are gone and we have a new assertion-based testsuite, the 
tests should only fail on things they are specifically looking for, 
possibly making them less suited for testing of a large number of 
configurations? I guess it will be a tradeoff, having many 
configurations or more tests.

> - The large number of decorators required per test could factor into
>large memory use for an entire suite, not sure of the overhead of each
>TestSetup.
>
> - Embedded tests would be running with the network server booted by
>default, not sure of the impact of this, starting the network server for
>each individual TestCase would slow an entire suite down too much
>  
>
Are you thinking of the impact of the network server running in the same 
vm as junit?

> - It does make running a single TestCase in all configurations easy
>
> - The potential requirement to modify all tests when a new
>configuration is added into the run, e.g. if I add a configuration that
>only works in 10% of the tests (that's my itch) then I have to modify
>the other 90% of tests to disable it from running. Assuming the default
>as you say is to run each test in all possible configurations, I assume
>this would be enforced in the base test class. Maybe that's the case
>today with the current approach where a test is ideally self describing.
>  
>
I guess that the default set of configurations that a TestCase supports 
could be something smaller than the set of all possible configurations. 
The default runtime configuration would be to run all tests in all 
possible configurations, but a TestCase with default settings would 
override this with its smaller (default) set. You would then have to fix 
your 10% tests by adding the new configuration to their supported set, 
instead of removing it from the other 90%. Still, that is work, but I 
would hope we don't end up with a jungle of configurations that anyhow 
would be a pain to maintain.

I do believe, though, that it makes sense that a TestCase itself 
maintains the information about what configurations/frameworks it 
supports or requires, and not suite classes located somewhere else.

> - Maybe there are some configurations that are best handled at the leaf
>level, e.g. embedded and client, but others are best handled at a suite
>level, e.g. db2 client, encryption. So a mix of the two approaches would
>be good.
>  
>
Yes, that may very well be. As I've mentioned above I also think it 
should be considered if all configurations are really needed or wether 
it can be tested as well or better with new tests instead.

>I'm going to continue my current setup approach because it's not opposed
>to what you approach, the majority of the work will be getting tests
>converted to JUnit and running in client & db2 client mode. Ie. most of
>the work overlaps anyway.
>  
>
Yes, that is true. The work you are doing now is making it a lot easier 
to imagine how things may fit together with a testsuite written and run 
only with junit, and to have opinions on that.

Vemund

Re: [junit] Top-level suites added

[Daniel John Debrunner <dj...@apache.org>]

Vemund Ostgaard wrote:


> I have a suggestion, based on the following view:
> 
> * I would avoid creating suites just to handle the various frameworks,
> or other configuration related things. This would mean a simpler
> structure of suites, easier to understand and maintain. If we want to
> have a  MATS suite for instance, we would not need an embedded_MATS
> suite and a client_MATS suite and so forth to build the structure.
> * It would be good if as little as possible (ideally no) magic was
> performed on a suite level. Tests should behave the same way when
> executed through the lowest level "leaf" suite as when executed as part
> of "derbyall", making it easier to reproduce problems.

The current approach has very little magic, one decorator for client and
probably one decorator for DB2 client. I strongly agree that tests
should not be overly influenced by the suite setup.

> How could this be accomplished?
> 
> Handle everything related to frameworks and configuration on a "leaf"
> level, I believe this could be done in the suite() method in each
> TestCase. Just add each Test once for each framework that this run is
> configured to use, using the correct decorator. If a test is not
> supposed to be run for a given framework, that information should be
> maintained in that TestCase in some way, so that test is not added for
> that framework. Default would be to run all Tests in all frameworks.

This is an interesting idea, some thoughts jump out at me:

 - A single test would potentially run in many configurations:
     embedded, client, db2 client, encryption DES, encryption AES, ...

 - The large number of decorators required per test could factor into
large memory use for an entire suite, not sure of the overhead of each
TestSetup.

 - Embedded tests would be running with the network server booted by
default, not sure of the impact of this, starting the network server for
each individual TestCase would slow an entire suite down too much

 - It does make running a single TestCase in all configurations easy

 - The potential requirement to modify all tests when a new
configuration is added into the run, e.g. if I add a configuration that
only works in 10% of the tests (that's my itch) then I have to modify
the other 90% of tests to disable it from running. Assuming the default
as you say is to run each test in all possible configurations, I assume
this would be enforced in the base test class. Maybe that's the case
today with the current approach where a test is ideally self describing.

 - Maybe there are some configurations that are best handled at the leaf
level, e.g. embedded and client, but others are best handled at a suite
level, e.g. db2 client, encryption. So a mix of the two approaches would
be good.

I'm going to continue my current setup approach because it's not opposed
to what you approach, the majority of the work will be getting tests
converted to JUnit and running in client & db2 client mode. Ie. most of
the work overlaps anyway.

Thanks for your thoughts,
Dan.

Re: [junit] Top-level suites added

[Vemund Ostgaard <Ve...@Sun.COM>]

Daniel John Debrunner wrote:

>I added some top-level suites that are intended to be run directly by
>JUnit runners, not via the test harness. They are in the package:
>
>org.apache.derbyTesting.functionTests.suites
>
>AllPackages - A suite of all the _Suite suites for the function test
>packages
>
>Embedded - All the tests for embedded. Currently just includes
>AllPackages.suite(), but future may add more in different
>configurations, e.g. encryption.
>
>Client - All the tests for client, runs the AllPackages.suite() with a
>decorator that changes the configuration and boots the network server
>(using the existing network server decorator)
>
>All - All the tests, currently includes Client.suite() and
>Embedded.suite(). (Eventual replacement for derbyall)
>
>It's not all working fine yet, a couple of issues:
>
>  1) The network server decorator in jdbcapi._Suite clashes with the one
>in Client
>  
>
I think that adding the network server decorater to a test should just 
be a way of saying that this test needs a server running, not knowing 
wether it might allready be running. If it is not running, start it. If 
it allready runs, leave it be.

>  2) Some tests fail in the Client suite, need to investigate.
>Additional tests were failing due to some test using the old checks to
>see if it was running embedded or not, rather than the utilities
>provided in the JUnit classes.
>
>I've checked these in for others to see the direction I'm proposing,
>with the goal being all tests running under JUnit with no harness.
>
>Example of running these:
>
>java -Dderby.system.home=${PWD} junit.textui.TestRunner
>          org.apache.derbyTesting.functionTests.suites.Embedded
>
>java -Dderby.system.home=${PWD} junit.textui.TestRunner
>          org.apache.derbyTesting.functionTests.suites.Client
>
>java -Dderby.system.home=${PWD} junit.textui.TestRunner
>          org.apache.derbyTesting.functionTests.suites.All
>
>Comments?
>  
>
I have a suggestion, based on the following view:

* I would avoid creating suites just to handle the various frameworks, 
or other configuration related things. This would mean a simpler 
structure of suites, easier to understand and maintain. If we want to 
have a  MATS suite for instance, we would not need an embedded_MATS 
suite and a client_MATS suite and so forth to build the structure.
* It would be good if as little as possible (ideally no) magic was 
performed on a suite level. Tests should behave the same way when 
executed through the lowest level "leaf" suite as when executed as part 
of "derbyall", making it easier to reproduce problems.

How could this be accomplished?

Handle everything related to frameworks and configuration on a "leaf" 
level, I believe this could be done in the suite() method in each 
TestCase. Just add each Test once for each framework that this run is 
configured to use, using the correct decorator. If a test is not 
supposed to be run for a given framework, that information should be 
maintained in that TestCase in some way, so that test is not added for 
that framework. Default would be to run all Tests in all frameworks.

This would mean that when executing a single TestCase class (or suite) 
with a junit runner, the default behaviour could be to run with all 
(supported) frameworks if that is what is most common. Running with just 
a selected framework could be possible with a system property.

Vemund