Compatibility test (was: Re: More. Trouble with JVMInfo)

[Kristian Waagan <kr...@oracle.com>]

On 27.09.11 18:46, Kathey Marsden wrote:
> On 9/27/2011 7:26 AM, Knut Anders Hatlen wrote:
>> Another question is whether mixed versions is a configuration we need
>> to support (I don't think it's stated explicitly anywhere that we
>> actually do support it?).
> Years ago, I know this support was very important to several of our
> large consumers, but perhaps needs have changed. I am sure those that
> expressed the need for this to keep working are not up to 10.7 yet. I
> had thought CompatibiltyTest was doing the basic testing for mixed jars.
> I guess though, that test is just for basic protocol testing and must
> use separate class loaders or the basic testing would have failed.

--- Compatibility test rewrite ---

I recently spent time rewriting the compatibility test (DERBY-2076), and 
can confirm that it doesn't test mixed version jars. It uses separate 
processes for the client and the server.

However, while rewriting the test, I did find one or two problems 
related to mixing code from different releases. For instance, you cannot 
shut down a really old server using a newer NetworkServerControl. In 
this case I had to fork off a process to shut down the server using the 
old code.
The test is dependent on being able to run the newest derbyTesting.jar 
with the client and the server code (for instance to find the code for 
stored procedures).

I'd be interested in hearing what people believe the compatibility test 
should be able to do. The new version of the test only runs a test suite 
with a set of client versions against a set of server versions. It 
doesn't yet support running the client and the server with different JVM 
versions.


--- Compatibility test coverage ---

I have no metrics for what kind of coverage the current compatibility 
test gives, but I suspect it is rather low. Adding tests to the 
compatibility suite is in theory independent of the test framework, but 
running for instance suites.All with mixed versions of the client and 
the server will fail for several reasons:
  o existing network server setup decorators will interfere with the 
running server
  o features not existing in the client will be attempted tested if the 
newest derbyTesting.jar is used


--- Mixed jars testing ---
--- Run only client or embedded tests when invoking _Suite ---

As an experiment I added functionality to the test framework to run only 
client or embedded tests. This included minor changes in 
TestConfiguration and relatively small changes in a whole lot of 
suite()-methods. I only carried out the experiment for lang and jdbcapi.

When running lang._Suite with a 10.6 server and trunk client/testing, I 
got this from the swing testrunner:
589/618, 67 errors, 23 failures.
Many of the issues were related to BOOLEAN and truncate table (not 
supported by 10.6).

When running lang._Suite with a trunk server/testing and 10.6 client, I 
got this from the swing testrunner:
611/618, 4 errors, 26 failures.
Again many of the failures were related BOOLEAN.

I don't know if it is worth doing this kind of testing, and how to 
interpret the results, but the ability to run only embedded or client 
tests may be something people find valuable.

Mixed jars testing requires manual inspection of the results, since the 
test cases don't carry information about which versions of Derby support 
the features being tested.

To detect regressions it might be better to run the (old) tests against 
a newer server?


-- 
Kristian


[ snip ]

Re: Compatibility test

[Kathey Marsden <km...@sbcglobal.net>]

On 10/5/2011 8:22 AM, Kristian Waagan wrote:
>
> To detect regressions it might be better to run the (old) tests 
> against a newer server?
>
Yes, running the old tests against the new product jars almost always  
pops issues, if not product issues, at least release note issues.    
When I run with mixed client and server versions I always use the old 
derbyTesting.jar.

Because the tests are really just a Derby application, I think they are 
likely to pop issues that users will encounter.   Analysis is hard when 
they are just run once a year or so, but I have always found the effort 
worth it.

One thought I had had would be that it would be nice  if the latest 
tests on the previous release branch were run and expected to continue 
running against trunk.  So if we made an incompatible change on the 
trunk, the tests on the 10.8 branch would need to be changed in the same 
we expect users to change their own applications and then we would be 
more likely to flag and understand  incompatibilities as they are 
introduced.  There is some general work that would have to be done with 
the tests first, such as some of the tests are dependent on the number 
of system tables.  It would be nice in that behavior changes might  be 
recognized and documented or fixed at the time the incompatibility is 
introduced. The risk is that people might be tempted to just backport 
the test changes  they made in trunk without taking the time to think of 
the impact to users. Also it might be time consuming.


Kathey