Re: Compatibility test

[Kathey Marsden <km...@sbcglobal.net>]

On 10/5/2011 8:22 AM, Kristian Waagan wrote:
>
> To detect regressions it might be better to run the (old) tests 
> against a newer server?
>
Yes, running the old tests against the new product jars almost always  
pops issues, if not product issues, at least release note issues.    
When I run with mixed client and server versions I always use the old 
derbyTesting.jar.

Because the tests are really just a Derby application, I think they are 
likely to pop issues that users will encounter.   Analysis is hard when 
they are just run once a year or so, but I have always found the effort 
worth it.

One thought I had had would be that it would be nice  if the latest 
tests on the previous release branch were run and expected to continue 
running against trunk.  So if we made an incompatible change on the 
trunk, the tests on the 10.8 branch would need to be changed in the same 
we expect users to change their own applications and then we would be 
more likely to flag and understand  incompatibilities as they are 
introduced.  There is some general work that would have to be done with 
the tests first, such as some of the tests are dependent on the number 
of system tables.  It would be nice in that behavior changes might  be 
recognized and documented or fixed at the time the incompatibility is 
introduced. The risk is that people might be tempted to just backport 
the test changes  they made in trunk without taking the time to think of 
the impact to users. Also it might be time consuming.


Kathey