upgrade/compatibility testing done in the past.

[mike matrigali <mi...@gmail.com>]

>> Last time Kathey ran the upgrade/compatibility testing (run with an
>> older release's derbyTesting.jar with newer derby*.jar) she ran into a
>> lot of test failures because of expected changes, and I think no real
>> failures. I  don't intend to make the effort at this time, I do not
>> have the time, and I doubt Kathey has either.
> Can you help me understand what this test entails beyond what's in the
> upgrade and compatibility tests we run every night?

the upgrade and compatibility tests are very small and test the things
that someone has thought of and added specific tests for.  The goal of 
this upgrade/compatibility testing was to catch
compatibility problems of existing applications with new software.

I believe the intent is to use as large a portion of the junit tests 
from some past release and use it as a model of expected behavior in
that release.  Then you run those exact tests against a new software
release, with the assumption that diffs are possible upward 
compatibility problems.    I believe it was most successful in calling 
out possible issues that should be noted for release notes.  I do not
know if the db used was old db in soft upgrade mode, an old db hard
upgraded to current, or a new db.  All probably are interesting, but
would lean toward old db in soft upgrade mode being most interesting
as a model for an old application looking to upgrade the db software
but get same behavior.

The problem was that this testing in the past has generated a lot of
noise to bug/release note ratio.  It was by hand and don't think much
was done to make future testing easier.  I think it only in the past
was done starting from either most recent or 2 release back to current
release.  I believe it was on the order of weeks to interpret part
time.

I think knut's recent run tests that upgraded db's work with new 
software as the new software expects.

I think this other testing looks to see if new software acts as old
applications expect.

Neither does a complete job, but I think both are a good leverage of the
existing test framework to test different code paths for compatibility.