Question about client compatibility testing and DERBY-516 (paging Rick #:))

[Kathey Marsden <km...@sbcglobal.net>]

I noticed that DERBY-516 was closed with the addition of  the new test
jdbcapi\CompatibilityTest.java. which adds a nice framework for some
basic compatibility testing.

By default it seems that this test  just runs with 10.2 client and 10.2
server so doesn't give us any additional coverage in derbyAll for
compatibility testing.  I was wondering if there were plans to check the
derby release jars into the trunk so that the compatibility testing
would be performed as part of the normal testing procedures?  These jars
could also be used in  DERBY-514 to  enable upgrade testing.


Thanks

Kathey

Re: Question about client compatibility testing and DERBY-516 (paging Rick #:))

[Kathey Marsden <km...@sbcglobal.net>]

Rick Hillegas wrote:

> Hi  Kathey,
>
> That sounds like a useful combination: 10.1.1.0 client against the
> mainline server, both running on the 1.4 vm. You see a lot of support
> cases and are in a good position to describe the main execution path.
> I don't have a lot of perspective here.
>
I think trunk server with 10.1.1.0 client and trunk client with 10.1.1.0
server with whatever jvm the developer is using for derbyall would
provide a good basic sanity check for protocol  compatibility with 
client and server changes on the trunk.

Ultimately as yous say  someone would be well advised to run derbyall
with the various clients on the branches before releases. 

Kathey

Re: Question about client compatibility testing and DERBY-516 (paging Rick #:))

[Rick Hillegas <Ri...@Sun.COM>]

Hi  Kathey,

That sounds like a useful combination: 10.1.1.0 client against the 
mainline server, both running on the 1.4 vm. You see a lot of support 
cases and are in a good position to describe the main execution path. I 
don't have a lot of perspective here.

Cheers,
-Rick

Kathey Marsden wrote:

>Rick Hillegas wrote:
>
>  
>
>>Hi Kathey,
>>
>>My initial ramblings on this topic start out at the end of August in
>>the email thread "client/server compatibility testing". There I
>>worried that over time, the compatibility tests could grow large
>>(taking maybe 5 minutes per combination) and so, if run for all the
>>combinations, would make derbyall take too long. That's why we're only
>>runing one combination as part of derbyall. That combination doesn't
>>really track a compatibility issue, it just tracks regressions which
>>might creep into the test as other code changes.
>>
>>I'm certainly in favor of running all the combinations on a nightly or
>>weekly basis and as a sanity check when cutting release candidates.
>>
>>    
>>
>It seems that it would be worthwhile to enable the test with the 10.1
>jars (original client release)  as it would make a clear statement of
>the minimum client/server jar combinations that are expected work, would
>probably catch most things that might break over time, and hopefully
>would not add too much time.
>
>What do you think?
>
>Kathey
>
>
>
>
>
>  
>

Re: Question about client compatibility testing and DERBY-516 (paging Rick #:))

[Kathey Marsden <km...@sbcglobal.net>]

Rick Hillegas wrote:

> Hi Kathey,
>
> My initial ramblings on this topic start out at the end of August in
> the email thread "client/server compatibility testing". There I
> worried that over time, the compatibility tests could grow large
> (taking maybe 5 minutes per combination) and so, if run for all the
> combinations, would make derbyall take too long. That's why we're only
> runing one combination as part of derbyall. That combination doesn't
> really track a compatibility issue, it just tracks regressions which
> might creep into the test as other code changes.
>
> I'm certainly in favor of running all the combinations on a nightly or
> weekly basis and as a sanity check when cutting release candidates.
>
It seems that it would be worthwhile to enable the test with the 10.1
jars (original client release)  as it would make a clear statement of
the minimum client/server jar combinations that are expected work, would
probably catch most things that might break over time, and hopefully
would not add too much time.

What do you think?

Kathey

Re: Question about client compatibility testing and DERBY-516 (paging Rick #:))

[Rick Hillegas <Ri...@Sun.COM>]

Hi Kathey,

My initial ramblings on this topic start out at the end of August in the 
email thread "client/server compatibility testing". There I worried that 
over time, the compatibility tests could grow large (taking maybe 5 
minutes per combination) and so, if run for all the combinations, would 
make derbyall take too long. That's why we're only runing one 
combination as part of derbyall. That combination doesn't really track a 
compatibility issue, it just tracks regressions which might creep into 
the test as other code changes.

I'm certainly in favor of running all the combinations on a nightly or 
weekly basis and as a sanity check when cutting release candidates.

I like the idea of wiring some upgrade tests into derbyall. I think it 
would tend to enforce the discipline that developers write upgrade logic 
and tests at the same time that they checkin code which raises upgrade 
problems. Writing upgrade tests is a tricky problem. Technically, an 
upgrade test case consists of two pieces: 1) the Setup code which is 
compiled and run against the previous release and 2) the Verification 
code which is compiled and run against the current release. If you want 
to tackle upgrade testing by checking in jar files, you'll have to add 
some tricky build logic which builds Setup code against the previous 
release and Verification code against the current release.

Just for comparison, here's the equally tricky solution which we adopted 
for Cloudscape back in the old days: 1) The Setup code was checked into 
the previous release branch, 2) the test developer ran a script to 
populate a previous rev database with all the Setup cases then tar up 
the resulting database, 3) that tarball was checked into the current 
release, 4) the derbyall piece of the upgrade tests then grabbed that 
tarball, upgraded it, and ran the Verification code against it.

Cheers,
-Rick

Kathey Marsden wrote:

>I noticed that DERBY-516 was closed with the addition of  the new test
>jdbcapi\CompatibilityTest.java. which adds a nice framework for some
>basic compatibility testing.
>
>By default it seems that this test  just runs with 10.2 client and 10.2
>server so doesn't give us any additional coverage in derbyAll for
>compatibility testing.  I was wondering if there were plans to check the
>derby release jars into the trunk so that the compatibility testing
>would be performed as part of the normal testing procedures?  These jars
>could also be used in  DERBY-514 to  enable upgrade testing.
>
>
>Thanks
>
>Kathey
>
>
>
>  
>