Using old Derby releases with tests

[Kristian Waagan <kr...@oracle.com>]

Hi,

As part of working on DERBY-2076, the issue of using old Derby releases 
in tests came up.
Currently, there are two mechanisms in play:
  a) If derbyTesting.oldReleasePath is set, use that directory.
  b) If the above isn't set, the upgrade test will attempt to download 
the required Derby files from the Apache web site.

I'd like the basic compatibility test, in a minimum acceptance test 
configuration, to run automatically as part of suites.All. To do that I 
need access to the old releases. With (a) everything is ok, but it's up 
to the developer to keep the bits updated. This is fine for nightly 
tests etc.

Mechanism (b) is getting problematic if more than one test is 
downloading the bits. I'd like to hear if people think it is feasible to 
improve that mechanism to allow multiple tests and ideally multiple test 
runs to share the downloaded files. These questions come to mind:
  1) Is it acceptable to go online and download Derby releases 
automatically onto the user machine?
  2) Where do we store them?
     Ideally, we wouldn't want to download the bits every time 
suites.All is run.
     Suggestions:
         o ~/.derbyTestingReleases (with a suitable alternative on Windows)
         o current dir used when running tests (i.e. alongside 'system/')
         o user specified directory, don't download if not specified 
(i.e. don't run the tests)

For simplicity we can also ditch the current download mechanism. That 
requires developers to always specify derbyTesting.oldReleasePath. I'm 
using a script to run the tests, so it's not a problem for me. However, 
if I were to run tests manually the chances that I would forget to 
specify that property are significant.

This feature would probably only be used by tests for upgrade and 
version compatibility. Have I forgotten any other use cases?


Thanks,
-- 
Kristian

Re: Using old Derby releases with tests

[Knut Anders Hatlen <kn...@oracle.com>]

Kristian Waagan <kr...@oracle.com> writes:

> Hi,
>
> As part of working on DERBY-2076, the issue of using old Derby
> releases in tests came up.
> Currently, there are two mechanisms in play:
>  a) If derbyTesting.oldReleasePath is set, use that directory.
>  b) If the above isn't set, the upgrade test will attempt to download
> the required Derby files from the Apache web site.
>
> I'd like the basic compatibility test, in a minimum acceptance test
> configuration, to run automatically as part of suites.All. To do that
> I need access to the old releases. With (a) everything is ok, but it's
> up to the developer to keep the bits updated. This is fine for nightly
> tests etc.
>
> Mechanism (b) is getting problematic if more than one test is
> downloading the bits. I'd like to hear if people think it is feasible
> to improve that mechanism to allow multiple tests and ideally multiple
> test runs to share the downloaded files. These questions come to mind:
>  1) Is it acceptable to go online and download Derby releases
> automatically onto the user machine?
>  2) Where do we store them?
>     Ideally, we wouldn't want to download the bits every time
> suites.All is run.
>     Suggestions:
>         o ~/.derbyTestingReleases (with a suitable alternative on Windows)
>         o current dir used when running tests (i.e. alongside 'system/')
>         o user specified directory, don't download if not specified
> (i.e. don't run the tests)
>
> For simplicity we can also ditch the current download mechanism. That
> requires developers to always specify derbyTesting.oldReleasePath. I'm
> using a script to run the tests, so it's not a problem for me.
> However, if I were to run tests manually the chances that I would
> forget to specify that property are significant.

Although it would be nice to have everything set up automatically, I'd
be fine with not downloading the old jars if derbyTesting.oldReleasePath
is not set. Combining it with your suggestion above and make the tests
look for the old jars at a well-known location (like
~/.derbyTestingReleases) would make it a one-time operation to set it up
(download, and possibly make a symlink if you don't want to store them
in ~/.derbyTestingReleases), and you wouldn't have to remember setting
derbyTesting.oldReleasePath every time you run the tests.

Currently, the upgrade tests are silently skipped if the jars cannot be
found. Whether we download the jars automatically or require them to be
downloaded manually, it might be more helpful if the tests had failed if
the jars couldn't be found for some reason, so that we notice if we have
a bad setup.

> This feature would probably only be used by tests for upgrade and
> version compatibility. Have I forgotten any other use cases?
>
>
> Thanks,

-- 
Knut Anders