[jira] [Updated] (DERBY-6210) Create a mechanism to exclude some testing of internal interfaces and likely to change behavior from compatibility testing

["Mike Matrigali (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/DERBY-6210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mike Matrigali updated DERBY-6210:
----------------------------------

    Component/s: Test

> Create a mechanism to exclude some testing of internal interfaces and likely to change behavior from compatibility testing
> --------------------------------------------------------------------------------------------------------------------------
>
>                 Key: DERBY-6210
>                 URL: https://issues.apache.org/jira/browse/DERBY-6210
>             Project: Derby
>          Issue Type: Improvement
>          Components: Test
>    Affects Versions: 10.10.1.1
>            Reporter: Kathey Marsden
>
> For compatibility and upgrade testing, in order to expose bugs and omitted release notes I run the tests from an older branch against new releases.  The premise is that the functional tests are really just a JDBC application which should continue to pass with the new version.
> The JUnit tests for Derby though are not pure functional tests which are backward compatible.  Often they make checks on things that are likely to change with new versions. This creates a large amount of noise in the testing: For example:
> 1) Checks for the exact contents or number of system tables.
> 2) Checks for "Not implemented" JDBC API calls which might become implemented.
> 3) Tests for client specific behavior which we expect might change to match embedded.
> 4) Unit tests which use internal interfaces that are likely to change.
> 5) Metadata tests which test the exact number of columns when 
> 6) Diff based tests which are likely to change with message changes.
> It would be good to have a flag to omit these types of tests when doing compatibility testing.
> My thought is to have a system property derby.tests.testCompat=true   and a method in BaseTestCase to test the property testCompat().
> Then blocks of testing which should not be run or should have a different behavior with compatibility testing can be flagged as:
> if (! testCompat())  {
> // do testing  that we think might change, e.g. system table query.
> }
> This will allow the more detailed testing under normal circumstances but take some of the pain out of the compatibility testing moving forward.  It would require folks to think as they add new tests whether their testing as to whether they expect the tested behavior  to remain stable moving forward and put the block around it if they do not.
> I welcome feedback on whether this or some other approach is preferable.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)