[Db-derby Wiki] Update of "TestRegressions" by DavidVanCouvering

[Apache Wiki <wi...@apache.org>]

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Db-derby Wiki" for change notification.

The following page has been changed by DavidVanCouvering:
http://wiki.apache.org/db-derby/TestRegressions

New page:
= Handling Test Regressions =

It is crucial that our regression test suite, derbyall, run with no failures.  This page describes mechanisms and guidelines to help ensure that this is the case.

== Awareness ==

It's important for Derby developers to be aware of the state of the regression suite.  For this reason, any time an automated regression test fails, an email will be sent to derby-dev indicating that there were test failures and pointing to which tests failed.

We are also creating a separate mailing list called derby-test-results that posts the results of each regression test run, regardless of whether it passed or failed.  A developer can subscribe to this if they want to keep a constant pulse of test results.  Of one particular use is knowing whether your recent checking caused tests to hang or fail to build so that test results don't even get sent out.

Automated regression test results with lots of detail about the failure and showing the changes that were applied for each revision can be obtained at http://www.multinet.no/~solberg/public/Apache/index.html.

== Managing Test Regressions ==

What follows are guidelines for how test regressions should be handled

* A committer is responsible for committing changes that do not regress the tests.  If they notice that a commit they made appears to have caused a test failure, they should immediately work to resolve it, working with the contributor if necessary.  Of course a committer can not test on all platforms, and may not even be able to resolve a test failure on a platform to which they do not have access, so in some cases, they may need to work with someone whose itch is to have Derby work on that platform to fix the test

* If anyone notices that a test that is failing and it is not immediately resolved, you are strongly encouraged to log a JIRA issue for it.  The issue should be given a high priority (Critical for any test failure that crosses all platforms) and assigned to the JIRA category Regression.

* If necessary, a committer may take the matter into their own hands by directly contacting the person they believe is responsible for the regression or even vetoing all code commits that are not related to removing test regressions.

* For some test regressions, the issue may be minor and fixing the test may be significant.  In this case the test should be removed from derbyall, or configured to not run in the configurations which cause failures, and the JIRA issue category should be changed to Test.