[jira] [Created] (PHOENIX-7980) Improve handling of test timeouts and resource loss

["Andrew Kyle Purtell (Jira)" <ji...@apache.org>]

Andrew Kyle Purtell created PHOENIX-7980:
--------------------------------------------

             Summary: Improve handling of test timeouts and resource loss
                 Key: PHOENIX-7980
                 URL: https://issues.apache.org/jira/browse/PHOENIX-7980
             Project: Phoenix
          Issue Type: Bug
          Components: test
            Reporter: Andrew Kyle Purtell
            Assignee: Andrew Kyle Purtell


Several Phoenix test classes run with reuseForks=true, sharing a single in-JVM minicluster across all classes that land on that same fork (via BaseTest.setUpTestDriver). This is important for reducing overall test runtime but can be problematic. When the fork's in-JVM minicluster dies this poisons every subsequent class assigned to that same reused fork and causes all manner of malfunction that currently are not handled all that well.

We should at least bound the failure blast radius with the following basic test improvements:

- Add {{<forkedProcessTimeoutInSeconds>1800</forkedProcessTimeoutInSeconds>}} to surefire/failsafe configuration. This kills any fork that hasn't finished its class in 30 min. 
- Add a global ClassRule {{Timeout(20, MINUTES)}} to {{BaseTest}}. Triggers from inside the fork and produces a failure report XML for the wedged test class.
- Detect and fail fast on discovery of a dead in-JVM cluster in a shared fork. In {{BaseTest.doSetup}} after the utility starts, add a health check that verifies that the master is up. This alone may reduce failure case wastage of test runner resources to seconds from hours.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)