[jira] [Created] (SUREFIRE-1638) untilFailureLoopCount property to run tests until failure

["Galder Zamarreño (JIRA)" <ji...@apache.org>]

Galder Zamarreño created SUREFIRE-1638:
------------------------------------------

             Summary: untilFailureLoopCount property to run tests until failure
                 Key: SUREFIRE-1638
                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1638
             Project: Maven Surefire
          Issue Type: New Feature
            Reporter: Galder Zamarreño


This is a proposal to add surefire/failsafe property called `untilFailureLoopCount`.

I often find myself in the situation where there's a random test failure happening in CI, and have to run the one or several tests continuously until it fails, maybe running them with TRACE logging.

Normally, I'd wrap the `mvn test...` call with a script that calls it in a loop, but this can be quite "slow" since each iteration is a new JVM launch...etc.

The idea behind `untilFailureLoopCount` is that you've give it a positive number, say 1000, and the test(s) would run continuously until either a failure happens or the test is executed that many times.

I've created [this randomly failing test](https://github.com/galderz/surefire-until-failure-loop-count/blob/master/src/test/java/com/acme/maven/surefire/untilfailure/UntilFailureLoopCountTest.java) and I've run it with `mvn -Dsurefire.untilFailureLoopCount=10 test` and the output would be something like:

{code}
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.034 s - in com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest
[INFO] Running com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s - in com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest
[INFO] Running com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest
[INFO] Running com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest
[INFO] Running com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.001 s <<< FAILURE! - in com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest
[ERROR] com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest.testEventualFailure Time elapsed: 0.001 s <<< FAILURE!
java.lang.AssertionError: Just messing with your testing
 at com.acme.maven.surefire.untilfailure.UntilFailureLoopCountTest.testEventualFailure(UntilFailureLoopCountTest.java:16)
{code}


If maintainers are interested in this, I can complete the PR with tests, documentation and support for other providers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)