[jira] [Created] (SUREFIRE-1219) skipAfterFailureCount should not count flaky tests as failures when using rerunFailingTestsCount

["Sean Flanigan (JIRA)" <ji...@apache.org>]

Sean Flanigan created SUREFIRE-1219:
---------------------------------------

             Summary: skipAfterFailureCount should not count flaky tests as failures when using rerunFailingTestsCount
                 Key: SUREFIRE-1219
                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1219
             Project: Maven Surefire
          Issue Type: Bug
    Affects Versions: 2.19.1
            Reporter: Sean Flanigan


According to https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html "failed tests within re-run phase are not included in skipAfterFailureCount". For instance, if skipAfterFailureCount is 1, but every test passes on the second attempt, the tests are "flaky" but not "failures".  This should be reflected in summaries, in XML result files and in fail fast (skipAfterFailureCount), but this is not the case.

If I have 10 flaky-but-not-failing tests, set rerunFailingTestsCount=1 and skipAfterFailureCount=5:

1. I would expect something like this to be printed on the console:
  Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Flakes: 10

2. The XML report should list <flakyFailure>s but no <failures>.

3. All tests should be run, not just 5 of them, in other words skipAfterFailureCount should not count the flaky tests as part of the failure count.

I have some tests demonstrating the problem, which I will attach to this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)