[jira] [Assigned] (SUREFIRE-1219) skipAfterFailureCount should not count flaky tests as failures when using rerunFailingTestsCount

["Tibor Digana (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SUREFIRE-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tibor Digana reassigned SUREFIRE-1219:
--------------------------------------

    Assignee: Tibor Digana

> skipAfterFailureCount should not count flaky tests as failures when using rerunFailingTestsCount
> ------------------------------------------------------------------------------------------------
>
>                 Key: SUREFIRE-1219
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1219
>             Project: Maven Surefire
>          Issue Type: Bug
>    Affects Versions: 2.19.1
>            Reporter: Sean Flanigan
>            Assignee: Tibor Digana
>         Attachments: SUREFIRE-1219-tests.zip
>
>
> According to https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html "failed tests within re-run phase are not included in skipAfterFailureCount". For instance, if skipAfterFailureCount is 1, but every test passes on the second attempt, the tests are "flaky" but not "failures".  This should be reflected in summaries, in XML result files and in fail fast (skipAfterFailureCount), but this is not the case.
> If I have 10 flaky-but-not-failing tests, set rerunFailingTestsCount=1 and skipAfterFailureCount=5:
> 1. I would expect something like this to be printed on the console:
>   Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Flakes: 10
> 2. The XML report should list <flakyFailure>s but no <failures>.
> 3. All tests should be run, not just 5 of them, in other words skipAfterFailureCount should not count the flaky tests as part of the failure count.
> I have some tests demonstrating the problem, which I will attach to this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)