[jira] [Commented] (SUREFIRE-1219) skipAfterFailureCount should not count flaky tests as failures when using rerunFailingTestsCount

["Tibor Digana (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SUREFIRE-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15127285#comment-15127285 ] 

Tibor Digana commented on SUREFIRE-1219:
----------------------------------------

[~sflanigan]
I know that it does not convince all parties but stable behavior is more important for me regardless whatever forking mechanism is set. Please see the code of junit4 and junit47-provider. The provider junit4 runs rerun per each class, however junit47 runs complete testset first and then the rerun.

> skipAfterFailureCount should not count flaky tests as failures when using rerunFailingTestsCount
> ------------------------------------------------------------------------------------------------
>
>                 Key: SUREFIRE-1219
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1219
>             Project: Maven Surefire
>          Issue Type: Bug
>    Affects Versions: 2.19.1
>            Reporter: Sean Flanigan
>         Attachments: SUREFIRE-1219-tests.zip
>
>
> According to https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html "failed tests within re-run phase are not included in skipAfterFailureCount". For instance, if skipAfterFailureCount is 1, but every test passes on the second attempt, the tests are "flaky" but not "failures".  This should be reflected in summaries, in XML result files and in fail fast (skipAfterFailureCount), but this is not the case.
> If I have 10 flaky-but-not-failing tests, set rerunFailingTestsCount=1 and skipAfterFailureCount=5:
> 1. I would expect something like this to be printed on the console:
>   Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Flakes: 10
> 2. The XML report should list <flakyFailure>s but no <failures>.
> 3. All tests should be run, not just 5 of them, in other words skipAfterFailureCount should not count the flaky tests as part of the failure count.
> I have some tests demonstrating the problem, which I will attach to this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)