[jira] [Commented] (SUREFIRE-2274) failOnFlakeCount does not work for failsafe

["Elliotte Rusty Harold (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SUREFIRE-2274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17907016#comment-17907016 ] 

Elliotte Rusty Harold commented on SUREFIRE-2274:
-------------------------------------------------

Possibly related: SUREFIRE-1644

> failOnFlakeCount does not work for failsafe
> -------------------------------------------
>
>                 Key: SUREFIRE-2274
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-2274
>             Project: Maven Surefire
>          Issue Type: Bug
>          Components: Maven Failsafe Plugin
>    Affects Versions: 3.5.0
>            Reporter: Daniel Fesenmeyer
>            Priority: Major
>
> The property failOnFlakeCount seems not to work for the failsafe plugin. At least not as it is documented here: [https://maven.apache.org/surefire/maven-failsafe-plugin/verify-mojo.html]
> h2. Expected behavior
> When using the failsafe configuration property {{{}failOnFlakeCount{}}}, I'd expect that the build fails in case there are more than the specified count of flakes.
> h2. Actual behavior
> The build succeeds, even when there are more than the specified count of flakes. On the console, and the _TEST-***.xml_ report files, flakes are correctly reported. However, the report {_}failsafe-summary.xml{_}, does not contain a {{flakes}} property, and thus the verify step fails.
> h2. How to reproduce
> I've created a reproducer here: [https://github.com/danielFesenmeyer/failsafe-flakecount-reproducer]
> Just run the integration tests for this simple maven project:
> mvn verify
> The build shows the following output, which is correct, except for the information that the build succeeded (and status code 0):
> {code:java}
> ...
> [INFO] Running de.fesenmeyer.reproducer.ReproducerIT
> [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s -- in de.fesenmeyer.reproducer.ReproducerIT
> [INFO] 
> [INFO] Results:
> [INFO] 
> [WARNING] Flakes: 
> [WARNING] de.fesenmeyer.reproducer.ReproducerIT.test1
> [ERROR]   Run 1: ReproducerIT.test1:12 » IllegalState Try 0 failed
> [ERROR]   Run 2: ReproducerIT.test1:12 » IllegalState Try 1 failed
> [INFO]   Run 3: PASS
> [INFO] 
> [WARNING] de.fesenmeyer.reproducer.ReproducerIT.test2
> [ERROR]   Run 1: ReproducerIT.test2:17 » IllegalState Try 0 failed
> [ERROR]   Run 2: ReproducerIT.test2:17 » IllegalState Try 1 failed
> [INFO]   Run 3: PASS
> [INFO] 
> [INFO] 
> [WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Flakes: 2
> [INFO] 
> [INFO] 
> [INFO] --- failsafe:3.5.0:verify (default) @ failsafe-flakecount-reproducer ---
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> ...
>  {code}
> h2. Hint
> Probably the Feature currently works for Surefire, it seems like it has been only tested with Surefire.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)