You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by "Daniel Fesenmeyer (Jira)" <ji...@apache.org> on 2024/10/01 15:35:00 UTC
[jira] [Updated] (SUREFIRE-2274) failOnFlakeCount does not work for failsafe
[ https://issues.apache.org/jira/browse/SUREFIRE-2274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Fesenmeyer updated SUREFIRE-2274:
----------------------------------------
Description:
The property failOnFlakeCount seems not to work for the failsafe plugin. At least not as it is documented here: [https://maven.apache.org/surefire/maven-failsafe-plugin/verify-mojo.html]
h2. Expected behavior
When using the failsafe configuration property {{{}failOnFlakeCount{}}}, I'd expect that the build fails in case there are more than the specified count of flakes.
h2. Actual behavior
The build succeeds, even when there are more than the specified count of flakes. On the console, and the _TEST-***.xml_ report files, flakes are correctly reported. However, the report {_}failsafe-summary.xml{_}, does not contain a {{flakes}} property, and thus the verify step fails.
h2. How to reproduce
I've created a reproducer here: [https://github.com/danielFesenmeyer/failsafe-flakecount-reproducer]
Just run the integration tests for this simple maven project:
mvn verify
The build shows the following output, which is correct, except for the information that the build succeeded (and status code 0):
{code:java}
...
[INFO] Running de.fesenmeyer.reproducer.ReproducerIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s -- in de.fesenmeyer.reproducer.ReproducerIT
[INFO]
[INFO] Results:
[INFO]
[WARNING] Flakes:
[WARNING] de.fesenmeyer.reproducer.ReproducerIT.test1
[ERROR] Run 1: ReproducerIT.test1:12 » IllegalState Try 0 failed
[ERROR] Run 2: ReproducerIT.test1:12 » IllegalState Try 1 failed
[INFO] Run 3: PASS
[INFO]
[WARNING] de.fesenmeyer.reproducer.ReproducerIT.test2
[ERROR] Run 1: ReproducerIT.test2:17 » IllegalState Try 0 failed
[ERROR] Run 2: ReproducerIT.test2:17 » IllegalState Try 1 failed
[INFO] Run 3: PASS
[INFO]
[INFO]
[WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Flakes: 2
[INFO]
[INFO]
[INFO] --- failsafe:3.5.0:verify (default) @ failsafe-flakecount-reproducer ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
...
{code}
h2. Hint
Probably the Feature currently works for Surefire, it seems like it has been only tested with Surefire.
was:
The property failOnFlakeCount seems not to work for the failsafe plugin. At least not as it is documented here: [https://maven.apache.org/surefire/maven-failsafe-plugin/verify-mojo.html]
h2. Expected behavior
[|https://github.com/danielFesenmeyer/failsafe-flakecount-reproducer#expected-behavior]
When using the failsafe configuration property {{{}failOnFlakeCount{}}}, I'd expect that the build fails in case there are more than the specified count of flakes.
h2. Actual behavior
[|https://github.com/danielFesenmeyer/failsafe-flakecount-reproducer#actual-behavior]
The build succeeds, even when there are more than the specified count of flakes. On the console, and the _TEST-***.xml_ report files, flakes are correctly reported. However, the report {_}failsafe-summary.xml{_}, does not contain a {{flakes}} property, and thus the verify step fails.
h2. How to reproduce
I've created a reproducer here: [https://github.com/danielFesenmeyer/failsafe-flakecount-reproducer]
Just run the integration tests for this simple maven project:
mvn verify
The build shows the following output, which is correct, except for the information that the build succeeded (and status code 0):
{code}
...
[INFO] Running de.fesenmeyer.reproducer.ReproducerIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s -- in de.fesenmeyer.reproducer.ReproducerIT
[INFO]
[INFO] Results:
[INFO]
[WARNING] Flakes:
[WARNING] de.fesenmeyer.reproducer.ReproducerIT.test1
[ERROR] Run 1: ReproducerIT.test1:12 » IllegalState Try 0 failed
[ERROR] Run 2: ReproducerIT.test1:12 » IllegalState Try 1 failed
[INFO] Run 3: PASS
[INFO]
[WARNING] de.fesenmeyer.reproducer.ReproducerIT.test2
[ERROR] Run 1: ReproducerIT.test2:17 » IllegalState Try 0 failed
[ERROR] Run 2: ReproducerIT.test2:17 » IllegalState Try 1 failed
[INFO] Run 3: PASS
[INFO]
[INFO]
[WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Flakes: 2
[INFO]
[INFO]
[INFO] --- failsafe:3.5.0:verify (default) @ failsafe-flakecount-reproducer ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
...
{code}
h2. Hint
Probably the Feature currently works for Surefire, it seems like it has been only tested with Surefire.
> failOnFlakeCount does not work for failsafe
> -------------------------------------------
>
> Key: SUREFIRE-2274
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2274
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Failsafe Plugin
> Affects Versions: 3.5.0
> Reporter: Daniel Fesenmeyer
> Priority: Major
>
> The property failOnFlakeCount seems not to work for the failsafe plugin. At least not as it is documented here: [https://maven.apache.org/surefire/maven-failsafe-plugin/verify-mojo.html]
> h2. Expected behavior
> When using the failsafe configuration property {{{}failOnFlakeCount{}}}, I'd expect that the build fails in case there are more than the specified count of flakes.
> h2. Actual behavior
> The build succeeds, even when there are more than the specified count of flakes. On the console, and the _TEST-***.xml_ report files, flakes are correctly reported. However, the report {_}failsafe-summary.xml{_}, does not contain a {{flakes}} property, and thus the verify step fails.
> h2. How to reproduce
> I've created a reproducer here: [https://github.com/danielFesenmeyer/failsafe-flakecount-reproducer]
> Just run the integration tests for this simple maven project:
> mvn verify
> The build shows the following output, which is correct, except for the information that the build succeeded (and status code 0):
> {code:java}
> ...
> [INFO] Running de.fesenmeyer.reproducer.ReproducerIT
> [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s -- in de.fesenmeyer.reproducer.ReproducerIT
> [INFO]
> [INFO] Results:
> [INFO]
> [WARNING] Flakes:
> [WARNING] de.fesenmeyer.reproducer.ReproducerIT.test1
> [ERROR] Run 1: ReproducerIT.test1:12 » IllegalState Try 0 failed
> [ERROR] Run 2: ReproducerIT.test1:12 » IllegalState Try 1 failed
> [INFO] Run 3: PASS
> [INFO]
> [WARNING] de.fesenmeyer.reproducer.ReproducerIT.test2
> [ERROR] Run 1: ReproducerIT.test2:17 » IllegalState Try 0 failed
> [ERROR] Run 2: ReproducerIT.test2:17 » IllegalState Try 1 failed
> [INFO] Run 3: PASS
> [INFO]
> [INFO]
> [WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Flakes: 2
> [INFO]
> [INFO]
> [INFO] --- failsafe:3.5.0:verify (default) @ failsafe-flakecount-reproducer ---
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> ...
> {code}
> h2. Hint
> Probably the Feature currently works for Surefire, it seems like it has been only tested with Surefire.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)