You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by "Elliotte Rusty Harold (Jira)" <ji...@apache.org> on 2022/12/09 12:38:00 UTC
[jira] [Resolved] (SUREFIRE-2087) rerunFailingTestsCount incorrectly marks failed parametrized test as pass
[ https://issues.apache.org/jira/browse/SUREFIRE-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Elliotte Rusty Harold resolved SUREFIRE-2087.
---------------------------------------------
Resolution: Not A Problem
"If 1st iteration of the test passes and the 2nd one fails, it still marks the test as flaky passed."
That sounds correct. If there's something I'm missing about this and you can point to docs that say otherwise, please reopen. A reproducible example might also help.
> rerunFailingTestsCount incorrectly marks failed parametrized test as pass
> -------------------------------------------------------------------------
>
> Key: SUREFIRE-2087
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2087
> Project: Maven Surefire
> Issue Type: Bug
> Affects Versions: 3.0.0-M4
> Reporter: Ramanan Rajendran
> Priority: Critical
>
> rerunFailingTestsCount is not working as expected with junit5 `@ParameterizedTest`
> If 1st iteration of the test passes and the 2nd one fails, it still marks the test as flaky passed.
> Eg: In the following run, iteration 1 with input A passed but iteration 2 with input B failed. Iteration 2 was rerun 2 times and failed in all the attempts. Instead of failing TestA, it's marked as flaky giving an overall pass for the test suite. I believe it's using the Run 1 PASS as a gate to mark the whole test as a pass.
>
> [INFO] Run 1: PASS
> [ERROR] Run 2: TestA.iteration2:101 1 expectation failed.
> Expected status code <400> doesn't match actual status code <401>.
> [ERROR] Run 3: TestA.iteration2:101 1 expectation failed.
> Expected status code <400> doesn't match actual status code <401>.
> [ERROR] Run 4: TestA.iteration2:101 1 expectation failed.
> <400> doesn't match actual status code <401>.
>
> Tests run: 23, Failures: 0, Errors: 0, Skipped: 2, Flakes: 1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)