[jira] [Commented] (SUREFIRE-2087) rerunFailingTestsCount incorrectly marks failed parametrized test as pass

["Ralph Weires (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SUREFIRE-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17678696#comment-17678696 ] 

Ralph Weires commented on SUREFIRE-2087:
----------------------------------------

I came across a very weird problem that seems related to this issue. I'm not sure if root cause is the same than described above, but it likewise concerns inconsistent surefire handling of JUnit5 tests with _@ParameterizedTest_ or {_}@RepeatedTest{_}, in a fairly specific case. We noticed that certain tests of ours (as in: specific instances of one {_}@ParameterizedTest-{_}method) were (consistently) failing / broken, but this got handled as a Flake by surefire, and in turn it would (in combination with {_}rerunFailingTestsCount{_}) +not+ break the build as it should have.

The symptom is similar than described above, in that surefire sometimes handles all instances of such templated tests as "one bag", and deems them to be flakes - even if it is always the very same instance (e.g. always the 2nd one) that fails - i.e. anything but a flaky case.

For some reason, I only have this problem for tests annotated with one specific custom {_}name{_}-value of the template-annotation, but that inconsistent handling is fully reproducible. Namely, with a test-method annotated like this, surefire is somehow not able to separate the different templated instances of the method from each other properly:

 
{code:java}
@ParameterizedTest(name = "[{index}]")
@RepeatedTest(value = 5, name = "[{currentRepetition}]"){code}
 

 

The problem seems limited to this very name-pattern - as soon as I change anything in that pattern (e.g. remove the square brackets, or add anything else to it), it works fine.  It seems quite counter-intuitive, but I could reproduce it consistently with 3.0.0-M8 (and also the same on all earlier 3.0.0 versions with JUnit5-support). I was not able to pinpoint anything myself, after having a quick look at the surefire source code. A workaround is simple -> just avoid / replace usages of this "problematic" name-pattern.

 

Repo with a minimal project to reproduce - just running _mvn test_ on it should show the discrepancies, i.e. Flakes that clearly aren't flakes:
 * [https://github.com/s-rwe/SUREFIRE-2087-mishandled-templated-tests]

 

Maybe this also helps in relation to the issues described in the comments above.

> rerunFailingTestsCount incorrectly marks failed parametrized test as pass
> -------------------------------------------------------------------------
>
>                 Key: SUREFIRE-2087
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-2087
>             Project: Maven Surefire
>          Issue Type: Bug
>    Affects Versions: 3.0.0-M4
>            Reporter: Ramanan Rajendran
>            Priority: Major
>
> rerunFailingTestsCount is not working as expected with junit5 `@ParameterizedTest` 
> If 1st iteration of the test passes and the 2nd one fails, it still marks the test as flaky passed. 
> Eg: In the following run, iteration 1 with input A passed but iteration 2 with input B failed. Iteration 2 was rerun 2 times and failed in all the attempts. Instead of failing TestA, it's marked as flaky giving an overall pass for the test suite. I believe it's using the Run 1 PASS as a gate to mark the whole test as a pass. 
>  
> [INFO]  Run 1: PASS
> [ERROR]  Run 2: TestA.iteration2:101 1 expectation failed.
> Expected status code <400> doesn't match actual status code <401>.
> [ERROR]  Run 3: TestA.iteration2:101 1 expectation failed.
> Expected status code <400> doesn't match actual status code <401>.
> [ERROR]  Run 4: TestA.iteration2:101 1 expectation failed.
>  <400> doesn't match actual status code <401>.
>  
> Tests run: 23, Failures: 0, Errors: 0, Skipped: 2, Flakes: 1



--
This message was sent by Atlassian Jira
(v8.20.10#820010)