[jira] [Created] (SUREFIRE-1970) rerunFailingTestsCount option marks some spock tests as false positive

["Casey Jones (Jira)" <ji...@apache.org>]

Casey Jones created SUREFIRE-1970:
-------------------------------------

             Summary: rerunFailingTestsCount option marks some spock tests as false positive
                 Key: SUREFIRE-1970
                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1970
             Project: Maven Surefire
          Issue Type: Bug
          Components: Maven Surefire Plugin
    Affects Versions: 3.0.0-M5, 3.0.0-M4
            Reporter: Casey Jones


I have a project which uses [Spock Framework|https://spockframework.org/] for tests. There is a single test which intentionally fails, no matter how many times it was ran:
{code:java}
class FalsePositiveTest extends Specification {
    def "test"() {
        expect:
        1 == number
        where:
        number << [2, 3]    
    }
} {code}
When test is executed with {{mvn test}} command, the build fails, as expected. 

However, when using surefire's [rerunFailingTestsCount|https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html] option, the test in question is executed two times, as expected, but then it is marked as flaky, and the build is successful:
{code:java}
mvn test -Dsurefire.rerunFailingTestsCount=1 
...
Warning:  Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
126[INFO] 
127[INFO] ------------------------------------------------------------------------
128[INFO] BUILD SUCCESS
129[INFO] ------------------------------------------------------------------------{code}
Please check the links below for a full build log.

This issue happens only when using Spock's data-driven tests with a [where block|https://spockframework.org/spock/docs/1.0/data_driven_testing.html] and surefire's {{rerunFailingTestsCount}} option together. It seems like there is something wrong in the way surefire detects if test has failed or passed.

I asked this [question on StackOverflow|https://stackoverflow.com/questions/70454980/spock-data-driven-tests-surefire-rerunfailingtestscount-gives-false-positive-t] and Spock's maintainer suggested to raise an issue in surefire's bugtracker.


Here is a simple github project which reproduces the issue: [https://github.com/streetturtle/FalsePositiveTest]

Please check build logs of a github action here: [https://github.com/streetturtle/FalsePositiveTest/runs/4613803891?check_suite_focus=true#step:4:128]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)