[jira] [Updated] (SUREFIRE-1627) Surefire report contains wrong totals on number of tests and errors when used with rerunFailingTestsCount flag

["Michael Osipov (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SUREFIRE-1627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael Osipov updated SUREFIRE-1627:
-------------------------------------
    Fix Version/s: 3.0.0-M9
                       (was: 3.0.0-M8)

> Surefire report contains wrong totals on number of tests and errors when used with rerunFailingTestsCount flag
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: SUREFIRE-1627
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1627
>             Project: Maven Surefire
>          Issue Type: Bug
>          Components: Maven Surefire Report Plugin
>    Affects Versions: 2.19.1
>            Reporter: Alex Filatau
>            Priority: Major
>             Fix For: 3.0.0-M9
>
>         Attachments: TEST-mainTest.xml
>
>
> Steps:
>  # Make sure you have at least one Test class with at least one test method
>  # Make this test method be constantly failing throwing an `AssertionError`
>  # Run the test via maven using rerun option: `mvn clean test -Dsurefire.rerunFailingTestsCount=5`
>  # Compare build output and xml report generated by Surefire.
> Issue: the report ` target/surefire-reports/TEST-mainTest.xml` contains wrong counts on total number of tests and failures (in the top level element):
> {code:xml}
> <testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd" name="mainTest" time="0.05" tests="6" errors="0" skipped="0" failures="6">
> {code}
> Please note that there is only 1 element *testcase* in the body (versus 6 in the totals), and only one element *failure* as well (versus 6 in the totals). The report is considering each re-run as a separate test and each *rerunFailure* as a failure count. That leads to inconsistencies when parsing the report since there is mismatch between totals and actual number of tree elements. 
> Also it contradicts general reporting logic one can see in the build output:
> {code:java}
> Results :
> Failed tests: 
> mainTest.failingTest(mainTest)
>   Run 1: mainTest.failingTest:9 Failing test
>   Run 2: mainTest.failingTest:9 Failing test
>   Run 3: mainTest.failingTest:9 Failing test
>   Run 4: mainTest.failingTest:9 Failing test
>   Run 5: mainTest.failingTest:9 Failing test
>   Run 6: mainTest.failingTest:9 Failing test
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> {code}
>  
> To sum up, reporting counts not just  *rerunFailure*, but also *flakyFailure* as a new failure, and each rerun as a new test for totals counts.
> I see the same issue with Surefire versions 2.22.1 and 3.0.0-M3 as well.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)