[jira] [Comment Edited] (SUREFIRE-1627) Surefire report contains wrong totals on number of tests and errors when used with rerunFailingTestsCount flag

["Tibor Digana (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SUREFIRE-1627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16750513#comment-16750513 ] 

Tibor Digana edited comment on SUREFIRE-1627 at 3/26/19 12:13 AM:
------------------------------------------------------------------

In the future version we would rework {{StatelessXmlReporter}}.
We will check this too.


was (Author: tibor17):
In the future version we reworked {{StatelessXmlReporter}}.
We will check this too.

> Surefire report contains wrong totals on number of tests and errors when used with rerunFailingTestsCount flag
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: SUREFIRE-1627
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1627
>             Project: Maven Surefire
>          Issue Type: Bug
>          Components: Maven Surefire Report Plugin
>    Affects Versions: 2.19.1
>            Reporter: Alex Filatau
>            Assignee: Tibor Digana
>            Priority: Major
>             Fix For: 3.0.0-M4
>
>         Attachments: TEST-mainTest.xml
>
>
> Steps:
>  # Make sure you have at least one Test class with at least one test method
>  # Make this test method be constantly failing throwing an `AssertionError`
>  # Run the test via maven using rerun option: `mvn clean test -Dsurefire.rerunFailingTestsCount=5`
>  # Compare build output and xml report generated by Surefire.
> Issue: the report ` target/surefire-reports/TEST-mainTest.xml` contains wrong counts on total number of tests and failures (in the top level element):
> {code:xml}
> <testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd" name="mainTest" time="0.05" tests="6" errors="0" skipped="0" failures="6">
> {code}
> Please note that there is only 1 element *testcase* in the body (versus 6 in the totals), and only one element *failure* as well (versus 6 in the totals). The report is considering each re-run as a separate test and each *rerunFailure* as a failure count. That leads to inconsistencies when parsing the report since there is mismatch between totals and actual number of tree elements. 
> Also it contradicts general reporting logic one can see in the build output:
> {code:java}
> Results :
> Failed tests: 
> mainTest.failingTest(mainTest)
>   Run 1: mainTest.failingTest:9 Failing test
>   Run 2: mainTest.failingTest:9 Failing test
>   Run 3: mainTest.failingTest:9 Failing test
>   Run 4: mainTest.failingTest:9 Failing test
>   Run 5: mainTest.failingTest:9 Failing test
>   Run 6: mainTest.failingTest:9 Failing test
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> {code}
>  
> To sum up, reporting counts not just  *rerunFailure*, but also *flakyFailure* as a new failure, and each rerun as a new test for totals counts.
> I see the same issue with Surefire versions 2.22.1 and 3.0.0-M3 as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)