You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by "Alex Filatau (JIRA)" <ji...@apache.org> on 2019/01/23 18:54:00 UTC
[jira] [Created] (SUREFIRE-1627) Surefire report contains wrong
totals on number of tests and errors when used with rerunFailingTestsCount
flag
Alex Filatau created SUREFIRE-1627:
--------------------------------------
Summary: Surefire report contains wrong totals on number of tests and errors when used with rerunFailingTestsCount flag
Key: SUREFIRE-1627
URL: https://issues.apache.org/jira/browse/SUREFIRE-1627
Project: Maven Surefire
Issue Type: Bug
Components: Maven Surefire Report Plugin
Affects Versions: 2.19.1
Reporter: Alex Filatau
Attachments: TEST-mainTest.xml
Steps:
# Make sure you have at least one Test class with at least one test method
# Make this test method be constantly failing throwing an `AssertionError`
# Run the test via maven using rerun option: `mvn clean test -Dsurefire.rerunFailingTestsCount=5`
# Compare build output and xml report generated by Surefire.
Issue: the report ` target/surefire-reports/TEST-mainTest.xml` contains wrong counts on total number of tests and failures (in the top level element):
{code:xml}
<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd" name="mainTest" time="0.05" tests="6" errors="0" skipped="0" failures="6">
{code}
Please note that there is only 1 element *testcase* in the body (versus 6 in the totals), and only one element *failure* as well (versus 6 in the totals). The report is considering each re-run as a separate test and each *rerunFailure* as a failure count. That leads to inconsistencies when parsing the report since there is mismatch between totals and actual number of tree elements.
Also it contradicts general reporting logic one can see in the build output:
{code:java}
Results :
Failed tests:
mainTest.failingTest(mainTest)
Run 1: mainTest.failingTest:9 Failing test
Run 2: mainTest.failingTest:9 Failing test
Run 3: mainTest.failingTest:9 Failing test
Run 4: mainTest.failingTest:9 Failing test
Run 5: mainTest.failingTest:9 Failing test
Run 6: mainTest.failingTest:9 Failing test
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
{code}
To sum up, reporting counts not just *rerunFailure*, but also *flakyFailure* as a new failure, and each rerun as a new test for totals counts.
I see the same issue with Surefire versions 2.22.1 and 3.0.0-M3 as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)