[jira] [Created] (SUREFIRE-1336) Sure-Fire test run totals don't match TestNG report on failure.

["Jeff Vincent (JIRA)" <ji...@apache.org>]

Jeff Vincent created SUREFIRE-1336:
--------------------------------------

             Summary: Sure-Fire test run totals don't match TestNG report on failure.
                 Key: SUREFIRE-1336
                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1336
             Project: Maven Surefire
          Issue Type: Bug
          Components: TestNG support
    Affects Versions: 2.19.1
         Environment: Oracle JDK 1.8.0_121, Maven 3.3.9, TestNG 6.10
            Reporter: Jeff Vincent
            Priority: Minor
         Attachments: testng-console.png, testng-reportnghtml.png

One of my TestNG test classes has 2 multi-threaded test methods that use DataProviders to feed in unique parameter values.  In total, I have a test run that runs a total of 530,000 unique test instances between the two test methods.  

In all test output files (XML, HTML, etc.), the test totals are correct (530,000 total, 22 failed, 529,978 passed.)  The first test method runs 10,000 times, the 2nd test method runs 520,000 times.  If all 520,000 pass, it reports 520,000 test cases succeed, but if any one of the iterations fail (or even multiple failures), then it only reports the single test method as a failure which is not accurate.

It seems that it isn't counting failures the same as it counts successes.  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)