[jira] [Created] (SUREFIRE-2284) Failures or errors in JUnit @BeforeAll will cause Surefire to throw an exception even if the test succeeds in a rerun.

["Bing Xu (Jira)" <ji...@apache.org>]

Bing Xu created SUREFIRE-2284:
---------------------------------

             Summary: Failures or errors in JUnit @BeforeAll will cause Surefire to throw an exception even if the test succeeds in a rerun.
                 Key: SUREFIRE-2284
                 URL: https://issues.apache.org/jira/browse/SUREFIRE-2284
             Project: Maven Surefire
          Issue Type: Bug
          Components: Maven Surefire Plugin
            Reporter: Bing Xu


Hi again from Atlassian

*Bug description*
If a Test fails (either *fails* (via an assertion or directly calling fail) or throws an {*}error{*}) in the {{@BeforeAll}} method, the interaction with the {{rerunFailingTestsCount}} property doesn't work properly. Even if the test succeeds in a re-run, the Surefire Mojo will throw an exception and fail the build.

*Scenario/Steps to reproduce*
A test fails or throws an exception (error) in the @BeforeAll method on an initial run (Run #1).
And then succeeds in a subsequent execution (Run #2).

*Current behaviour*
The surefire plugin will throw an exception because of the failure/error in the initial run of the test class.

*Expected behaviour:*
The build should pass if a subsequent rerun of the test and no exception should be thrown.

*Root cause*
The root cause seems to be because surefire has logic to merge the result of multiple runs of the same test inside the following class:
{code:java}
maven-surefire-common/src/main/java/org/apache/maven/plugin/surefire/report/DefaultReporterFactory.java{code}
This class does so by matching test method stats with the same testClassMethodName. This does not work when the test fails in @BeforeAll because the test method name is `null`, so it appears to surefire as if this is a separate method altogether that has never succeeded. 

This is the problematic part of the code:

{code:java}
Map<String, List<TestMethodStats>> mergedTestHistoryResult = new HashMap<>();
// Merge all the stats for tests from listeners
for (TestSetRunListener listener : listeners) {
    for (TestMethodStats methodStats : listener.getTestMethodStats()) {
        List<TestMethodStats> currentMethodStats =
                mergedTestHistoryResult.get(methodStats.getTestClassMethodName());
        if (currentMethodStats == null) {
            currentMethodStats = new ArrayList<>();
            currentMethodStats.add(methodStats);
            mergedTestHistoryResult.put(methodStats.getTestClassMethodName(), currentMethodStats);
        } else {
            currentMethodStats.add(methodStats);
        }
    }
} {code}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)