[jira] [Created] (SUREFIRE-1799) Exception thrown in @MethodSource method results in "no tests"

["Jens Reimann (Jira)" <ji...@apache.org>]

Jens Reimann created SUREFIRE-1799:
--------------------------------------

             Summary: Exception thrown in @MethodSource method results in "no tests"
                 Key: SUREFIRE-1799
                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1799
             Project: Maven Surefire
          Issue Type: Bug
          Components: JUnit 5.x support
    Affects Versions: 3.0.0-M4
            Reporter: Jens Reimann


Running the following code:


{code:java}
import java.util.List;

import org.junit.jupiter.api.TestInstance;
import org.junit.jupiter.api.TestInstance.Lifecycle;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.MethodSource;

@TestInstance(Lifecycle.PER_CLASS)
public class MethodSourceErrorTest {

    public List<String> getValues() {
        throw new RuntimeException("Foo"); // fails
        // return Arrays.asList("a"); // works
    }

    @ParameterizedTest
    @MethodSource("getValues")
    public void test1(String value) {
        System.out.println(value);
    }

}
{code}

Results in:

{code}
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running foo.MethodSourceErrorTest
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.018 s - in foo.MethodSourceErrorTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
{code}

My expectation would be that:

 * JUnit reports an error for the tests using the method source
 * The build fails

Additional information:

 * It fails the same way, being a static method

Used versions (Jupiter/Vintage/Platform): 5.6.2, 1.6.2

I raised an issue with JUnit 5 (https://github.com/junit-team/junit5/issues/2319), however they claim that it is an issue with the tool running the test (Maven Surefire in this case).

And indeed the TestExecutionListener receives an exception:

{code:java}
public class MyTestExecutionListener implements TestExecutionListener {

    public MyTestExecutionListener() {
        System.out.println("Here");
    }

    @Override
    public void executionFinished(TestIdentifier testIdentifier, TestExecutionResult testExecutionResult) {
        System.out.format("finished - id: %s, result: %s%n", testIdentifier, testExecutionResult);
    }
}
{code}

Shows the exception as:

{code}
[INFO] --- maven-surefire-plugin:3.0.0-M4:test (default-test) @ bar ---
[WARNING] useSystemClassloader setting has no effect when not forking
Here
[INFO] Running foo.MethodSourceErrorTest
finished - id: TestIdentifier [uniqueId = '[engine:junit-jupiter]/[class:foo.MethodSourceErrorTest]/[test-template:test1(java.lang.String)]', parentId = '[engine:junit-jupiter]/[class:foo.MethodSourceErrorTest]', displayName = 'test1(String)', legacyReportingName = 'test1(String)', source = MethodSource [className = 'foo.MethodSourceErrorTest', methodName = 'test1', methodParameterTypes = 'java.lang.String'], tags = [], type = CONTAINER], result: TestExecutionResult [status = FAILED, throwable = java.lang.RuntimeException: Foo]
finished - id: TestIdentifier [uniqueId = '[engine:junit-jupiter]/[class:foo.MethodSourceErrorTest]', parentId = '[engine:junit-jupiter]', displayName = 'MethodSourceErrorTest', legacyReportingName = 'foo.MethodSourceErrorTest', source = ClassSource [className = 'foo.MethodSourceErrorTest', filePosition = null], tags = [], type = CONTAINER], result: TestExecutionResult [status = SUCCESSFUL, throwable = null]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 s - in foo.MethodSourceErrorTest
finished - id: TestIdentifier [uniqueId = '[engine:junit-jupiter]', parentId = null, displayName = 'JUnit Jupiter', legacyReportingName = 'JUnit Jupiter', source = null, tags = [], type = CONTAINER], result: TestExecutionResult [status = SUCCESSFUL, throwable = null]
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)