[jira] [Commented] (SUREFIRE-1727) Failures during test template creation are ignored

["Tibor Digana (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SUREFIRE-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17024041#comment-17024041 ] 

Tibor Digana commented on SUREFIRE-1727:
----------------------------------------

[~t-8ch]
Is this issue identical with SUREFIRE-1727?

> Failures during test template creation are ignored
> --------------------------------------------------
>
>                 Key: SUREFIRE-1727
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1727
>             Project: Maven Surefire
>          Issue Type: Bug
>          Components: JUnit 5.x support
>    Affects Versions: 3.0.0-M4
>            Reporter: Thomas Weißschuh
>            Priority: Major
>             Fix For: Backlog
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> Junit5 allows dynamic creation of tests via {{ @TestTemplate }} and {{TestTemplateInvocationContextProvider}}.
> If the {{TestTemplateInvocationContextProvider}} fails with an exception itself or Junit itself rejects it then surefire still marks the test as successful.
> Both Intellij IDEA and the junit 5 console launcher mark the test method as failed.
> Examples:
> {code:java}
> package surefirebug;
> import java.util.stream.Stream;
> import org.junit.jupiter.api.extension.ExtensionContext;
> import org.junit.jupiter.api.extension.TestTemplateInvocationContext;
> import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider;
> public class FailingTestTemplateInvocationContextProvider
>     implements TestTemplateInvocationContextProvider {
>   @Override
>   public boolean supportsTestTemplate(ExtensionContext context) {
>     return true;
>   }
>   @Override
>   public Stream<TestTemplateInvocationContext> provideTestTemplateInvocationContexts(
>       ExtensionContext context) {
>     //throw new IllegalStateException(""); // throw a custom exception
>     return Stream.of(); // this will be rejected by Junit itself with a PreconditionViolationException
>   }
> }
> {code}
> {code:java}
> package surefirebug;
> import org.junit.jupiter.api.TestTemplate;
> import org.junit.jupiter.api.extension.ExtendWith;
> @ExtendWith(FailingTestTemplateInvocationContextProvider.class)
> class DemonstrationTest {
>   @TestTemplate
>   void testOne() {
>     System.out.println("testOne");
>   }
> }
> {code}
> Display in surefire:
> {noformat}
> ...
> [INFO] --- maven-surefire-plugin:3.0.0-M4:test (default-test) @ amps-utils ---
> [INFO] 
> [INFO] -------------------------------------------------------
> [INFO]  T E S T S
> [INFO] -------------------------------------------------------
> [INFO] Running surefirebug.DemonstrationTest
> [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 s - in surefirebug.DemonstrationTest
> [INFO] 
> [INFO] Results:
> [INFO] 
> [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> [INFO] 
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> {noformat}
> Display in the console runner:
> {noformat}
> ╷
> ├─ JUnit Jupiter ✔
> │  └─ DemonstrationTest ✔
> │     └─ testOne() ✘ No supporting TestTemplateInvocationContextProvider provided an invocation context
> └─ JUnit Vintage ✔
> Failures (1):
>   JUnit Jupiter:DemonstrationTest:testOne()
>     MethodSource [className = 'surefirebug.DemonstrationTest', methodName = 'testOne', methodParameterTypes = '']
>     => org.junit.platform.commons.PreconditionViolationException: No supporting TestTemplateInvocationContextProvider provided an invocation context
>        org.junit.platform.commons.util.Preconditions.condition(Preconditions.java:296)
>        org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.validateWasAtLeastInvokedOnce(TestTemplateTestDescriptor.java:142)
>        org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:108)
>        org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:41)
>        org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
>        org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>        org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
>        org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
>        org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
>        org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>        [...]
> Test run finished after 40 ms
> [         4 containers found      ]
> [         0 containers skipped    ]
> [         4 containers started    ]
> [         0 containers aborted    ]
> [         3 containers successful ]
> [         1 containers failed     ]
> [         0 tests found           ]
> [         0 tests skipped         ]
> [         0 tests started         ]
> [         0 tests aborted         ]
> [         0 tests successful      ]
> [         0 tests failed          ]
> {noformat}
> I debugged a bit and this is how the error is communicated to surefire by junit:
>  {{RunListenerAdapter.executionFinished()}} receives the following data:
> {noformat}
> testIdentifier.getSource() instanceof MethodSource
> testExecutionResult.getStatus() == failed
> testExecutionResult.getThrowable() instanceof PreconditionViolationException
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)