[jira] [Comment Edited] (SUREFIRE-1776) DefaultReportFactory doesn't report errors when engine fails

["Tibor Digana (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SUREFIRE-1776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085901#comment-17085901 ] 

Tibor Digana edited comment on SUREFIRE-1776 at 4/17/20, 4:09 PM:
------------------------------------------------------------------

The code in 3.0 is different. I have to switch back to the old version 2.22.0.
Anyway, i sent an email to the Tycho developers in the official mailing list with the request to change this. I think the Tycho builds another plugin on the top of Surefire. And my idea is to stop this because it would be finally easier for both and produce a provider for the Tycho. Technically it is not so straightforward because Tycho has some prerequisites with dependencies resolver. If we find out what changes are done to Surefire by Tycho developers, we would better find the way on how to provide the same behavior without changing our source code of Surefire in Tycho.


was (Author: tibor17):
The code in 3.0 is different. I have to switch back to the old version 2.22.0.
Anyway, i sent an email to the Tycho developers in the official mailing list with the request to change this. I think the Tycho builds another plugin on the top of Surefire. And my idea is to stop this because it would be finally easier for both and produce a provider for the Tycho. Technically it is not so straightforward because Tycho some prerequisites with dependencies resolver. If we find out what changes are done to Surefire by Tycho developers, we would better find the way on how to provide the same behavior without changing our source code in Surefire.

> DefaultReportFactory doesn't report errors when engine fails
> ------------------------------------------------------------
>
>                 Key: SUREFIRE-1776
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1776
>             Project: Maven Surefire
>          Issue Type: Bug
>          Components: Maven Surefire Plugin
>    Affects Versions: 2.22.0
>            Reporter: Mickael Istria
>            Priority: Major
>
> We face an issue with tycho-surefire-plugin reporting "No tests found" when a test is actually found but the engine fails (for a tycho-specific reason) when preparing the test.
> tycho-surefire-plugin basically builds an OSGi container as defined in the usual Tycho way and uses Surefire API to run the tests inside that container and report it as a plain surefire does regularly.
> To do so, it does mainly uses `RunResult result = ProviderFactory.invokeProvider(...)`. However, it can happen that an error happen in the engine, before any test method starts (although the test method is scanned and detected and an executionRequest is emitted for it). The cases for it are relatively probable with Tycho/OSGi.
>  In such case, even if the engine faced an error, we get a RunResult mentioning no error, actually reporting no test execution, but no error, usually making the test report say everything is fine or telling "No tests found" (which is also incorrect because test is found, it could not just be started).
> I could dig a bit more and I think the DefaultTestReporter.mergeTestHistoryResult() method just fails at handling the cases of errors when the error do not happen on the test method, because it wrongly assumes that `listener.getTestMethodStats()` is exhaustive.
>  More specifically, when debugging this method, I do see in debugger that `this.listeners[0].detailsForThis` has `completedCount==1` and `errors==1` but the return value says completedCount==0 and errors=0; its reportEntries show one error entry, which has for source `<unrooted>` so they're not returned by `listener.getTestMethodStats()`.
> I believe the DefaultReportFactory should handle the case to verify before returning whether such unrooted error wasn't missed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)