[jira] [Comment Edited] (SUREFIRE-1691) surefire-junit47 reports results against incorrect test

["Tibor Digana (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SUREFIRE-1691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16922338#comment-16922338 ] 

Tibor Digana edited comment on SUREFIRE-1691 at 1/26/20 11:53 PM:
------------------------------------------------------------------

[~mpkorstanje]
A fix is not worth the spent time because we want to destroy the ThreadLocal-based solution and we want the StatelessXmlReporter been able to resolve parallel events received from the tests. This solution in reporter is needed in JUni5 because the tests with JUnit5 provider have problem to separate parallel logs. This way we would solve two bugs by one fix. The fix was planned for M4 but honestly it is so difficult that we have to postpone it to M5 or later.


was (Author: tibor17):
[~mpkorstanje]
A fix is not worth the spent time because we want to destroy the ThreadLocal-based solution and we want the StatelessXmlReporter been able to resolve parallel events received from the tests. This solution in reporter is needed in JUni5 because the tests with JUnit5 provider have problem to separate parallel logs. This way we would solve two bugs by one fix. The fix was planned for M4 but honestly it is so difficult that we have to postpone it to M5.

> surefire-junit47 reports results against incorrect test
> -------------------------------------------------------
>
>                 Key: SUREFIRE-1691
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1691
>             Project: Maven Surefire
>          Issue Type: Bug
>          Components: Junit 4.x support
>    Affects Versions: 3.0.0-M3
>            Reporter: M.P. Korstanje
>            Priority: Major
>
> This is a follow up on:
>  - https://issues.apache.org/jira/browse/SUREFIRE-1156
>  - [https://github.com/cucumber/cucumber-jvm/issues/865]
> The latest reproducer can be found here:
>  - [https://github.com/mpkorstanje/surefire-test/tree/07d0ee98ea424b38a3c548981a875dde1ef4f2c2]
> In short Cucumber has a hierarchy of tests that looks at follows:
> {noformat}
> Class annotated with @RunWith(Cucumber.class)
>  |- Feature File 1
>  |  |- Scenario 1a
>  |  |- Scenario 1b
>  |- Feature File 2
>  |  |- Scenario 2a
>  |  |- Scenario 2b
> {noformat}
> Surefire tries to group output in tests sets, either per class or per test suite. As Cucumber currently only emits test started/finished events for Scenarios sure fire can not group scenarios into a test set. They are instead grouped with the previous test set. However even after adding test started/finished events for features, sure fire groups all results in a single test set.
> I believe this is caused by an error in {{NonConcurrentRunListener.describesNewTestSet}}.
> {code:java}
>     private boolean describesNewTestSet( Description description )
>     {
>         if ( currentTestSetDescription != null )
>         {
>             if ( null != description.getTestClass() )
>             {
>                 return !description.getTestClass().equals( currentTestSetDescription.getTestClass() );
>             }
>             else if ( description.isSuite() )
>             {
>                 return description.getChildren().equals( currentTestSetDescription.getChildren() );
>             }
>             return false;
>         }
>         return true;
>     }
> {code}
> The value of {{description.getChildren().equals( currentTestSetDescription.getChildren() );}} should be negated so so different test suits are properly recognized as test sets.
> Fixing this locally works and can be reproduced by:
> 1. Negating the line and building sure fire locally
>  2. Building [https://github.com/cucumber/cucumber-jvm/pull/1765] with `mvn clean install -DskipTests` 
>  3. Running `mvn clean test -Pjunit47` on [https://github.com/mpkorstanje/surefire-test]
> It does however does result in another unwanted result: A feature file with two scenarios is now reported as having 3 results. One for each scenario and one for the feature.
> {noformat}
> -------------------------------------------------------------------------------
> Test set: Some Feature 1
> -------------------------------------------------------------------------------
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112 s - in Some_Feature_1
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)