[jira] [Updated] (SUREFIRE-1691) surefire-junit47 reports results against incorrect test

["Tibor Digana (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SUREFIRE-1691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tibor Digana updated SUREFIRE-1691:
-----------------------------------
    Fix Version/s: 3.0

> surefire-junit47 reports results against incorrect test
> -------------------------------------------------------
>
>                 Key: SUREFIRE-1691
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1691
>             Project: Maven Surefire
>          Issue Type: Bug
>          Components: Junit 4.x support
>    Affects Versions: 3.0.0-M3
>            Reporter: M.P. Korstanje
>            Priority: Major
>             Fix For: 3.0
>
>
> This is a follow up on:
>  - https://issues.apache.org/jira/browse/SUREFIRE-1156
>  - [https://github.com/cucumber/cucumber-jvm/issues/865]
> The latest reproducer can be found here:
>  - [https://github.com/mpkorstanje/surefire-test/tree/07d0ee98ea424b38a3c548981a875dde1ef4f2c2]
> In short Cucumber has a hierarchy of tests that looks at follows:
> {noformat}
> Class annotated with @RunWith(Cucumber.class)
>  |- Feature File 1
>  |  |- Scenario 1a
>  |  |- Scenario 1b
>  |- Feature File 2
>  |  |- Scenario 2a
>  |  |- Scenario 2b
> {noformat}
> Surefire tries to group output in tests sets, either per class or per test suite. As Cucumber currently only emits test started/finished events for Scenarios sure fire can not group scenarios into a test set. They are instead grouped with the previous test set. However even after adding test started/finished events for features, sure fire groups all results in a single test set.
> I believe this is caused by an error in {{NonConcurrentRunListener.describesNewTestSet}}.
> {code:java}
>     private boolean describesNewTestSet( Description description )
>     {
>         if ( currentTestSetDescription != null )
>         {
>             if ( null != description.getTestClass() )
>             {
>                 return !description.getTestClass().equals( currentTestSetDescription.getTestClass() );
>             }
>             else if ( description.isSuite() )
>             {
>                 return description.getChildren().equals( currentTestSetDescription.getChildren() );
>             }
>             return false;
>         }
>         return true;
>     }
> {code}
> The value of {{description.getChildren().equals( currentTestSetDescription.getChildren() );}} should be negated so so different test suits are properly recognized as test sets.
> Fixing this locally works and can be reproduced by:
> 1. Negating the line and building sure fire locally
>  2. Building [https://github.com/cucumber/cucumber-jvm/pull/1765] with `mvn clean install -DskipTests` 
>  3. Running `mvn clean test -Pjunit47` on [https://github.com/mpkorstanje/surefire-test]
> It does however does result in another unwanted result: A feature file with two scenarios is now reported as having 3 results. One for each scenario and one for the feature.
> {noformat}
> -------------------------------------------------------------------------------
> Test set: Some Feature 1
> -------------------------------------------------------------------------------
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112 s - in Some_Feature_1
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)