[jira] [Commented] (SUREFIRE-2111) Nested Tests reports are unordered

["Andreas Pabst (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SUREFIRE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17652510#comment-17652510 ] 

Andreas Pabst commented on SUREFIRE-2111:
-----------------------------------------

I've looked into this and noticed that the behaviour is different if you use {{-Dtest=NestedExampleTest*}} and also if you use {{-Dsurefire.excludes=someNonExistingPattern}}. Even {{-Dsurefire.excludes}} does the job. 

If you want to fix it in the surefire configuration, this works:

{code:xml}
<configuration>
    <excludes>
        <exclude />
    </excludes>
</configuration>
{code}

The output is then as follows:

{noformat}
[INFO] Running surefire2032.NestedExampleTest
[INFO] Running surefire2032.NestedExampleTest$InnerTest
[INFO] Running surefire2032.NestedExampleTest$InnerTest$InnerInnerTest
[INFO] Running surefire2032.NestedExampleTest$InnerTest$InnerInnerTest$InnerInnerInnerTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.037 s - in surefire2032.NestedExampleTest$InnerTest$InnerInnerTest$InnerInnerInnerTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.048 s - in surefire2032.NestedExampleTest$InnerTest$InnerInnerTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.055 s - in surefire2032.NestedExampleTest$InnerTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.07 s - in surefire2032.NestedExampleTest
{noformat}

XML reporting is also correctly assigning the test results.

However, it also means that the tests will be run in a depth-first order.

Setting the excludes makes a difference because in the part of the code that decides what class pattern to send to the fork, Surefire excludes inner classes by default, see AbstractSurefireMojo#getDefaultExcludes. In the fork itself they are then re-included, but they are still part of one test set, so the TestSetStats are shared. If inner classes are not excluded, they are treated as separate test sets which is what you want.

> Nested Tests reports are unordered
> ----------------------------------
>
>                 Key: SUREFIRE-2111
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-2111
>             Project: Maven Surefire
>          Issue Type: Bug
>          Components: Maven Surefire Plugin
>            Reporter: Fabrício Yamamoto
>            Priority: Major
>
> When Nested class are run, it's printed unordered.
> See logs:
> {code:java}
> [INFO] -------------------------------------------------------
> [INFO]  T E S T S
> [INFO] -------------------------------------------------------
> [INFO] Running NestedExampleTest
> [INFO] Running NestedExampleTest$InnerTest
> [INFO] Running NestedExampleTest$InnerTest$InnerInnerTest
> [INFO] Running NestedExampleTest$InnerTest$InnerInnerTest$InnerInnerInnerTest
> [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.405 s - in NestedExampleTest$InnerTest$InnerInnerTest$InnerInnerInnerTest
> [INFO] NestedExampleTest.test  Time elapsed: 0.12 s
> [INFO] NestedExampleTest.test2  Time elapsed: 0.001 s
> [INFO] NestedExampleTest$InnerTest.test  Time elapsed: 0.202 s
> [INFO] NestedExampleTest$InnerTest$InnerInnerTest.test  Time elapsed: 0.305 s
> [INFO] NestedExampleTest$InnerTest$InnerInnerTest$InnerInnerInnerTest.test  Time elapsed: 0.402 s
> [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.717 s - in NestedExampleTest$InnerTest$InnerInnerTest
> [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.923 s - in NestedExampleTest$InnerTest
> [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.064 s - in NestedExampleTest
> {code}
> It uses 1 event to run all tests, and 3 remaining events to print class timing.
> Is it possible to run each nest test in a separate event?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)