You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@maven.apache.org by ol...@apache.org on 2026/10/14 01:04:22 UTC

(maven-surefire) 02/03: Document the runOrder changes in 3.6.x (#3510)

This is an automated email from the ASF dual-hosted git repository.

olamy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/maven-surefire.git

commit eccd1535c9078cecd07944a6d667b81b954f7623
Author: Olivier Lamy <ol...@apache.org>
AuthorDate: Sun Oct 11 21:56:22 2026 +1000

    Document the runOrder changes in 3.6.x (#3510)
    
    Signed-off-by: Olivier Lamy <ol...@apache.org>
---
 .../maven/plugin/failsafe/IntegrationTestMojo.java | 10 ++++++
 .../apache/maven/plugin/surefire/SurefireMojo.java | 10 ++++++
 .../fork-options-and-parallel-execution.md.vm      |  2 +-
 .../src/site/markdown/index.md.vm                  |  2 +-
 .../{whats-new-3-6-0.md => whats-new-3-6-x.md}     | 42 +++++++++++++++++++++-
 maven-surefire-plugin/src/site/site.xml            |  2 +-
 src/site/markdown/index.md                         |  4 +--
 7 files changed, 66 insertions(+), 6 deletions(-)

diff --git a/maven-failsafe-plugin/src/main/java/org/apache/maven/plugin/failsafe/IntegrationTestMojo.java b/maven-failsafe-plugin/src/main/java/org/apache/maven/plugin/failsafe/IntegrationTestMojo.java
index bc41aaeee..ca0810487 100644
--- a/maven-failsafe-plugin/src/main/java/org/apache/maven/plugin/failsafe/IntegrationTestMojo.java
+++ b/maven-failsafe-plugin/src/main/java/org/apache/maven/plugin/failsafe/IntegrationTestMojo.java
@@ -354,6 +354,16 @@ public class IntegrationTestMojo extends AbstractSurefireMojo {
      * should not be checked into version control. The "XXXXX" is the SHA1 checksum of the entire surefire
      * configuration, so different configurations will have different statistics files, meaning if you change any
      * configuration settings you will re-run once before new statistics data can be established.
+     * <br>
+     * <br>
+     * Since 3.6.0, {@code alphabetical}, {@code reversealphabetical} and {@code random} also sort the test methods
+     * inside each class for JUnit 5/6 tests. A class with {@code @TestMethodOrder} keeps its own order. To keep the
+     * JUnit default method order, add an empty {@code junit.jupiter.testmethod.order.default=} to
+     * {@code configurationParameters} in {@code properties}. It has no effect in {@code junit-platform.properties}
+     * or as a system property.
+     * <br>
+     * <br>
+     * {@code hourly} was removed in 3.6.0. Use {@code random} instead.
      *
      * @since 2.7
      */
diff --git a/maven-surefire-plugin/src/main/java/org/apache/maven/plugin/surefire/SurefireMojo.java b/maven-surefire-plugin/src/main/java/org/apache/maven/plugin/surefire/SurefireMojo.java
index 8e75d27d0..fb8abc2d2 100644
--- a/maven-surefire-plugin/src/main/java/org/apache/maven/plugin/surefire/SurefireMojo.java
+++ b/maven-surefire-plugin/src/main/java/org/apache/maven/plugin/surefire/SurefireMojo.java
@@ -318,6 +318,16 @@ public class SurefireMojo extends AbstractSurefireMojo implements SurefireReport
      * should not be checked into version control. The "XXXXX" is the SHA1 checksum of the entire surefire
      * configuration, so different configurations will have different statistics files, meaning if you change any
      * configuration settings you will re-run once before new statistics data can be established.
+     * <br>
+     * <br>
+     * Since 3.6.0, {@code alphabetical}, {@code reversealphabetical} and {@code random} also sort the test methods
+     * inside each class for JUnit 5/6 tests. A class with {@code @TestMethodOrder} keeps its own order. To keep the
+     * JUnit default method order, add an empty {@code junit.jupiter.testmethod.order.default=} to
+     * {@code configurationParameters} in {@code properties}. It has no effect in {@code junit-platform.properties}
+     * or as a system property.
+     * <br>
+     * <br>
+     * {@code hourly} was removed in 3.6.0. Use {@code random} instead.
      *
      * @since 2.7
      */
diff --git a/maven-surefire-plugin/src/site/markdown/examples/fork-options-and-parallel-execution.md.vm b/maven-surefire-plugin/src/site/markdown/examples/fork-options-and-parallel-execution.md.vm
index 580b92cc6..01667b4e6 100644
--- a/maven-surefire-plugin/src/site/markdown/examples/fork-options-and-parallel-execution.md.vm
+++ b/maven-surefire-plugin/src/site/markdown/examples/fork-options-and-parallel-execution.md.vm
@@ -48,7 +48,7 @@ Basically, there are two ways in ${project.artifactId} to achieve parallel test
 Everything else is ignored: the suite values of `parallel` (`suites`, `suitesAndClasses`, `suitesAndMethods`, `all`) and
 the deprecated `perCoreThreadCount`, `useUnlimitedThreads`, `threadCountSuites`, `parallelOptimized`,
 `parallelTestsTimeoutInSeconds` and `parallelTestsTimeoutForcedInSeconds`. The rest of this section describes how
-the JUnit 4.7 provider of 3.5.x used them. See [What's New in 3.6.0](../whats-new-3-6-0.html#Parameter_Changes)
+the JUnit 4.7 provider of 3.5.x used them. See [What's New in 3.6.x](../whats-new-3-6-x.html#Parameter_Changes)
 for details.
 
 The most obvious one is by using the `parallel` parameter. The possible values depend on the test provider used. For JUnit 4.7 and onwards, this may be `methods`, `classes`, `both`, `suites`, `suitesAndClasses`, `suitesAndMethods`, `classesAndMethods` or `all`. As a prerequisite in JUnit tests, the JUnit runner should extend `org.junit.runners.ParentRunner`. If no runner is specified through the annotation `@org.junit.runner.RunWith`, the prerequisite is accomplished.
diff --git a/maven-surefire-plugin/src/site/markdown/index.md.vm b/maven-surefire-plugin/src/site/markdown/index.md.vm
index f7143f0e8..7c77e5028 100644
--- a/maven-surefire-plugin/src/site/markdown/index.md.vm
+++ b/maven-surefire-plugin/src/site/markdown/index.md.vm
@@ -112,4 +112,4 @@ The following examples show how to use the ${thisPlugin} Plugin in more advanced
 - [Using Console Logs](./examples/logging.html)
 - [Shutdown of Forked JVM](./examples/shutdown.html)
 - [Run tests with Java 9](./java9.html)
-- [What's New in 3.6.0](./whats-new-3-6-0.html)
+- [What's New in 3.6.x](./whats-new-3-6-x.html)
diff --git a/maven-surefire-plugin/src/site/markdown/whats-new-3-6-0.md b/maven-surefire-plugin/src/site/markdown/whats-new-3-6-x.md
similarity index 82%
rename from maven-surefire-plugin/src/site/markdown/whats-new-3-6-0.md
rename to maven-surefire-plugin/src/site/markdown/whats-new-3-6-x.md
index 994796f11..42a19bb50 100644
--- a/maven-surefire-plugin/src/site/markdown/whats-new-3-6-0.md
+++ b/maven-surefire-plugin/src/site/markdown/whats-new-3-6-x.md
@@ -17,7 +17,7 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-# What's New in Surefire 3.6.0
+# What's New in Surefire 3.6.x
 
 ## Overview
 
@@ -95,6 +95,7 @@ These apply to both `surefire:test` and `failsafe:integration-test` unless noted
 | `parallelTestsTimeoutInSeconds`, `parallelTestsTimeoutForcedInSeconds` | Deprecated and ignored. | Use `forkedProcessTimeoutInSeconds`. |
 | `objectFactory` | Deprecated and ignored. The TestNG engine has no such option. | Remove it. |
 | `parallel` | Only some values still work, see below. | |
+| `runOrder` | Now also sorts JUnit 5/6 test methods. `hourly` is gone. | See [Run Order](#Run_Order) below. |
 | `encoding` (`failsafe:verify` only) | Removed in 3.5.5. It had been deprecated since 2.20.1 and did nothing. | Remove it. |
 
 Since 3.6.1 you get a warning when one of the ignored parameters is set. A legacy provider added as a plugin
@@ -110,6 +111,45 @@ Which `parallel` values still work:
 - **JUnit 5/6**: `parallel` isn't used. Use the Jupiter `junit.jupiter.execution.parallel.*` settings instead,
   see [Using JUnit 5 Platform](examples/junit-platform.html).
 
+### Run Order
+
+`alphabetical`, `reversealphabetical` and `random` now sort the test methods inside each class too, not only the
+classes. This only applies to JUnit 5/6 tests. JUnit 4 and TestNG methods run in the same order as before.
+
+| `runOrder` | Methods run |
+|------------|-------------|
+| `alphabetical` | by name |
+| `reversealphabetical` | by name, reversed |
+| `random` | in random order, using `runOrderRandomSeed` |
+
+A test that only passed because its methods ran in a certain order can now fail. This often happens with
+`@TestInstance(PER_CLASS)` classes that share state between methods.
+
+To keep the JUnit default method order, either:
+
+- add `@TestMethodOrder` to the class. Surefire never overrides it. If your methods depend on each other, this is
+  the best fix: use `@TestMethodOrder(MethodOrderer.OrderAnnotation.class)` and put `@Order` on each method.
+- or clear `junit.jupiter.testmethod.order.default`, the parameter Surefire sets, in `configurationParameters`.
+  Classes are still sorted by `runOrder`.
+
+```xml
+<configuration>
+    <runOrder>alphabetical</runOrder>
+    <properties>
+        <configurationParameters>
+            junit.jupiter.testmethod.order.default=
+        </configurationParameters>
+    </properties>
+</configuration>
+```
+
+With JUnit 6 you can also set it to `org.junit.jupiter.api.MethodOrderer$Default`. Setting it in
+`junit-platform.properties` or as a system property doesn't work, the value from Surefire wins.
+
+`hourly` was removed, and the build now fails with `There's no RunOrder with the name hourly`. It ran the tests
+alphabetically during even hours and in reverse during odd hours. Use `random` instead: the seed is printed in the
+build log, so you can replay a failing order with `runOrderRandomSeed`.
+
 ## Migration Guide
 
 ### JUnit 5/6 Users
diff --git a/maven-surefire-plugin/src/site/site.xml b/maven-surefire-plugin/src/site/site.xml
index 186151709..4c44032a3 100644
--- a/maven-surefire-plugin/src/site/site.xml
+++ b/maven-surefire-plugin/src/site/site.xml
@@ -28,7 +28,7 @@
   <body>
     <menu name="Overview">
       <item name="Introduction" href="index.html"/>
-      <item name="What's New in 3.6.0" href="whats-new-3-6-0.html"/>
+      <item name="What's New in 3.6.x" href="whats-new-3-6-x.html"/>
       <item name="Plugin Documentation" href="plugin-info.html"/>
       <item name="Usage" href="usage.html"/>
       <item name="API" href="api.html"/>
diff --git a/src/site/markdown/index.md b/src/site/markdown/index.md
index ad69459fe..ca17cfac7 100644
--- a/src/site/markdown/index.md
+++ b/src/site/markdown/index.md
@@ -52,13 +52,13 @@ for the full list of resolved issues.
 ### 3.6.0-M1
 
 The big one: every test framework now runs through a single, unified
-[JUnit Platform provider](maven-surefire-plugin/whats-new-3-6-0.html). JUnit 5 runs natively,
+[JUnit Platform provider](maven-surefire-plugin/whats-new-3-6-x.html). JUnit 5 runs natively,
 and JUnit 4 and TestNG run through it too — giving you more consistent behaviour and far less
 to configure. JUnit 5 tests can also run in parallel, and the JUnit 5 XML reports are tidier.
 
 If you're upgrading, note that 3.6.0-M1 raises the minimum versions (JUnit 4.12+, TestNG
 6.14.3+) and drops the old per-framework providers. The
-[What's New in 3.6.0](maven-surefire-plugin/whats-new-3-6-0.html) guide walks you through it,
+[What's New in 3.6.x](maven-surefire-plugin/whats-new-3-6-x.html) guide walks you through it,
 and the [3.6.0-M1 release notes](https://github.com/apache/maven-surefire/releases/tag/surefire-3.6.0-M1)
 list everything.