You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/10/23 00:22:17 UTC
(spark) branch branch-4.3 updated: [SPARK-60461][MLLIB] Restore Strategy constructor binary compatibility
This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new 4a669e86a940 [SPARK-60461][MLLIB] Restore Strategy constructor binary compatibility
4a669e86a940 is described below
commit 4a669e86a940d9d54896ba00dd5c7fb930a05551
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Fri Oct 23 08:21:05 2026 +0800
[SPARK-60461][MLLIB] Restore Strategy constructor binary compatibility
### What changes were proposed in this pull request?
Restore the 15-argument `Strategy` constructor used in Spark 4.2, delegating to the primary constructor with pruning enabled. Keep `bootstrap` in its original parameter position, after `minWeightFractionPerNode`, so its constructor default getter continues to return `false`. Make `pruneTree` `private[spark]`, preserve both `bootstrap` and `pruneTree` in `Strategy.copy`, and remove the corresponding MiMa constructor exclusion.
Please backport to `branch-4.x` and `branch-4.3`.
### Why are the changes needed?
#55763 added `pruneTree` before `bootstrap`, changing the JVM constructor from 15 to 16 parameters. Existing applications compiled against Spark 4.2 fail with `NoSuchMethodError` on the current 4.3 branch. For example, a client compiled against 4.2 with `new Strategy(Algo.Classification, Gini, maxDepth = 3)` fails at construction.
Adding only the old constructor would leave another compatibility problem: the 15th constructor default getter would return the new pruning default (`true`) instead of the old bootstrap default (`false`). Retaining the original parameter positions preserves both linkage and behavior.
`private[spark] var bootstrap` restricts Scala access to the stored property and its accessors. It does not restrict callers from supplying that constructor argument or make the constructor package-private. Spark 4.2's 15-argument constructor was public, so the compatibility constructor must also remain public.
For example, code in a package outside Spark can supply `bootstrap` either by name, using defaults for the other optional arguments, or explicitly as the last of all 15 arguments:
```scala
package example
import org.apache.spark.mllib.tree.configuration.{Algo, QuantileStrategy, Strategy}
import org.apache.spark.mllib.tree.impurity.Gini
object StrategyClient {
val named = new Strategy(Algo.Classification, Gini, maxDepth = 3, bootstrap = true)
val explicit = new Strategy(
Algo.Classification, Gini, 3, 2, 32, QuantileStrategy.Sort,
Map.empty[Int, Int], 1, 0.0, 256, 1.0, false, 10, 0.0, true)
}
```
Both calls compile against Spark 4.2; the final `true` in the second call is the `bootstrap` argument. The compiler fills the omitted defaults in the first call and invokes the same public 15-argument constructor.
Reading or writing the property afterward is still rejected in that external package. These examples intentionally do not compile:
```scala
object InvalidAccess {
val bootstrap = StrategyClient.named.bootstrap
StrategyClient.named.bootstrap = false
}
```
The constructor examples compile against both Spark 4.2 and the patched `Strategy`, while both property-access examples are rejected by the Scala compiler in each version.
### Does this PR introduce _any_ user-facing change?
Yes. Applications compiled against Spark 4.2 can construct `Strategy` on Spark 4.3 again, preserving the default and explicit bootstrap values. Pruning remains enabled by default.
### How was this patch tested?
Added three regression tests in `DecisionTreeSuite` covering the old JVM constructor signature, the bootstrap constructor default getter, and independent pruning/bootstrapping configuration, including copying all four combinations of the two flags. All three passed using the ScalaTest runner with the `Strategy` test-name filter with locally cached Spark dependencies.
Compiled a separate Scala client against the published Spark 4.2.0 MLlib artifact. It reproduces `NoSuchMethodError` with the unpatched 4.3 class and runs successfully with the patched class, preserving the default bootstrap value and both explicit boolean values. The changed class and regression suite were compiled directly with Scala 2.13.18. Full SBT and MiMa runs were not performed.
Audited the `Strategy` constructor/factory call sites, both training copies, ML classifier parameter conversion, and recursive node conversion; no further propagation changes were needed. Also ran the two existing pruning regression tests and three temporary integration tests covering legacy decision-tree/random-forest classification and regression, ML classifiers, and gradient boosting with pruning enabled and disabled. All eight focused tests passed.
Checked whitespace, ASCII, and changed-line lengths. The patch applies cleanly to `branch-4.3`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6)
Closes #59704 from zhengruifeng/ml-strategy-constructor-compatibility.
Authored-by: Ruifeng Zheng <ru...@apache.org>
Signed-off-by: Ruifeng Zheng <ru...@foxmail.com>
(cherry picked from commit fce7d51a3760953097fe34df8aa9a4f5326c74ff)
Signed-off-by: Ruifeng Zheng <ru...@foxmail.com>
---
.../spark/mllib/tree/configuration/Strategy.scala | 35 +++++++++++++++++---
.../spark/mllib/tree/DecisionTreeSuite.scala | 37 ++++++++++++++++++++++
project/MimaExcludes.scala | 2 --
3 files changed, 67 insertions(+), 7 deletions(-)
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/Strategy.scala b/mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/Strategy.scala
index c1f82157e82f..0e8501df4a8f 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/Strategy.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/Strategy.scala
@@ -65,8 +65,6 @@ import org.apache.spark.mllib.tree.impurity.{Entropy, Gini, Impurity, Variance}
* E.g. 10 means that the cache will get checkpointed every 10 updates. If
* the checkpoint directory is not set in
* [[org.apache.spark.SparkContext]], this setting is ignored.
- * @param pruneTree If this is true, the final training tree will undergo a pruning in which
- * nodes with the same prediction are merged.
*/
@Since("1.0.0")
class Strategy @Since("1.3.0") (
@@ -84,8 +82,8 @@ class Strategy @Since("1.3.0") (
@Since("1.2.0") @BeanProperty var useNodeIdCache: Boolean = false,
@Since("1.2.0") @BeanProperty var checkpointInterval: Int = 10,
@Since("3.0.0") @BeanProperty var minWeightFractionPerNode: Double = 0.0,
- @Since("4.3.0") @BeanProperty var pruneTree: Boolean = true,
- @BeanProperty private[spark] var bootstrap: Boolean = false) extends Serializable {
+ @BeanProperty private[spark] var bootstrap: Boolean = false,
+ @BeanProperty private[spark] var pruneTree: Boolean = true) extends Serializable {
/**
*/
@@ -102,6 +100,31 @@ class Strategy @Since("1.3.0") (
}
// scalastyle:off argcount
+ /**
+ * Retains the constructor signature used before the pruning parameter was introduced.
+ */
+ def this(
+ algo: Algo,
+ impurity: Impurity,
+ maxDepth: Int,
+ numClasses: Int,
+ maxBins: Int,
+ quantileCalculationStrategy: QuantileStrategy,
+ categoricalFeaturesInfo: Map[Int, Int],
+ minInstancesPerNode: Int,
+ minInfoGain: Double,
+ maxMemoryInMB: Int,
+ subsamplingRate: Double,
+ useNodeIdCache: Boolean,
+ checkpointInterval: Int,
+ minWeightFractionPerNode: Double,
+ bootstrap: Boolean) = {
+ this(algo, impurity, maxDepth, numClasses, maxBins, quantileCalculationStrategy,
+ categoricalFeaturesInfo, minInstancesPerNode, minInfoGain, maxMemoryInMB,
+ subsamplingRate, useNodeIdCache, checkpointInterval, minWeightFractionPerNode,
+ bootstrap, pruneTree = true)
+ }
+
/**
* Backwards compatible constructor for [[org.apache.spark.mllib.tree.configuration.Strategy]]
*/
@@ -204,7 +227,9 @@ class Strategy @Since("1.3.0") (
new Strategy(algo, impurity, maxDepth, numClasses, maxBins,
quantileCalculationStrategy, categoricalFeaturesInfo, minInstancesPerNode,
minInfoGain, maxMemoryInMB, subsamplingRate, useNodeIdCache,
- checkpointInterval, minWeightFractionPerNode, pruneTree)
+ checkpointInterval, minWeightFractionPerNode,
+ bootstrap = bootstrap,
+ pruneTree = pruneTree)
}
}
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/tree/DecisionTreeSuite.scala b/mllib/src/test/scala/org/apache/spark/mllib/tree/DecisionTreeSuite.scala
index 2f5579acea46..488f7b9e200e 100644
--- a/mllib/src/test/scala/org/apache/spark/mllib/tree/DecisionTreeSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/mllib/tree/DecisionTreeSuite.scala
@@ -25,6 +25,7 @@ import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.tree.configuration.Algo._
import org.apache.spark.mllib.tree.configuration.FeatureType._
+import org.apache.spark.mllib.tree.configuration.QuantileStrategy.Sort
import org.apache.spark.mllib.tree.configuration.Strategy
import org.apache.spark.mllib.tree.impurity.{Entropy, Gini, Variance}
import org.apache.spark.mllib.tree.model._
@@ -35,6 +36,42 @@ import org.apache.spark.util.Utils
class DecisionTreeSuite extends SparkFunSuite with MLlibTestSparkContext {
+ /////////////////////////////////////////////////////////////////////////////
+ // Tests for Strategy configuration
+ /////////////////////////////////////////////////////////////////////////////
+
+ test("Strategy should retain the constructor signature used by Spark 4.2") {
+ assert(classOf[Strategy].getConstructors.exists(_.getParameterCount == 15))
+ for (bootstrap <- Seq(false, true)) {
+ val strategy = new Strategy(Classification, Gini, 3, 2, 32, Sort, Map.empty[Int, Int],
+ 1, 0.0, 256, 1.0, false, 10, 0.0, bootstrap)
+
+ assert(strategy.bootstrap == bootstrap)
+ assert(strategy.pruneTree)
+ }
+ }
+
+ test("Strategy should retain the bootstrap default getter used by compiled callers") {
+ val bootstrapDefault = Strategy.getClass.getMethod("$lessinit$greater$default$15")
+ assert(bootstrapDefault.invoke(Strategy) == java.lang.Boolean.FALSE)
+ }
+
+ test("Strategy should configure pruning independently of bootstrapping") {
+ for {
+ bootstrap <- Seq(false, true)
+ pruneTree <- Seq(false, true)
+ } {
+ val strategy = new Strategy(Classification, Gini, 3,
+ bootstrap = bootstrap, pruneTree = pruneTree)
+ assert(strategy.bootstrap == bootstrap)
+ assert(strategy.pruneTree == pruneTree)
+
+ val copied = strategy.copy
+ assert(copied.bootstrap == bootstrap)
+ assert(copied.pruneTree == pruneTree)
+ }
+ }
+
/////////////////////////////////////////////////////////////////////////////
// Tests calling train()
/////////////////////////////////////////////////////////////////////////////
diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala
index c99a72a1e2a8..37139b23c56f 100644
--- a/project/MimaExcludes.scala
+++ b/project/MimaExcludes.scala
@@ -98,8 +98,6 @@ object MimaExcludes {
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.addTaskInterruptListener"),
// [SPARK-56700][SS] Make DataStreamReader.name public
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.streaming.DataStreamReader.name"),
- // [SPARK-34591][ML] Add pruneTree parameter to Strategy
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.mllib.tree.configuration.Strategy.this"),
// [SPARK-56395][SQL] Add NEAREST BY top-K ranking join
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.Dataset.nearestByJoin"),
// [SPARK-57332][SQL] MySQLDialect no longer overrides the visit methods below; the public
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org