You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/10/20 00:28:31 UTC
(spark) branch master updated: [SPARK-60330][ML] Register classification tree nodes with Kryo
This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new ee904684e78c [SPARK-60330][ML] Register classification tree nodes with Kryo
ee904684e78c is described below
commit ee904684e78c7722b223ef27004e2c49a20d4ffa
Author: luka-erceg-data-oss <lu...@databricks.com>
AuthorDate: Tue Oct 20 08:28:11 2026 +0800
[SPARK-60330][ML] Register classification tree nodes with Kryo
### What changes were proposed in this pull request?
Add built-in Kryo registrations for classification tree node arrays, internal and leaf nodes, continuous and categorical splits, and Gini and entropy impurity calculators.
Add `NodeKryoSuite` to round-trip the complete node graphs with strict registration enabled. The registrations use the existing optional-class loading mechanism, so Spark Core does not acquire a dependency on MLlib.
### Why are the changes needed?
These built-in Spark types are missing from the registration list. Serializing an `Array[Node]` with `spark.kryo.registrationRequired=true` fails unless users register the graph's classes themselves. Without strict registration, Kryo writes class-name metadata for the unregistered types.
### Does this PR introduce _any_ user-facing change?
Yes. Classification tree node graphs can be serialized with Spark's Kryo serializer and strict registration enabled without manually registering these built-in classes. No new API or configuration is introduced.
### How was this patch tested?
Added `NodeKryoSuite`, covering both classification impurity types, continuous split boundaries, and categorical sets of different sizes. It checks predictions, leaf indices, and class-count statistics after deserialization.
Passed `NodeKryoSuite` on Java 17: 2 tests (Gini and entropy), no failures. Static whitespace, added-line length, and ASCII checks also pass.
```shell
build/sbt 'mllib/testOnly org.apache.spark.ml.tree.NodeKryoSuite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex CLI 0.154.0 (GPT-5.2)
Closes #59539 from luka-erceg-data-oss/luka-erceg_data/kryo-tree-node-registration-oss.
Authored-by: luka-erceg-data-oss <lu...@databricks.com>
Signed-off-by: Ruifeng Zheng <ru...@apache.org>
---
.../apache/spark/serializer/KryoSerializer.scala | 9 +++-
.../org/apache/spark/ml/tree/NodeKryoSuite.scala | 58 ++++++++++++++++++++++
2 files changed, 66 insertions(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala b/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
index 9b120d2f5bdb..ace1c385343c 100644
--- a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
+++ b/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
@@ -644,6 +644,11 @@ private[serializer] object KryoSerializer {
"org.apache.spark.ml.linalg.SparseVector",
"org.apache.spark.ml.linalg.Vector",
"org.apache.spark.ml.stat.distribution.MultivariateGaussian",
+ "[Lorg.apache.spark.ml.tree.Node;",
+ "org.apache.spark.ml.tree.InternalNode",
+ "org.apache.spark.ml.tree.LeafNode",
+ "org.apache.spark.ml.tree.ContinuousSplit",
+ "org.apache.spark.ml.tree.CategoricalSplit",
"org.apache.spark.ml.tree.impl.TreePoint",
"org.apache.spark.mllib.clustering.VectorWithNorm",
"org.apache.spark.mllib.linalg.DenseMatrix",
@@ -653,7 +658,9 @@ private[serializer] object KryoSerializer {
"org.apache.spark.mllib.linalg.SparseVector",
"org.apache.spark.mllib.linalg.Vector",
"org.apache.spark.mllib.regression.LabeledPoint",
- "org.apache.spark.mllib.stat.distribution.MultivariateGaussian"
+ "org.apache.spark.mllib.stat.distribution.MultivariateGaussian",
+ "org.apache.spark.mllib.tree.impurity.GiniCalculator",
+ "org.apache.spark.mllib.tree.impurity.EntropyCalculator"
).flatMap { name =>
try {
Some[Class[_]](Utils.classForName(name))
diff --git a/mllib/src/test/scala/org/apache/spark/ml/tree/NodeKryoSuite.scala b/mllib/src/test/scala/org/apache/spark/ml/tree/NodeKryoSuite.scala
new file mode 100644
index 000000000000..869d9e838f51
--- /dev/null
+++ b/mllib/src/test/scala/org/apache/spark/ml/tree/NodeKryoSuite.scala
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.tree
+
+import org.apache.spark.{SparkConf, SparkFunSuite}
+import org.apache.spark.internal.config.Kryo.KRYO_REGISTRATION_REQUIRED
+import org.apache.spark.ml.linalg.Vectors
+import org.apache.spark.mllib.tree.impurity.ImpurityCalculator
+import org.apache.spark.serializer.KryoSerializer
+
+class NodeKryoSuite extends SparkFunSuite {
+
+ gridTest("tree nodes round-trip with strict Kryo registration")(Seq("gini", "entropy")) {
+ impurity =>
+ val conf = new SparkConf(false).set(KRYO_REGISTRATION_REQUIRED, true)
+ val serializer = new KryoSerializer(conf).newInstance()
+ val splits: Seq[Split] = Seq(new ContinuousSplit(0, 0.5)) ++
+ Seq(0, 1, 2, 3, 4, 5, 6, 7, 12).map { size =>
+ new CategoricalSplit(1, Array.tabulate(size)(_.toDouble), 12)
+ }
+ val roots: Array[Node] = splits.map { split =>
+ val leftStats = ImpurityCalculator.getCalculator(impurity, Array(3.0, 0.0, 0.0), 3L)
+ val rightStats = ImpurityCalculator.getCalculator(impurity, Array(0.0, 0.0, 3.0), 3L)
+ val parentStats = ImpurityCalculator.getCalculator(impurity, Array(3.0, 0.0, 3.0), 6L)
+ new InternalNode(0.0, parentStats.calculate(), 1.0,
+ new LeafNode(0.0, 0.0, leftStats, 0), new LeafNode(2.0, 0.0, rightStats, 1),
+ split, parentStats)
+ }.toArray
+ val features = for (continuous <- Seq(0.0, 0.5, 1.0); category <- 0 until 12) yield {
+ Vectors.dense(continuous, category.toDouble)
+ }
+
+ val decoded = serializer.deserialize[Array[Node]](serializer.serialize(roots))
+ assert(decoded.length == roots.length)
+ for ((original, restored) <- roots.zip(decoded); vector <- features) {
+ val originalLeaf = original.predictImpl(vector)
+ val restoredLeaf = restored.predictImpl(vector)
+ assert(originalLeaf.prediction == restoredLeaf.prediction)
+ assert(originalLeaf.leafIndex == restoredLeaf.leafIndex)
+ assert(originalLeaf.impurityStats.stats.sameElements(restoredLeaf.impurityStats.stats))
+ }
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org