You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2025/01/16 07:13:56 UTC
[PR] [SPARK-50825][ML][PYTHON][CONNECT] Support Tree Classifiers in ML Connect [spark]
zhengruifeng opened a new pull request, #49525:
URL: https://github.com/apache/spark/pull/49525
### What changes were proposed in this pull request?
Support Tree Classifiers in ML Connect
### Why are the changes needed?
for parity with spark clasic
### Does this PR introduce _any_ user-facing change?
yes, new algorithms supported on ml
### How was this patch tested?
added tests
### Was this patch authored or co-authored using generative AI tooling?
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50825][ML][PYTHON][CONNECT] Support Tree Classifiers in ML Connect [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #49525:
URL: https://github.com/apache/spark/pull/49525#discussion_r1917860839
##########
python/pyspark/ml/tests/test_classification.py:
##########
@@ -283,6 +293,318 @@ def test_logistic_regression(self):
except OSError:
pass
+ def test_decision_tree_classifier(self):
+ df = (
+ self.spark.createDataFrame(
+ [
+ (1.0, 1.0, Vectors.dense(0.0, 5.0)),
+ (0.0, 2.0, Vectors.dense(1.0, 2.0)),
+ (1.0, 3.0, Vectors.dense(2.0, 1.0)),
+ (0.0, 4.0, Vectors.dense(3.0, 3.0)),
+ ],
+ ["label", "weight", "features"],
+ )
+ .coalesce(1)
+ .sortWithinPartitions("weight")
Review Comment:
`spark.createDataFrame` output different partitioning between connect and classic,
which slightly affect the training and model coefficients.
Need `coalesce.sortWithinPartitions` to make training deterministic, otherwise, the parity test would fail
##########
python/pyspark/ml/tests/test_classification.py:
##########
@@ -283,6 +293,318 @@ def test_logistic_regression(self):
except OSError:
pass
+ def test_decision_tree_classifier(self):
+ df = (
+ self.spark.createDataFrame(
+ [
+ (1.0, 1.0, Vectors.dense(0.0, 5.0)),
+ (0.0, 2.0, Vectors.dense(1.0, 2.0)),
+ (1.0, 3.0, Vectors.dense(2.0, 1.0)),
+ (0.0, 4.0, Vectors.dense(3.0, 3.0)),
+ ],
+ ["label", "weight", "features"],
+ )
+ .coalesce(1)
+ .sortWithinPartitions("weight")
Review Comment:
`spark.createDataFrame` outputs different partitioning between connect and classic,
which slightly affect the training and model coefficients.
Need `coalesce.sortWithinPartitions` to make training deterministic, otherwise, the parity test would fail
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50825][ML][PYTHON][CONNECT] Support Tree Classifiers in ML Connect [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #49525: [SPARK-50825][ML][PYTHON][CONNECT] Support Tree Classifiers in ML Connect
URL: https://github.com/apache/spark/pull/49525
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50825][ML][PYTHON][CONNECT] Support Tree Classifiers in ML Connect [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #49525:
URL: https://github.com/apache/spark/pull/49525#discussion_r1917861976
##########
python/pyspark/ml/tests/test_classification.py:
##########
@@ -283,6 +293,318 @@ def test_logistic_regression(self):
except OSError:
pass
+ def test_decision_tree_classifier(self):
+ df = (
+ self.spark.createDataFrame(
+ [
+ (1.0, 1.0, Vectors.dense(0.0, 5.0)),
+ (0.0, 2.0, Vectors.dense(1.0, 2.0)),
+ (1.0, 3.0, Vectors.dense(2.0, 1.0)),
+ (0.0, 4.0, Vectors.dense(3.0, 3.0)),
+ ],
+ ["label", "weight", "features"],
+ )
+ .coalesce(1)
+ .sortWithinPartitions("weight")
+ )
+
+ dt = DecisionTreeClassifier(
+ maxDepth=2,
+ labelCol="label",
+ leafCol="leaf",
+ seed=1,
+ )
+ self.assertEqual(dt.getMaxDepth(), 2)
+ self.assertEqual(dt.getSeed(), 1)
+ self.assertEqual(dt.getLabelCol(), "label")
+ self.assertEqual(dt.getLeafCol(), "leaf")
+
+ # Estimator save & load
+ with tempfile.TemporaryDirectory(prefix="decision_tree_classifier") as d:
+ dt.write().overwrite().save(d)
+ dt2 = DecisionTreeClassifier.load(d)
+ self.assertEqual(str(dt), str(dt2))
+
+ model = dt.fit(df)
+ self.assertEqual(model.numClasses, 2)
+ self.assertEqual(model.numFeatures, 2)
+ self.assertEqual(model.depth, 2)
+ self.assertEqual(model.numNodes, 5)
+ self.assertEqual(model.featureImportances, Vectors.sparse(2, [0], [1.0]))
+ self.assertTrue("depth=2, numNodes=5, numClasses=2, numFeatures=2" in model.toDebugString)
+ self.assertTrue("If (feature 0 <= 2.5)" in model.toDebugString)
+
+ vec = Vectors.dense(0.0, 5.0)
+ self.assertEqual(model.predict(vec), 1.0)
+ self.assertEqual(model.predictRaw(vec), Vectors.dense(0.0, 1.0))
+ self.assertEqual(model.predictProbability(vec), Vectors.dense(0.0, 1.0))
+ self.assertEqual(model.predictLeaf(vec), 0.0)
+
+ output = model.transform(df)
+ expected_cols = [
+ "label",
+ "weight",
+ "features",
+ "rawPrediction",
+ "probability",
+ "prediction",
+ "leaf",
+ ]
+ self.assertEqual(output.columns, expected_cols)
+ self.assertEqual(output.count(), 4)
+
+ # Model save & load
+ with tempfile.TemporaryDirectory(prefix="decision_tree_model") as d:
+ model.write().overwrite().save(d)
+ model2 = DecisionTreeClassificationModel.load(d)
+ self.assertEqual(str(model), str(model2))
+ self.assertEqual(model.toDebugString, model2.toDebugString)
+
+ def test_gbt_classifier(self):
+ df = (
+ self.spark.createDataFrame(
+ [
+ (1.0, 1.0, Vectors.dense(0.0, 5.0)),
+ (0.0, 2.0, Vectors.dense(1.0, 2.0)),
+ (1.0, 3.0, Vectors.dense(2.0, 1.0)),
+ (0.0, 4.0, Vectors.dense(3.0, 3.0)),
+ ],
+ ["label", "weight", "features"],
+ )
+ .coalesce(1)
+ .sortWithinPartitions("weight")
+ )
+
+ gbt = GBTClassifier(
+ maxIter=3,
+ maxDepth=2,
+ labelCol="label",
+ leafCol="leaf",
+ seed=1,
+ )
+ self.assertEqual(gbt.getMaxIter(), 3)
+ self.assertEqual(gbt.getMaxDepth(), 2)
+ self.assertEqual(gbt.getSeed(), 1)
+ self.assertEqual(gbt.getLabelCol(), "label")
+ self.assertEqual(gbt.getLeafCol(), "leaf")
+
+ # Estimator save & load
+ with tempfile.TemporaryDirectory(prefix="gbt_classifier") as d:
+ gbt.write().overwrite().save(d)
+ gbt2 = GBTClassifier.load(d)
+ self.assertEqual(str(gbt), str(gbt2))
+
+ model = gbt.fit(df)
+ self.assertEqual(model.numClasses, 2)
+ self.assertEqual(model.numFeatures, 2)
+ # TODO(SPARK-50843): Support access submodel in TreeEnsembleModel
+ # model.trees
Review Comment:
`GBTClassificationModel.trees` not supported, file SPARK-50843 to track it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50825][ML][PYTHON][CONNECT] Support Tree Classifiers in ML Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on PR #49525:
URL: https://github.com/apache/spark/pull/49525#issuecomment-2594812892
LGTM. Thx @zhengruifeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50825][ML][PYTHON][CONNECT] Support Tree Classifiers in ML Connect [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #49525:
URL: https://github.com/apache/spark/pull/49525#discussion_r1917861976
##########
python/pyspark/ml/tests/test_classification.py:
##########
@@ -283,6 +293,318 @@ def test_logistic_regression(self):
except OSError:
pass
+ def test_decision_tree_classifier(self):
+ df = (
+ self.spark.createDataFrame(
+ [
+ (1.0, 1.0, Vectors.dense(0.0, 5.0)),
+ (0.0, 2.0, Vectors.dense(1.0, 2.0)),
+ (1.0, 3.0, Vectors.dense(2.0, 1.0)),
+ (0.0, 4.0, Vectors.dense(3.0, 3.0)),
+ ],
+ ["label", "weight", "features"],
+ )
+ .coalesce(1)
+ .sortWithinPartitions("weight")
+ )
+
+ dt = DecisionTreeClassifier(
+ maxDepth=2,
+ labelCol="label",
+ leafCol="leaf",
+ seed=1,
+ )
+ self.assertEqual(dt.getMaxDepth(), 2)
+ self.assertEqual(dt.getSeed(), 1)
+ self.assertEqual(dt.getLabelCol(), "label")
+ self.assertEqual(dt.getLeafCol(), "leaf")
+
+ # Estimator save & load
+ with tempfile.TemporaryDirectory(prefix="decision_tree_classifier") as d:
+ dt.write().overwrite().save(d)
+ dt2 = DecisionTreeClassifier.load(d)
+ self.assertEqual(str(dt), str(dt2))
+
+ model = dt.fit(df)
+ self.assertEqual(model.numClasses, 2)
+ self.assertEqual(model.numFeatures, 2)
+ self.assertEqual(model.depth, 2)
+ self.assertEqual(model.numNodes, 5)
+ self.assertEqual(model.featureImportances, Vectors.sparse(2, [0], [1.0]))
+ self.assertTrue("depth=2, numNodes=5, numClasses=2, numFeatures=2" in model.toDebugString)
+ self.assertTrue("If (feature 0 <= 2.5)" in model.toDebugString)
+
+ vec = Vectors.dense(0.0, 5.0)
+ self.assertEqual(model.predict(vec), 1.0)
+ self.assertEqual(model.predictRaw(vec), Vectors.dense(0.0, 1.0))
+ self.assertEqual(model.predictProbability(vec), Vectors.dense(0.0, 1.0))
+ self.assertEqual(model.predictLeaf(vec), 0.0)
+
+ output = model.transform(df)
+ expected_cols = [
+ "label",
+ "weight",
+ "features",
+ "rawPrediction",
+ "probability",
+ "prediction",
+ "leaf",
+ ]
+ self.assertEqual(output.columns, expected_cols)
+ self.assertEqual(output.count(), 4)
+
+ # Model save & load
+ with tempfile.TemporaryDirectory(prefix="decision_tree_model") as d:
+ model.write().overwrite().save(d)
+ model2 = DecisionTreeClassificationModel.load(d)
+ self.assertEqual(str(model), str(model2))
+ self.assertEqual(model.toDebugString, model2.toDebugString)
+
+ def test_gbt_classifier(self):
+ df = (
+ self.spark.createDataFrame(
+ [
+ (1.0, 1.0, Vectors.dense(0.0, 5.0)),
+ (0.0, 2.0, Vectors.dense(1.0, 2.0)),
+ (1.0, 3.0, Vectors.dense(2.0, 1.0)),
+ (0.0, 4.0, Vectors.dense(3.0, 3.0)),
+ ],
+ ["label", "weight", "features"],
+ )
+ .coalesce(1)
+ .sortWithinPartitions("weight")
+ )
+
+ gbt = GBTClassifier(
+ maxIter=3,
+ maxDepth=2,
+ labelCol="label",
+ leafCol="leaf",
+ seed=1,
+ )
+ self.assertEqual(gbt.getMaxIter(), 3)
+ self.assertEqual(gbt.getMaxDepth(), 2)
+ self.assertEqual(gbt.getSeed(), 1)
+ self.assertEqual(gbt.getLabelCol(), "label")
+ self.assertEqual(gbt.getLeafCol(), "leaf")
+
+ # Estimator save & load
+ with tempfile.TemporaryDirectory(prefix="gbt_classifier") as d:
+ gbt.write().overwrite().save(d)
+ gbt2 = GBTClassifier.load(d)
+ self.assertEqual(str(gbt), str(gbt2))
+
+ model = gbt.fit(df)
+ self.assertEqual(model.numClasses, 2)
+ self.assertEqual(model.numFeatures, 2)
+ # TODO(SPARK-50843): Support access submodel in TreeEnsembleModel
+ # model.trees
Review Comment:
`GBTClassificationModel.trees` not supported, file SPARK-50843 to trace it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50825][ML][PYTHON][CONNECT] Support Tree Classifiers in ML Connect [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #49525:
URL: https://github.com/apache/spark/pull/49525#issuecomment-2594703983
cc @wbo4958
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50825][ML][PYTHON][CONNECT] Support Tree Classifiers in ML Connect [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #49525:
URL: https://github.com/apache/spark/pull/49525#discussion_r1917861976
##########
python/pyspark/ml/tests/test_classification.py:
##########
@@ -283,6 +293,318 @@ def test_logistic_regression(self):
except OSError:
pass
+ def test_decision_tree_classifier(self):
+ df = (
+ self.spark.createDataFrame(
+ [
+ (1.0, 1.0, Vectors.dense(0.0, 5.0)),
+ (0.0, 2.0, Vectors.dense(1.0, 2.0)),
+ (1.0, 3.0, Vectors.dense(2.0, 1.0)),
+ (0.0, 4.0, Vectors.dense(3.0, 3.0)),
+ ],
+ ["label", "weight", "features"],
+ )
+ .coalesce(1)
+ .sortWithinPartitions("weight")
+ )
+
+ dt = DecisionTreeClassifier(
+ maxDepth=2,
+ labelCol="label",
+ leafCol="leaf",
+ seed=1,
+ )
+ self.assertEqual(dt.getMaxDepth(), 2)
+ self.assertEqual(dt.getSeed(), 1)
+ self.assertEqual(dt.getLabelCol(), "label")
+ self.assertEqual(dt.getLeafCol(), "leaf")
+
+ # Estimator save & load
+ with tempfile.TemporaryDirectory(prefix="decision_tree_classifier") as d:
+ dt.write().overwrite().save(d)
+ dt2 = DecisionTreeClassifier.load(d)
+ self.assertEqual(str(dt), str(dt2))
+
+ model = dt.fit(df)
+ self.assertEqual(model.numClasses, 2)
+ self.assertEqual(model.numFeatures, 2)
+ self.assertEqual(model.depth, 2)
+ self.assertEqual(model.numNodes, 5)
+ self.assertEqual(model.featureImportances, Vectors.sparse(2, [0], [1.0]))
+ self.assertTrue("depth=2, numNodes=5, numClasses=2, numFeatures=2" in model.toDebugString)
+ self.assertTrue("If (feature 0 <= 2.5)" in model.toDebugString)
+
+ vec = Vectors.dense(0.0, 5.0)
+ self.assertEqual(model.predict(vec), 1.0)
+ self.assertEqual(model.predictRaw(vec), Vectors.dense(0.0, 1.0))
+ self.assertEqual(model.predictProbability(vec), Vectors.dense(0.0, 1.0))
+ self.assertEqual(model.predictLeaf(vec), 0.0)
+
+ output = model.transform(df)
+ expected_cols = [
+ "label",
+ "weight",
+ "features",
+ "rawPrediction",
+ "probability",
+ "prediction",
+ "leaf",
+ ]
+ self.assertEqual(output.columns, expected_cols)
+ self.assertEqual(output.count(), 4)
+
+ # Model save & load
+ with tempfile.TemporaryDirectory(prefix="decision_tree_model") as d:
+ model.write().overwrite().save(d)
+ model2 = DecisionTreeClassificationModel.load(d)
+ self.assertEqual(str(model), str(model2))
+ self.assertEqual(model.toDebugString, model2.toDebugString)
+
+ def test_gbt_classifier(self):
+ df = (
+ self.spark.createDataFrame(
+ [
+ (1.0, 1.0, Vectors.dense(0.0, 5.0)),
+ (0.0, 2.0, Vectors.dense(1.0, 2.0)),
+ (1.0, 3.0, Vectors.dense(2.0, 1.0)),
+ (0.0, 4.0, Vectors.dense(3.0, 3.0)),
+ ],
+ ["label", "weight", "features"],
+ )
+ .coalesce(1)
+ .sortWithinPartitions("weight")
+ )
+
+ gbt = GBTClassifier(
+ maxIter=3,
+ maxDepth=2,
+ labelCol="label",
+ leafCol="leaf",
+ seed=1,
+ )
+ self.assertEqual(gbt.getMaxIter(), 3)
+ self.assertEqual(gbt.getMaxDepth(), 2)
+ self.assertEqual(gbt.getSeed(), 1)
+ self.assertEqual(gbt.getLabelCol(), "label")
+ self.assertEqual(gbt.getLeafCol(), "leaf")
+
+ # Estimator save & load
+ with tempfile.TemporaryDirectory(prefix="gbt_classifier") as d:
+ gbt.write().overwrite().save(d)
+ gbt2 = GBTClassifier.load(d)
+ self.assertEqual(str(gbt), str(gbt2))
+
+ model = gbt.fit(df)
+ self.assertEqual(model.numClasses, 2)
+ self.assertEqual(model.numFeatures, 2)
+ # TODO(SPARK-50843): Support access submodel in TreeEnsembleModel
+ # model.trees
Review Comment:
`GBTClassificationModel.trees` no supported, file SPARK-50843 to trace it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50825][ML][PYTHON][CONNECT] Support Tree Classifiers in ML Connect [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #49525:
URL: https://github.com/apache/spark/pull/49525#issuecomment-2595059891
thanks, merged to master/4.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50825][ML][PYTHON][CONNECT] Support Tree Classifiers in ML Connect [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #49525:
URL: https://github.com/apache/spark/pull/49525#discussion_r1917858681
##########
dev/sparktestsupport/modules.py:
##########
@@ -1116,7 +1116,7 @@ def __hash__(self):
"pyspark.ml.tests.connect.test_connect_classification",
"pyspark.ml.tests.connect.test_connect_pipeline",
"pyspark.ml.tests.connect.test_connect_tuning",
- "pyspark.ml.tests.connect.test_connect_spark_ml_classification",
+ "pyspark.ml.tests.connect.test_parity_classification",
Review Comment:
rename to follow the `test_parity_*` pattern used in other python modules
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org