You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2025/02/03 10:20:44 UTC
(spark) branch branch-4.0 updated: [SPARK-51060][ML][PYTHON][CONNECT][FOLLOW-UP] Fix the `uid` of `Bucketizer` fitted by `QuantileDiscretizer`
This is an automated email from the ASF dual-hosted git repository.
ruifengz pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 2fee4b5bed3b [SPARK-51060][ML][PYTHON][CONNECT][FOLLOW-UP] Fix the `uid` of `Bucketizer` fitted by `QuantileDiscretizer`
2fee4b5bed3b is described below
commit 2fee4b5bed3b0da49e7428137a488309673a0239
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Mon Feb 3 18:20:04 2025 +0800
[SPARK-51060][ML][PYTHON][CONNECT][FOLLOW-UP] Fix the `uid` of `Bucketizer` fitted by `QuantileDiscretizer`
### What changes were proposed in this pull request?
Fix the `uid` of `Bucketizer` fitted by `QuantileDiscretizer`
### Why are the changes needed?
On connect, the `Bucketizer` fitted by `QuantileDiscretizer` has the same `uid` as `QuantileDiscretizer`.
This is not consistent with pyspark classic.
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
added tests
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #49770 from zhengruifeng/ml_connect_qd_uid.
Authored-by: Ruifeng Zheng <ru...@apache.org>
Signed-off-by: Ruifeng Zheng <ru...@apache.org>
(cherry picked from commit bff86f11375c669e3d33e85624f07249eb4a04a7)
Signed-off-by: Ruifeng Zheng <ru...@apache.org>
---
python/pyspark/ml/tests/test_feature.py | 8 ++++++--
python/pyspark/ml/util.py | 3 ++-
2 files changed, 8 insertions(+), 3 deletions(-)
diff --git a/python/pyspark/ml/tests/test_feature.py b/python/pyspark/ml/tests/test_feature.py
index b458a4eb7a25..068ce8f5bb6d 100644
--- a/python/pyspark/ml/tests/test_feature.py
+++ b/python/pyspark/ml/tests/test_feature.py
@@ -998,7 +998,9 @@ class FeatureTestsMixin:
bucketizer = qds.fit(df)
self.assertIsInstance(bucketizer, Bucketizer)
# Bucketizer doesn't inherit uid from QuantileDiscretizer
- # self.assertEqual(qds.uid, bucketizer.uid)
+ self.assertNotEqual(qds.uid, bucketizer.uid)
+ self.assertTrue(qds.uid.startswith("QuantileDiscretizer"))
+ self.assertTrue(bucketizer.uid.startswith("Bucketizer"))
# check model coefficients
self.assertEqual(bucketizer.getSplits(), [float("-inf"), 0.4, float("inf")])
@@ -1046,7 +1048,9 @@ class FeatureTestsMixin:
bucketizer = qds.fit(df)
self.assertIsInstance(bucketizer, Bucketizer)
# Bucketizer doesn't inherit uid from QuantileDiscretizer
- # self.assertEqual(qds.uid, bucketizer.uid)
+ self.assertNotEqual(qds.uid, bucketizer.uid)
+ self.assertTrue(qds.uid.startswith("QuantileDiscretizer"))
+ self.assertTrue(bucketizer.uid.startswith("Bucketizer"))
# check model coefficients
self.assertEqual(
diff --git a/python/pyspark/ml/util.py b/python/pyspark/ml/util.py
index b895a86ae535..7d4c1b9460f8 100644
--- a/python/pyspark/ml/util.py
+++ b/python/pyspark/ml/util.py
@@ -136,7 +136,8 @@ def try_remote_fit(f: FuncT) -> FuncT:
model_info = deserialize(properties)
client.add_ml_cache(model_info.obj_ref.id)
model = self._create_model(model_info.obj_ref.id)
- model._resetUid(self.uid)
+ if model.__class__.__name__ not in ["Bucketizer"]:
+ model._resetUid(self.uid)
return self._copyValues(model)
else:
return f(self, dataset)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org