You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2024/10/23 05:08:49 UTC
(spark) branch master updated: [SPARK-50064][PYTHON][TESTS][FOLLOW-UP] Make pysaprk-ml tests passing without optional dependencies
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 51e915a0ca2d [SPARK-50064][PYTHON][TESTS][FOLLOW-UP] Make pysaprk-ml tests passing without optional dependencies
51e915a0ca2d is described below
commit 51e915a0ca2d3f1c9d8337d1ee9692247026c40f
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Wed Oct 23 14:08:37 2024 +0900
[SPARK-50064][PYTHON][TESTS][FOLLOW-UP] Make pysaprk-ml tests passing without optional dependencies
### What changes were proposed in this pull request?
This PR is a followup of https://github.com/apache/spark/pull/48587 that proposes to make pysaprk-ml tests passing without optional dependencies
### Why are the changes needed?
To make the tests passing without optional dependencies. See https://github.com/apache/spark/actions/runs/11447673972/job/31849621508
### Does this PR introduce _any_ user-facing change?
No, test-only.
### How was this patch tested?
Manually ran it locally
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #48606 from HyukjinKwon/SPARK-50064-followup.
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
.../ml/tests/connect/test_legacy_mode_tuning.py | 49 ++++++++++------------
1 file changed, 23 insertions(+), 26 deletions(-)
diff --git a/python/pyspark/ml/tests/connect/test_legacy_mode_tuning.py b/python/pyspark/ml/tests/connect/test_legacy_mode_tuning.py
index 14f52d75e6d6..0e3482157be5 100644
--- a/python/pyspark/ml/tests/connect/test_legacy_mode_tuning.py
+++ b/python/pyspark/ml/tests/connect/test_legacy_mode_tuning.py
@@ -47,37 +47,34 @@ if should_test_connect:
from pyspark.ml.connect.tuning import CrossValidator, CrossValidatorModel
from pyspark.ml.connect.evaluation import BinaryClassificationEvaluator, RegressionEvaluator
+ class HasInducedError(Params):
+ def __init__(self):
+ super(HasInducedError, self).__init__()
+ self.inducedError = Param(
+ self, "inducedError", "Uniformly-distributed error added to feature"
+ )
-class HasInducedError(Params):
- def __init__(self):
- super(HasInducedError, self).__init__()
- self.inducedError = Param(
- self, "inducedError", "Uniformly-distributed error added to feature"
- )
-
- def getInducedError(self):
- return self.getOrDefault(self.inducedError)
-
-
-class InducedErrorModel(Model, HasInducedError):
- def __init__(self):
- super(InducedErrorModel, self).__init__()
+ def getInducedError(self):
+ return self.getOrDefault(self.inducedError)
- def _transform(self, dataset):
- return dataset.withColumn(
- "prediction", dataset.feature + (rand(0) * self.getInducedError())
- )
+ class InducedErrorModel(Model, HasInducedError):
+ def __init__(self):
+ super(InducedErrorModel, self).__init__()
+ def _transform(self, dataset):
+ return dataset.withColumn(
+ "prediction", dataset.feature + (rand(0) * self.getInducedError())
+ )
-class InducedErrorEstimator(Estimator, HasInducedError):
- def __init__(self, inducedError=1.0):
- super(InducedErrorEstimator, self).__init__()
- self._set(inducedError=inducedError)
+ class InducedErrorEstimator(Estimator, HasInducedError):
+ def __init__(self, inducedError=1.0):
+ super(InducedErrorEstimator, self).__init__()
+ self._set(inducedError=inducedError)
- def _fit(self, dataset):
- model = InducedErrorModel()
- self._copyValues(model)
- return model
+ def _fit(self, dataset):
+ model = InducedErrorModel()
+ self._copyValues(model)
+ return model
class CrossValidatorTestsMixin:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org