(spark) branch branch-4.2 updated: [SPARK-56762][PYTHON][TESTS] Speed up test_crossvalidator_on_pipeline

[gu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.2 by this push:
     new cebd28b512bd [SPARK-56762][PYTHON][TESTS] Speed up test_crossvalidator_on_pipeline
cebd28b512bd is described below

commit cebd28b512bd52631e207c396da8a8a282a90775
Author: Tian Gao <ga...@hotmail.com>
AuthorDate: Thu May 7 18:24:38 2026 +0900

    [SPARK-56762][PYTHON][TESTS] Speed up test_crossvalidator_on_pipeline
    
    ### What changes were proposed in this pull request?
    
    Tuned some parameters to speed up the slowest test in our test suite. With this change, we can reduce the time of this test by about 50%.
    
    Note that the test is for `CrossValidator`, so tuning the model a bit is totally fine for the purpose.
    
    ### Why are the changes needed?
    
    This test takes 100+s to finish on our CI.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Locally passed. CI needed.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #55723 from gaogaotiantian/speed-up-crossvalidator.
    
    Authored-by: Tian Gao <ga...@hotmail.com>
    Signed-off-by: Hyukjin Kwon <gu...@apache.org>
    (cherry picked from commit b4c1880baf52777fdb8ed606715dc1f278fd6342)
    Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
 python/pyspark/ml/tests/connect/test_legacy_mode_tuning.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/python/pyspark/ml/tests/connect/test_legacy_mode_tuning.py b/python/pyspark/ml/tests/connect/test_legacy_mode_tuning.py
index e2cde9ce7554..ac99e113c447 100644
--- a/python/pyspark/ml/tests/connect/test_legacy_mode_tuning.py
+++ b/python/pyspark/ml/tests/connect/test_legacy_mode_tuning.py
@@ -199,7 +199,7 @@ class CrossValidatorTestsMixin:
         )
 
         scaler = StandardScaler(inputCol="features", outputCol="scaled_features")
-        lorv2 = LORV2(numTrainWorkers=2, featuresCol="scaled_features")
+        lorv2 = LORV2(numTrainWorkers=1, featuresCol="scaled_features")
         pipeline = Pipeline(stages=[scaler, lorv2])
 
         grid2 = ParamGridBuilder().addGrid(lorv2.maxIter, [2, 3]).build()
@@ -208,6 +208,7 @@ class CrossValidatorTestsMixin:
             estimatorParamMaps=grid2,
             parallelism=2,
             evaluator=BinaryClassificationEvaluator(),
+            numFolds=2,
         )
         cv_model = cv.fit(train_dataset)
         transformed_result = (


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org