You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ga...@apache.org on 2026/10/11 08:34:51 UTC
(spark) branch branch-4.x updated: [SPARK-60151][ML][PYTHON] Fix CrossValidatorModel.load returning the last fold's sub-models for every fold
This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new d30396e04024 [SPARK-60151][ML][PYTHON] Fix CrossValidatorModel.load returning the last fold's sub-models for every fold
d30396e04024 is described below
commit d30396e040243314268d506798972e6b1968d6bd
Author: LaPalace <r1...@ntu.edu.tw>
AuthorDate: Sun Oct 11 01:34:15 2026 -0700
[SPARK-60151][ML][PYTHON] Fix CrossValidatorModel.load returning the last fold's sub-models for every fold
### What changes were proposed in this pull request?
Fix `CrossValidatorModelReader.load` in `python/pyspark/ml/tuning.py` so that each fold gets its own list of sub-models:
```python
# before
subModels = [[None] * len(estimatorParamMaps)] * numFolds
# after
subModels = [[None] * len(estimatorParamMaps) for _ in range(numFolds)]
```
`[x] * n` repeats a reference to the same inner list, so every fold shared one list and each fold's assignment overwrote the previous one. `CrossValidator._fit`, `CrossValidatorModel.copy`, `CrossValidatorModel._from_java` and the Scala reader (`Array.fill(numFolds)(...)`) already build an independent list per fold.
The existing test in `test_tuning.py` compared `model.subModels[i][j].uid` with itself, so it always passed. After loading it only checked list lengths. This PR replaces that check with a comparison of each loaded sub-model's coefficients and intercept against the original. Comparing uids would not catch this bug either, because all sub-models share the estimator's uid.
### Why are the changes needed?
After `CrossValidatorModel.load`, every entry of `subModels` returns the last fold's models. The files on disk are correct; only the PySpark reader is affected. Saving a loaded model again writes the last fold's models into every fold directory, so the original sub-models are lost. `bestModel` and `avgMetrics` are not affected.
Reproduction (`numFolds=3`, 2 param maps, `collectSubModels=True`), first coefficient of each sub-model:
| | fold0 | fold1 | fold2 |
|---|---|---|---|
| after `fit` | [0.0705, 0.0439] | [0.0067, 0.0047] | [-0.0155, -0.0108] |
| on disk after `save` | [0.0705, 0.0439] | [0.0067, 0.0047] | [-0.0155, -0.0108] |
| after `CrossValidatorModel.load` | [-0.0155, -0.0108] | [-0.0155, -0.0108] | [-0.0155, -0.0108] |
`loaded.subModels[0] is loaded.subModels[1]` returns `True`.
The bug was introduced by SPARK-33520.
### Does this PR introduce _any_ user-facing change?
Yes, as a bug fix. `CrossValidatorModel.load` now returns the correct sub-models for each fold. Previously every fold returned the last fold's sub-models.
### How was this patch tested?
Updated `TuningTestsMixin.test_cross_validator` to check that the loaded sub-models match the original ones. The test fails without the fix (`AssertionError: 0.6466271649250525 != 0.10536051565782635` on the intercept) and passes with it. The mixin is also used by the Spark Connect parity test.
```
python/run-tests --testnames 'pyspark.ml.tests.test_tuning,pyspark.ml.tests.tuning.test_tuning,pyspark.ml.tests.tuning.test_cv_io_basic,pyspark.ml.tests.tuning.test_cv_io_nested,pyspark.ml.tests.tuning.test_cv_io_pipeline,pyspark.ml.tests.tuning.test_tvs_io_basic,pyspark.ml.tests.connect.test_parity_tuning,pyspark.ml.tests.connect.test_legacy_mode_tuning'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
Closes #59353 from chinaoel/SPARK-60151.
Authored-by: LaPalace <r1...@ntu.edu.tw>
Signed-off-by: Tian Gao <ga...@hotmail.com>
(cherry picked from commit 1a32dcaa6b786243bfe56ef748726c9ffc711832)
Signed-off-by: Tian Gao <ga...@hotmail.com>
---
python/pyspark/ml/tests/test_tuning.py | 10 +++++++---
python/pyspark/ml/tuning.py | 2 +-
2 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/python/pyspark/ml/tests/test_tuning.py b/python/pyspark/ml/tests/test_tuning.py
index 6e2a03a3049f..a4c70a108dde 100644
--- a/python/pyspark/ml/tests/test_tuning.py
+++ b/python/pyspark/ml/tests/test_tuning.py
@@ -154,9 +154,12 @@ class TuningTestsMixin:
checkSubModels(model.subModels)
- for i in range(numFolds):
- for j in range(len(grid)):
- self.assertEqual(model.subModels[i][j].uid, model.subModels[i][j].uid)
+ def checkSubModelsEqual(subModels1, subModels2):
+ for i in range(numFolds):
+ for j in range(len(grid)):
+ m1, m2 = subModels1[i][j], subModels2[i][j]
+ self.assertEqual(m1.coefficients, m2.coefficients)
+ self.assertEqual(m1.intercept, m2.intercept)
# save & load
with tempfile.TemporaryDirectory(prefix="cv") as d:
@@ -169,6 +172,7 @@ class TuningTestsMixin:
model.write().overwrite().save(d)
model2 = CrossValidatorModel.load(d)
checkSubModels(model2.subModels)
+ checkSubModelsEqual(model.subModels, model2.subModels)
self.assertEqual(str(model), str(model2))
self.assertEqual(str(model.getEstimator()), str(model2.getEstimator()))
self.assertEqual(str(model.getEvaluator()), str(model2.getEvaluator()))
diff --git a/python/pyspark/ml/tuning.py b/python/pyspark/ml/tuning.py
index e4bac8d302c5..25bf1e1a6a38 100644
--- a/python/pyspark/ml/tuning.py
+++ b/python/pyspark/ml/tuning.py
@@ -569,7 +569,7 @@ class CrossValidatorModelReader(MLReader["CrossValidatorModel"]):
persistSubModels = ("persistSubModels" in metadata) and metadata["persistSubModels"]
if persistSubModels:
- subModels = [[None] * len(estimatorParamMaps)] * numFolds
+ subModels = [[None] * len(estimatorParamMaps) for _ in range(numFolds)]
for splitIndex in range(numFolds):
for paramIndex in range(len(estimatorParamMaps)):
modelPath = os.path.join(
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org