[jira] [Created] (SPARK-33592) Pyspark ML Validator writer may lost params in estimatorParamMaps

["Weichen Xu (Jira)" <ji...@apache.org>]

Weichen Xu created SPARK-33592:
----------------------------------

             Summary: Pyspark ML Validator writer may lost params in estimatorParamMaps
                 Key: SPARK-33592
                 URL: https://issues.apache.org/jira/browse/SPARK-33592
             Project: Spark
          Issue Type: Bug
          Components: ML, PySpark
    Affects Versions: 3.0.0, 3.1.0
            Reporter: Weichen Xu


Two typical cases to reproduce it:
(1)
{code: python}
tokenizer = Tokenizer(inputCol="text", outputCol="words")
hashingTF = HashingTF(inputCol=tokenizer.getOutputCol(), outputCol="features")
lr = LogisticRegression()
pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])

paramGrid = ParamGridBuilder() \
    .addGrid(hashingTF.numFeatures, [10, 100]) \
    .addGrid(lr.maxIter, [100, 200]) \
    .build()
tvs = TrainValidationSplit(estimator=pipeline,
                           estimatorParamMaps=paramGrid,
                           evaluator=MulticlassClassificationEvaluator())

tvs.save(tvsPath)
loadedTvs = TrainValidationSplit.load(tvsPath)

{code}

Then we can check `loadedTvs.getEstimatorParamMaps()`, the tuning params `hashingTF.numFeatures` and `lr.maxIter` are lost.


(2)
{code: python}
lr = LogisticRegression()
ova = OneVsRest(classifier=lr)
grid = ParamGridBuilder().addGrid(lr.maxIter, [100, 200]).build()
evaluator = MulticlassClassificationEvaluator()
tvs = TrainValidationSplit(estimator=ova, estimatorParamMaps=grid, evaluator=evaluator)

tvs.save(tvsPath)
loadedTvs = TrainValidationSplit.load(tvsPath)

{code}
Then we can check `loadedTvs.getEstimatorParamMaps()`, the tuning params`lr.maxIter` are lost.


Both CrossValidator and TrainValidationSplit has this issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org