You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "shahid (Jira)" <ji...@apache.org> on 2019/09/25 05:50:00 UTC
[jira] [Commented] (SPARK-29235) CrossValidatorModel.avgMetrics
disappears after model is written/read again
[ https://issues.apache.org/jira/browse/SPARK-29235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16937427#comment-16937427 ]
shahid commented on SPARK-29235:
--------------------------------
I would like to analyze the issue.
> CrossValidatorModel.avgMetrics disappears after model is written/read again
> ---------------------------------------------------------------------------
>
> Key: SPARK-29235
> URL: https://issues.apache.org/jira/browse/SPARK-29235
> Project: Spark
> Issue Type: Bug
> Components: ML
> Affects Versions: 2.4.1, 2.4.3
> Environment: Databricks cluster:
> {
> "num_workers": 4,
> "cluster_name": "mabedfor-test-classfix",
> "spark_version": "5.3.x-cpu-ml-scala2.11",
> "spark_conf": {
> "spark.databricks.delta.preview.enabled": "true"
> },
> "node_type_id": "Standard_DS12_v2",
> "driver_node_type_id": "Standard_DS12_v2",
> "ssh_public_keys": [],
> "custom_tags": {},
> "spark_env_vars": {
> "PYSPARK_PYTHON": "/databricks/python3/bin/python3"
> },
> "autotermination_minutes": 120,
> "enable_elastic_disk": true,
> "cluster_source": "UI",
> "init_scripts": [],
> "cluster_id": "0722-165622-calls746"
> }
> Reporter: Matthew Bedford
> Priority: Minor
>
>
> Right after a CrossValidatorModel is trained, it has avgMetrics. After the model is written to disk and read later, it no longer has avgMetrics. To reproduce:
> {{from pyspark.ml.tuning import CrossValidator, CrossValidatorModel}}
> {{cv = CrossValidator(...) #fill with params}}
> {{cvModel = cv.fit(trainDF) #given dataframe with training data}}
> {{data}}{{print(cvModel.avgMetrics) #prints a nonempty list as expected}}
> {{cvModel.write().save({color:#172b4d}"/tmp/model"{color})}}
> {{cvModel2 = CrossValidatorModel.read().load({color:#172b4d}"/tmp/model"{color})}}
> {{print(cvModel2.avgMetrics) #BUG - prints an empty list}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org