[jira] [Created] (SPARK-29235) CrossValidatorModel.avgMetrics disappears after model is written/read again

["Matthew Bedford (Jira)" <ji...@apache.org>]

Matthew Bedford created SPARK-29235:
---------------------------------------

             Summary: CrossValidatorModel.avgMetrics disappears after model is written/read again
                 Key: SPARK-29235
                 URL: https://issues.apache.org/jira/browse/SPARK-29235
             Project: Spark
          Issue Type: Bug
          Components: ML
    Affects Versions: 2.4.1
         Environment: Databricks cluster:

{
    "num_workers": 4,
    "cluster_name": "mabedfor-test-classfix",
    "spark_version": "5.3.x-cpu-ml-scala2.11",
    "spark_conf": {
        "spark.databricks.delta.preview.enabled": "true"
    },
    "node_type_id": "Standard_DS12_v2",
    "driver_node_type_id": "Standard_DS12_v2",
    "ssh_public_keys": [],
    "custom_tags": {},
    "spark_env_vars": {
        "PYSPARK_PYTHON": "/databricks/python3/bin/python3"
    },
    "autotermination_minutes": 120,
    "enable_elastic_disk": true,
    "cluster_source": "UI",
    "init_scripts": [],
    "cluster_id": "0722-165622-calls746"
}
            Reporter: Matthew Bedford


 
Right after a CrossValidatorModel is trained, it has avgMetrics.  After the model is written to disk and read later, it no longer has avgMetrics.  To reproduce:
{{from pyspark.ml.tuning import CrossValidator, CrossValidatorModel
}}{{}}

{{cv = CrossValidator(...) #fill with params
}}{{}}

{{cvModel = cv.fit(trainDF) #given dataframe with training}}

{{data}}{{print(cvModel.avgMetrics) #prints a nonempty list as expected}}

{{cvModel.write().save({color:#172b4d}"/tmp/model"{color})}}

{{cvModel2 = CrossValidatorModel.read().load({color:#172b4d}{color:#172b4d}"/tmp/model"{color}{color})}}{{{color:#172b4d}print(cvModel2.avgMetrics) #BUG - prints an empty list{color}}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org