[jira] [Comment Edited] (SPARK-29428) Can't persist/set None-valued param

["Borys Biletskyy (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-29428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951747#comment-16951747 ] 

Borys Biletskyy edited comment on SPARK-29428 at 10/15/19 9:12 AM:
-------------------------------------------------------------------

[~bryanc], I tried to implement some logic depending on the default/non-default value of parameters. I used isSet(...) method to understand if the parameter was set, until I found out that that after persistence isSet(...) was not preserved (https://issues.apache.org/jira/browse/SPARK-29414). So I decided to use None values as a workaround to distinguish default from non-default param values. It seemed to work, until I tried to persist it.

Having None params doesn't seem to be a good idea to me either, especially when there are isSet(...) and isDefined(...) methods.


was (Author: borys.biletskyy):
[~bryanc], I tried to implement some logic depending on the default/non-default value of parameters. I used isSet(...) method to understand if the parameter was set, until I found out that that after persistence isSet(...) was not preserved (https://issues.apache.org/jira/browse/SPARK-29414). So I decided to use None values as a workaround to distinguish default from non-default param values. It seemed to work, until I tried to persist it.

> Can't persist/set None-valued param 
> ------------------------------------
>
>                 Key: SPARK-29428
>                 URL: https://issues.apache.org/jira/browse/SPARK-29428
>             Project: Spark
>          Issue Type: Bug
>          Components: ML, PySpark
>    Affects Versions: 2.3.2
>            Reporter: Borys Biletskyy
>            Priority: Major
>
> {code:java}
> import pytest
> from pyspark import keyword_only
> from pyspark.ml import Model
> from pyspark.sql import DataFrame
> from pyspark.ml.util import DefaultParamsReadable, DefaultParamsWritable
> from pyspark.ml.param.shared import HasInputCol
> from pyspark.sql.functions import *
> class NoneParamTester(Model,
>                       HasInputCol,
>                       DefaultParamsReadable,
>                       DefaultParamsWritable
>                       ):
>     @keyword_only
>     def __init__(self, inputCol: str = None):
>         super(NoneParamTester, self).__init__()
>         kwargs = self._input_kwargs
>         self.setParams(**kwargs)
>     @keyword_only
>     def setParams(self, inputCol: str = None):
>         kwargs = self._input_kwargs
>         self._set(**kwargs)
>         return self
>     def _transform(self, data: DataFrame) -> DataFrame:
>         return data
> class TestNoneParam(object):
>     def test_persist_none(self, spark, temp_dir):
>         path = temp_dir + '/test_model'
>         model = NoneParamTester(inputCol=None)
>         assert model.isDefined(model.inputCol)
>         assert model.isSet(model.inputCol)
>         assert model.getInputCol() is None
>         model.write().overwrite().save(path)
>         NoneParamTester.load(path)  # TypeError: Could not convert <class 'NoneType'> to string type
>     def test_set_none(self, spark):
>         model = NoneParamTester(inputCol=None)
>         assert model.isDefined(model.inputCol)
>         assert model.isSet(model.inputCol)
>         assert model.getInputCol() is None
>         model.set(model.inputCol, None)  # TypeError: Could not convert <class 'NoneType'> to string type
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org