You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:01:11 UTC
[jira] [Updated] (SPARK-21556) PySpark, Unable to save pipeline of
non-spark transformers
[ https://issues.apache.org/jira/browse/SPARK-21556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-21556:
---------------------------------
Labels: bulk-closed (was: )
> PySpark, Unable to save pipeline of non-spark transformers
> ----------------------------------------------------------
>
> Key: SPARK-21556
> URL: https://issues.apache.org/jira/browse/SPARK-21556
> Project: Spark
> Issue Type: Bug
> Components: ML, PySpark
> Affects Versions: 2.1.1
> Reporter: Saif Addin
> Priority: Minor
> Labels: bulk-closed
>
> We are working on creating some new ML transformers following the same Spark / PyPark design pattern.
> When in PySpark, though, we are unable to deserialize, or read Pipelines, made of such new Transformers, due to a hardcoded class path name in *wrapper.py*
> https://github.com/apache/spark/blob/master/python/pyspark/ml/wrapper.py#L200
> So this line makes pipeline components work only if JVM classes are equivalent to Python classes with the root replaced. But, would not be working for more general use cases.
> The first workaround that comes to mind, is use the same pathing for pyspark side than jvm side.
> The error, when trying to load a Pipeline from path in such circumstances is
> {code:java}
> E
> ======================================================================
> ERROR: runTest (test.annotators.PipelineTestSpec)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/home/saif/IdeaProjects/this_project/test/annotators.py", line 208, in runTest
> loaded_pipeline = Pipeline.read().load(pipe_path)
> File "/home/saif/apps/spark-2.1.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/ml/util.py", line 198, in load
> return self._clazz._from_java(java_obj)
> File "/home/saif/apps/spark-2.1.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/ml/pipeline.py", line 155, in _from_java
> py_stages = [JavaParams._from_java(s) for s in java_stage.getStages()]
> File "/home/saif/apps/spark-2.1.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/ml/pipeline.py", line 155, in <listcomp>
> py_stages = [JavaParams._from_java(s) for s in java_stage.getStages()]
> File "/home/saif/apps/spark-2.1.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/ml/wrapper.py", line 173, in _from_java
> py_type = __get_class(stage_name)
> File "/home/saif/apps/spark-2.1.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/ml/wrapper.py", line 167, in __get_class
> m = __import__(module)
> ModuleNotFoundError: No module named 'com.frh'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org