[jira] [Created] (SPARK-21556) PySpark, Unable to save pipeline of non-spark transformers

["Saif Addin (JIRA)" <ji...@apache.org>]

Saif Addin created SPARK-21556:
----------------------------------

             Summary: PySpark, Unable to save pipeline of non-spark transformers
                 Key: SPARK-21556
                 URL: https://issues.apache.org/jira/browse/SPARK-21556
             Project: Spark
          Issue Type: Bug
          Components: ML, PySpark
    Affects Versions: 2.1.1
            Reporter: Saif Addin
            Priority: Minor


We are working on creating some new ML transformers following the same Spark / PyPark design pattern.
When in PySpark, though, we are unable to deserialize, or read Pipelines, made of such new Transformers, due to a hardcoded class path name in *wrapper.py*

https://github.com/apache/spark/blob/master/python/pyspark/ml/wrapper.py#L200

So this line makes pipeline components work only if JVM classes are equivalent to Python classes with the root replaced. But, would not be working for more general use cases.

The first workaround that comes to mind, is use the same pathing for pyspark side than jvm side.

The error, when trying to load a Pipeline from path in such circumstances is 

{code:java}

E
======================================================================
ERROR: runTest (test.annotators.PipelineTestSpec)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/saif/IdeaProjects/this_project/test/annotators.py", line 208, in runTest
    loaded_pipeline = Pipeline.read().load(pipe_path)
  File "/home/saif/apps/spark-2.1.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/ml/util.py", line 198, in load
    return self._clazz._from_java(java_obj)
  File "/home/saif/apps/spark-2.1.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/ml/pipeline.py", line 155, in _from_java
    py_stages = [JavaParams._from_java(s) for s in java_stage.getStages()]
  File "/home/saif/apps/spark-2.1.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/ml/pipeline.py", line 155, in <listcomp>
    py_stages = [JavaParams._from_java(s) for s in java_stage.getStages()]
  File "/home/saif/apps/spark-2.1.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/ml/wrapper.py", line 173, in _from_java
    py_type = __get_class(stage_name)
  File "/home/saif/apps/spark-2.1.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/ml/wrapper.py", line 167, in __get_class
    m = __import__(module)
ModuleNotFoundError: No module named 'com.frh'

{code}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org