You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jkbradley <gi...@git.apache.org> on 2016/03/21 21:54:08 UTC

[GitHub] spark pull request: [SPARK-13951][ML][PYTHON] Nested Pipeline pers...

GitHub user jkbradley opened a pull request:

    https://github.com/apache/spark/pull/11866

    [SPARK-13951][ML][PYTHON] Nested Pipeline persistence

    ## What changes were proposed in this pull request?
    
    Adds support for saving and loading nested ML Pipelines from Python.  Pipeline and PipelineModel do not extend JavaWrapper, but they are able to utilize the JavaMLWriter, JavaMLReader implementations.
    
    Also:
    * Separates out interfaces from Java wrapper implementations for MLWritable, MLReadable, MLWriter, MLReader.
    * Moves methods _stages_java2py, _stages_py2java into Pipeline, PipelineModel as _transfer_stage_from_java, _transfer_stage_to_java
    
    ## How was this patch tested?
    
    Added new unit test for nested Pipelines.  Abstracted validity check into a helper method for the 2 unit tests.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jkbradley/spark nested-pipeline-io

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/11866.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #11866
    
----
commit 89d538208a0704c43b3752eaec10f5ca7f06b975
Author: Joseph K. Bradley <jo...@databricks.com>
Date:   2016-03-21T20:39:02Z

    Added nested Pipeline persistence

commit 1c070e738236407559c00360eea1012880229076
Author: Joseph K. Bradley <jo...@databricks.com>
Date:   2016-03-21T20:48:16Z

    cleanups, and style fixes

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13951][ML][PYTHON] Nested Pipeline pers...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11866#issuecomment-199914795
  
    **[Test build #53785 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53785/consoleFull)** for PR 11866 at commit [`be879e0`](https://github.com/apache/spark/commit/be879e0e35d0a9fe79a07e27044fd7ddccf501bf).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13951][ML][PYTHON] Nested Pipeline pers...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11866#discussion_r56911884
  
    --- Diff: python/pyspark/ml/pipeline.py ---
    @@ -206,49 +167,65 @@ def save(self, path):
         @classmethod
         @since("2.0.0")
         def read(cls):
    -        """Returns an JavaMLReader instance for this class."""
    +        """Returns an MLReader instance for this class."""
             return PipelineMLReader(cls)
     
         @classmethod
    -    @since("2.0.0")
    -    def load(cls, path):
    -        """Reads an ML instance from the input path, a shortcut of `read().load(path)`."""
    -        return cls.read().load(path)
    +    def _transfer_stage_from_java(cls, java_stage):
    --- End diff --
    
    It is a little confusing to call it `java_stage` while the expected input is a pipeline. Shall we rename it to `_from_java_pipeline` and `_transfer_stage_to_java` to `_to_java_pipeline`? If we have to use the same method name, I would recommend `_from_java` and `_to_java` instead.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13951][ML][PYTHON] Nested Pipeline pers...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11866#issuecomment-199488375
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53711/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13951][ML][PYTHON] Nested Pipeline pers...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/11866#issuecomment-199969824
  
    LGTM. Merged into master. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13951][ML][PYTHON] Nested Pipeline pers...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11866#issuecomment-199915016
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13951][ML][PYTHON] Nested Pipeline pers...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/11866


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13951][ML][PYTHON] Nested Pipeline pers...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11866#issuecomment-199915018
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53785/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13951][ML][PYTHON] Nested Pipeline pers...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11866#issuecomment-199493469
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13951][ML][PYTHON] Nested Pipeline pers...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11866#issuecomment-199910426
  
    **[Test build #53785 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53785/consoleFull)** for PR 11866 at commit [`be879e0`](https://github.com/apache/spark/commit/be879e0e35d0a9fe79a07e27044fd7ddccf501bf).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13951][ML][PYTHON] Nested Pipeline pers...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/11866#issuecomment-199524883
  
    Not part of this PR, in PySpark `DataFrame.write` is a property instead of a method. Shall we follow the same convention?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13951][ML][PYTHON] Nested Pipeline pers...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/11866#issuecomment-199910190
  
    Making MLWritable.write a property sounds good.  I'll create a JIRA for it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13951][ML][PYTHON] Nested Pipeline pers...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11866#discussion_r56911889
  
    --- Diff: python/pyspark/ml/pipeline.py ---
    @@ -206,49 +167,65 @@ def save(self, path):
         @classmethod
         @since("2.0.0")
         def read(cls):
    -        """Returns an JavaMLReader instance for this class."""
    +        """Returns an MLReader instance for this class."""
             return PipelineMLReader(cls)
     
         @classmethod
    -    @since("2.0.0")
    -    def load(cls, path):
    -        """Reads an ML instance from the input path, a shortcut of `read().load(path)`."""
    -        return cls.read().load(path)
    +    def _transfer_stage_from_java(cls, java_stage):
    +        """
    +        Given a Java Pipeline, create and return a Python wrapper of it.
    +        Used for ML persistence.
    +        """
    +        # Create a new instance of this stage.
    +        py_stage = cls()
    +        # Load information from java_stage to the instance.
    +        py_stages = [JavaWrapper._transfer_stage_from_java(s) for s in java_stage.getStages()]
    +        py_stage.setStages(py_stages)
    +        py_stage._resetUid(java_stage.uid())
    +        return py_stage
    +
    +    def _transfer_stage_to_java(self):
    +        """
    +        Transfer this instance to a Java object.  Used for ML persistence.
    --- End diff --
    
    `Java object` -> `Java pipeline`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13951][ML][PYTHON] Nested Pipeline pers...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11866#issuecomment-199493291
  
    **[Test build #53712 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53712/consoleFull)** for PR 11866 at commit [`6d74b97`](https://github.com/apache/spark/commit/6d74b97546a0f50ff9ad08df59c5229874bcb838).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13951][ML][PYTHON] Nested Pipeline pers...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11866#issuecomment-199493474
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53712/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13951][ML][PYTHON] Nested Pipeline pers...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11866#discussion_r57026479
  
    --- Diff: python/pyspark/ml/pipeline.py ---
    @@ -206,49 +167,65 @@ def save(self, path):
         @classmethod
         @since("2.0.0")
         def read(cls):
    -        """Returns an JavaMLReader instance for this class."""
    +        """Returns an MLReader instance for this class."""
             return PipelineMLReader(cls)
     
         @classmethod
    -    @since("2.0.0")
    -    def load(cls, path):
    -        """Reads an ML instance from the input path, a shortcut of `read().load(path)`."""
    -        return cls.read().load(path)
    +    def _transfer_stage_from_java(cls, java_stage):
    --- End diff --
    
    I'd like to use the same method name, so I'll switch to ```_from/to_java```.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13951][ML][PYTHON] Nested Pipeline pers...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11866#issuecomment-199481216
  
    **[Test build #53711 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53711/consoleFull)** for PR 11866 at commit [`1c070e7`](https://github.com/apache/spark/commit/1c070e738236407559c00360eea1012880229076).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13951][ML][PYTHON] Nested Pipeline pers...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11866#issuecomment-199488368
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13951][ML][PYTHON] Nested Pipeline pers...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11866#issuecomment-199489277
  
    **[Test build #53712 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53712/consoleFull)** for PR 11866 at commit [`6d74b97`](https://github.com/apache/spark/commit/6d74b97546a0f50ff9ad08df59c5229874bcb838).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13951][ML][PYTHON] Nested Pipeline pers...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11866#discussion_r57026496
  
    --- Diff: python/pyspark/ml/pipeline.py ---
    @@ -206,49 +167,65 @@ def save(self, path):
         @classmethod
         @since("2.0.0")
         def read(cls):
    -        """Returns an JavaMLReader instance for this class."""
    +        """Returns an MLReader instance for this class."""
             return PipelineMLReader(cls)
     
         @classmethod
    -    @since("2.0.0")
    -    def load(cls, path):
    -        """Reads an ML instance from the input path, a shortcut of `read().load(path)`."""
    -        return cls.read().load(path)
    +    def _transfer_stage_from_java(cls, java_stage):
    +        """
    +        Given a Java Pipeline, create and return a Python wrapper of it.
    +        Used for ML persistence.
    +        """
    +        # Create a new instance of this stage.
    +        py_stage = cls()
    +        # Load information from java_stage to the instance.
    +        py_stages = [JavaWrapper._transfer_stage_from_java(s) for s in java_stage.getStages()]
    +        py_stage.setStages(py_stages)
    +        py_stage._resetUid(java_stage.uid())
    +        return py_stage
    +
    +    def _transfer_stage_to_java(self):
    +        """
    +        Transfer this instance to a Java object.  Used for ML persistence.
    --- End diff --
    
    Done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-13951][ML][PYTHON] Nested Pipeline pers...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11866#issuecomment-199488291
  
    **[Test build #53711 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53711/consoleFull)** for PR 11866 at commit [`1c070e7`](https://github.com/apache/spark/commit/1c070e738236407559c00360eea1012880229076).
     * This patch **fails PySpark unit tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class ElementwiseProduct(JavaTransformer, HasInputCol, HasOutputCol, JavaMLReadable,`
      * `class HashingTF(JavaTransformer, HasInputCol, HasOutputCol, HasNumFeatures, JavaMLReadable,`
      * `class PolynomialExpansion(JavaTransformer, HasInputCol, HasOutputCol, JavaMLReadable,`
      * `class JavaMLWritable(MLWritable):`
      * `class JavaMLReader(MLReader):`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org