You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by jkbradley <gi...@git.apache.org> on 2016/01/27 06:21:02 UTC

[GitHub] spark pull request: [ML][PYTHON][SPARK-12834] Change ser/de of Jav...

GitHub user jkbradley opened a pull request:

    https://github.com/apache/spark/pull/10941

    [ML][PYTHON][SPARK-12834] Change ser/de of JavaArray and JavaList

    Backport of SPARK-12834 for branch-1.6
    
    Original PR: https://github.com/apache/spark/pull/10772
    
    Original commit message:
    We use `SerDe.dumps()` to serialize `JavaArray` and `JavaList` in `PythonMLLibAPI`, then deserialize them with `PickleSerializer` in Python side. However, there is no need to transform them in such an inefficient way. Instead of it, we can use type conversion to convert them, e.g. `list(JavaArray)` or `list(JavaList)`. What's more, there is an issue to Ser/De Scala Array as I said in https://issues.apache.org/jira/browse/SPARK-12780

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jkbradley/spark yinxusen-SPARK-12834-1.6

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/10941.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #10941
    
----
commit 4a621bc46545b6a1a80b9925ce10e2f2a4ccf384
Author: Xusen Yin <yi...@gmail.com>
Date:   2016-01-26T06:41:52Z

    [SPARK-12834] Change ser/de of JavaArray and JavaList
    
    https://issues.apache.org/jira/browse/SPARK-12834
    
    We use `SerDe.dumps()` to serialize `JavaArray` and `JavaList` in `PythonMLLibAPI`, then deserialize them with `PickleSerializer` in Python side. However, there is no need to transform them in such an inefficient way. Instead of it, we can use type conversion to convert them, e.g. `list(JavaArray)` or `list(JavaList)`. What's more, there is an issue to Ser/De Scala Array as I said in https://issues.apache.org/jira/browse/SPARK-12780
    
    Author: Xusen Yin <yi...@gmail.com>
    
    Closes #10772 from yinxusen/SPARK-12834.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [ML][PYTHON][SPARK-12834][BACKPORT] Change ser...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/10941#issuecomment-175411066
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [ML][PYTHON][SPARK-12834][BACKPORT] Change ser...

Posted by yinxusen <gi...@git.apache.org>.

Github user yinxusen commented on the pull request:

    https://github.com/apache/spark/pull/10941#issuecomment-175475755
  
    Sure, LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [ML][PYTHON][SPARK-12834][BACKPORT] Change ser...

Posted by jkbradley <gi...@git.apache.org>.

Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/10941#issuecomment-175479263
  
    OK thanks!  merging with branch-1.6


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [ML][PYTHON][SPARK-12834][BACKPORT] Change ser...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10941#issuecomment-175411062
  
    **[Test build #50172 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50172/consoleFull)** for PR 10941 at commit [`4a621bc`](https://github.com/apache/spark/commit/4a621bc46545b6a1a80b9925ce10e2f2a4ccf384).
     * This patch **fails to build**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [ML][PYTHON][SPARK-12834][BACKPORT] Change ser...

Posted by marmbrus <gi...@git.apache.org>.

Github user marmbrus commented on the pull request:

    https://github.com/apache/spark/pull/10941#issuecomment-178731963
  
    Can you close this PR? (PRs not against master don't auto close)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [ML][PYTHON][SPARK-12834][BACKPORT] Change ser...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10941#issuecomment-175442210
  
    **[Test build #2465 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2465/consoleFull)** for PR 10941 at commit [`4a621bc`](https://github.com/apache/spark/commit/4a621bc46545b6a1a80b9925ce10e2f2a4ccf384).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [ML][PYTHON][SPARK-12834][BACKPORT] Change ser...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/10941#issuecomment-175465553
  
    **[Test build #2465 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2465/consoleFull)** for PR 10941 at commit [`4a621bc`](https://github.com/apache/spark/commit/4a621bc46545b6a1a80b9925ce10e2f2a4ccf384).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org