[jira] [Closed] (SPARK-6728) Improve performance of py4j for large bytearray

["Davies Liu (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Davies Liu closed SPARK-6728.
-----------------------------
    Resolution: Won't Fix

This can not be fixed without changes in Py4j, but this is not in the roadmap of Py4j yet.

> Improve performance of py4j for large bytearray
> -----------------------------------------------
>
>                 Key: SPARK-6728
>                 URL: https://issues.apache.org/jira/browse/SPARK-6728
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>    Affects Versions: 1.3.0
>            Reporter: Davies Liu
>            Priority: Critical
>
> PySpark relies on py4j to transfer function arguments and return between Python and JVM, it's very slow to pass a large bytearray (larger than 10M). 
> In MLlib, it's possible to have a Vector with more than 100M bytes, which will need few GB memory, may crash.
> The reason is that py4j use text protocol, it will encode the bytearray as base64, and do multiple string concat. 
> Binary will help a lot, create a issue for py4j: https://github.com/bartdag/py4j/issues/159



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org