[jira] [Commented] (SPARK-14242) avoid too many copies in network when a network frame is large

["Apache Spark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-14242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15216166#comment-15216166 ] 

Apache Spark commented on SPARK-14242:
--------------------------------------

User 'liyezhang556520' has created a pull request for this issue:
https://github.com/apache/spark/pull/12038

> avoid too many copies in network when a network frame is large
> --------------------------------------------------------------
>
>                 Key: SPARK-14242
>                 URL: https://issues.apache.org/jira/browse/SPARK-14242
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 1.6.0, 1.6.1, 2.0.0
>            Reporter: Zhang, Liye
>            Priority: Critical
>
> when a shuffle block size is huge, say a large array (array size more than 128MB), there will be performance issue for getting remote blocks. This is because network frame size is large, and when we are using a composite buffer, which will consolidate when the components number reaches maximum components number (default is 16) in netty underlying, performance issue will occurs. There will be too many memory copies inside netty's *compositeBuffer*.
> How to reproduce:
> {code}
> sc.parallelize(Array(1,2,3),3).mapPartitions(a=>Array(new Array[Double](1024 * 1024 * 50)).iterator).reduce((a,b)=> a).length
> {code}
> In this case, the serialized result size of each task is about 400MB, the result will be transferred to driver as *indirectResult*. We can see after the data transferred to driver, on driver side there will still need a lot of time to process and the 3 CPUs (in this case, parallelism is 3) are fully utilized with system call very high. And this processing time is calculated as result getting time on webUI.
> Such cases are very common in ML applications, which will return a large array from each executor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org