You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Patrick Wendell (JIRA)" <ji...@apache.org> on 2014/12/04 20:01:12 UTC
[jira] [Comment Edited] (SPARK-4740) Netty's network bandwidth is
much lower than NIO in spark-perf and Netty takes longer running time
[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234479#comment-14234479 ]
Patrick Wendell edited comment on SPARK-4740 at 12/4/14 7:00 PM:
-----------------------------------------------------------------
Thanks for reporting this. We've run a bunch of tests and never found netty to be slower than NIO, so this is a helpful piece of feedback. One unique thing about your environment is that you have 48 cores per node. Do you observe the same effect if you limit the parallelism on each node to fewer cores?
/cc [~adav] [~rxin]
was (Author: pwendell):
Thanks for reporting this. We've run a bunch of tests and never found netty to be slower than NIO, so this is a helpful piece of feedback. One unique thing about your environment is that you have 48 cores per node. Do you observe the same effect if you limit the parallelism on each node to fewer cores?
> Netty's network bandwidth is much lower than NIO in spark-perf and Netty takes longer running time
> --------------------------------------------------------------------------------------------------
>
> Key: SPARK-4740
> URL: https://issues.apache.org/jira/browse/SPARK-4740
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle, Spark Core
> Affects Versions: 1.2.0
> Reporter: Zhang, Liye
> Attachments: Spark-perf Test Report.pdf
>
>
> When testing current spark master (1.3.0-snapshot) with spark-perf (sort-by-key, aggregate-by-key, etc), Netty based shuffle transferService takes much longer time than NIO based shuffle transferService. The network throughput of Netty is only about half of that of NIO.
> We tested with standalone mode, and the data set we used for test is 20 billion records, and the total size is about 400GB. Spark-perf test is Running on a 4 node cluster with 10G NIC, 48 cpu cores per node and each executor memory is 64GB. The reduce tasks number is set to 1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org