You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Peter Liu (JIRA)" <ji...@apache.org> on 2016/06/30 20:42:10 UTC
[jira] [Comment Edited] (SPARK-16332) the history server of
spark2.0-preview (may-24 build) consumes more than 1000% cpu
[ https://issues.apache.org/jira/browse/SPARK-16332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15357811#comment-15357811 ]
Peter Liu edited comment on SPARK-16332 at 6/30/16 8:41 PM:
------------------------------------------------------------
I think this is likely related to issue "SPARK-16333" due the huge amount of data per each run (5Gb per run). when the cluster is idle, it does the initialization and consumes 1000% cpu; when under load, the cpu consumption goes above 2000% (per linux top command);
the net is: for the same scenario (exactly the same amount of data and the source code), when running spark 1.6.1, the jvm of the spark history server only consumes ~30% under load (where is spark event data amount is also much smaller as said in the other ticket).
the fix of this issue can be made dependent of the fix of "SPARK-16333", I think.
thanks ...
Peter
was (Author: petergangliu):
I think this is likely related to issue "SPARK-16333" due the huge amount of data per each run (5Gb per run). when the cluster is idle, it does the initialization and consumes 1000% cpu; when under load, the cpu consumption goes above 2000% (per linux top command);
the net is: for the same scenario (exactly the same amount of data and the source code), when running spark 1.6.1, the jvm of the spark history server only consumes ~30% under load (where is data amount is also much smaller).
the fix of this issue can be made dependent of the fix of "SPARK-16333", I think.
thanks ...
Peter
> the history server of spark2.0-preview (may-24 build) consumes more than 1000% cpu
> ----------------------------------------------------------------------------------
>
> Key: SPARK-16332
> URL: https://issues.apache.org/jira/browse/SPARK-16332
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.0.0
> Environment: this is seen on both x86 (Intel(R) Xeon(R), E5-2699 ) and ppc platform IBM Power8 Habanero (Model: 8348-21C), Red Hat Enterprise Linux Server release 7.2 (Maipo), Spark2.0.0-preview (May-24, 2016)
> Reporter: Peter Liu
>
> the JVM instance of the Spark history server of spark2.0-preview (may-24 build) consumes more than 1000% cpu without the spark standalone cluster (of 6 nodes) being under load.
> When under load (1TB input data for a SQL query scenario), the JVM instance of the Spark history server of spark2.0-preview consumes 2000% cpu (as seen with "top" on linux 3.10)
> Note: can't see a proper Component selection here, surely not a Web GUI issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org