You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "duyanghao (JIRA)" <ji...@apache.org> on 2017/07/08 02:25:00 UTC
[jira] [Updated] (SPARK-21348) Spark history Server load too slow
when eventlog is large
[ https://issues.apache.org/jira/browse/SPARK-21348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
duyanghao updated SPARK-21348:
------------------------------
Description:
There is a eventlog file in hdfs as below:
```
-rwxrwx--- 3 root supergroup 9634238709 2017-07-08 04:37 hdfs://xxxxx/shared/spark-logs/xxxxx/xxxx.lz4
```
and i start a spark history server to loading that eventlog file,but it is too slow to get spark web ui through that spark history server.
it has requested for several hours but nothing about spark application web ui appears in the browser
And the cpu usage of spark history server is 105.3%
total used-memory in host is 9G
The host has 48 Cores+130G Memory,enough Computing resources for spark history server.
was:
There is a eventlog file in hdfs as below:
```
-rwxrwx--- 3 root supergroup 9634238709 2017-07-08 04:37 hdfs://xxxxx/shared/spark-logs/xxxxx/xxxx.lz4
```
and i start a spark history server to loading that eventlog file,but it is too slow to get spark web ui through that spark history server.
it has requested for several hours but nothing about spark application web ui appears in the browser
And the cpu usage of spark history server is 105.3%
total used-memory in host is 9G
The host have 48 Cores+130G Memory,enough Computing resources for spark history server.
> Spark history Server load too slow when eventlog is large
> ---------------------------------------------------------
>
> Key: SPARK-21348
> URL: https://issues.apache.org/jira/browse/SPARK-21348
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 2.1.0
> Reporter: duyanghao
> Priority: Critical
>
> There is a eventlog file in hdfs as below:
> ```
> -rwxrwx--- 3 root supergroup 9634238709 2017-07-08 04:37 hdfs://xxxxx/shared/spark-logs/xxxxx/xxxx.lz4
> ```
> and i start a spark history server to loading that eventlog file,but it is too slow to get spark web ui through that spark history server.
> it has requested for several hours but nothing about spark application web ui appears in the browser
> And the cpu usage of spark history server is 105.3%
> total used-memory in host is 9G
> The host has 48 Cores+130G Memory,enough Computing resources for spark history server.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org