[jira] [Resolved] (SPARK-8667) Improve Spark UI behavior at scale

["Patrick Wendell (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-8667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Patrick Wendell resolved SPARK-8667.
------------------------------------
    Resolution: Duplicate

> Improve Spark UI behavior at scale
> ----------------------------------
>
>                 Key: SPARK-8667
>                 URL: https://issues.apache.org/jira/browse/SPARK-8667
>             Project: Spark
>          Issue Type: Improvement
>          Components: Web UI
>            Reporter: Patrick Wendell
>            Assignee: Shixiong Zhu
>
> This is a parent ticket and we can create child tickets when solving specific issues. The main problem I would like to solve is the fact that the Spark UI has issues at very large scale.
> The worst issue is when there is a stage page with more than a few thousand tasks. In this case:
> 1. The page itself is very slow to load and becomes unresponsive with huge number of tasks.
> 2. The Scala XML output can become so large that it crashes the driver program due to OOM for a page with a huge number of tasks.
> I am not sure if (1) is caused by javascript slowness, or maybe just the raw amount of data sent over the wire. If it is the latter, it might be possible to add compression to the HTTP payload to help improve load time.
> It would be nice to reproduce+investigate these issues further and create specific sub tasks to improve them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org