[jira] [Commented] (SPARK-2017) web ui stage page becomes unresponsive when the number of tasks is large

["Mridul Muralidharan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14019394#comment-14019394 ] 

Mridul Muralidharan commented on SPARK-2017:
--------------------------------------------

Currently, for our jobs, I run with spark.ui.retainedStages=3 (so that there is some visibility into past stages) : this is to prevent OOM's in the master when number of tasks per stage is not low (50k for example is not very high imo)

The stage details UI becomes very sluggish to pretty much unresponsive for our tasks where tasks > 30k ... though that might also be a browser issue (firefox/chrome) ?

> web ui stage page becomes unresponsive when the number of tasks is large
> ------------------------------------------------------------------------
>
>                 Key: SPARK-2017
>                 URL: https://issues.apache.org/jira/browse/SPARK-2017
>             Project: Spark
>          Issue Type: Sub-task
>            Reporter: Reynold Xin
>              Labels: starter
>
> {code}
> sc.parallelize(1 to 1000000, 1000000).count()
> {code}
> The above code creates one million tasks to be executed. The stage detail web ui page takes forever to load (if it ever completes).
> There are again a few different alternatives:
> 0. Limit the number of tasks we show.
> 1. Pagination
> 2. By default only show the aggregate metrics and failed tasks, and hide the successful ones.



--
This message was sent by Atlassian JIRA
(v6.2#6252)