[jira] [Resolved] (SPARK-27394) The staleness of UI may last minutes or hours when no tasks start or finish

["Marcelo Vanzin (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-27394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marcelo Vanzin resolved SPARK-27394.
------------------------------------
       Resolution: Fixed
    Fix Version/s: 3.0.0

> The staleness of UI may last minutes or hours when no tasks start or finish
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-27394
>                 URL: https://issues.apache.org/jira/browse/SPARK-27394
>             Project: Spark
>          Issue Type: Bug
>          Components: Web UI
>    Affects Versions: 2.4.0, 2.4.1
>            Reporter: Shixiong Zhu
>            Assignee: Shixiong Zhu
>            Priority: Major
>             Fix For: 3.0.0
>
>
> Run the following codes on a cluster that has at least 2 cores.
> {code}
> sc.makeRDD(1 to 1000, 1000).foreach { i =>
>   Thread.sleep(300000)
> }
> {code}
> The jobs page will just show one running task.
> This is because when the second task event calls "AppStatusListener.maybeUpdate" for a job, it will just ignore since the gap between two events is smaller than `spark.ui.liveUpdate.period`.
> After the second task event, in the above case, because there won't be any other task events, the Spark UI will be always stale until the next task event gets fired (after 300 seconds).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org