You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Patrick Wendell (JIRA)" <ji...@apache.org> on 2015/01/16 07:22:34 UTC

[jira] [Commented] (SPARK-5216) Spark Ui should report estimated time remaining for each stage.

    [ https://issues.apache.org/jira/browse/SPARK-5216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279863#comment-14279863 ] 

Patrick Wendell commented on SPARK-5216:
----------------------------------------

This has been proposed before, but in the past we decided not to do it. Trying to extrapolate the finish time of a stage accurately is basically impossible since in many workloads stragglers dominate the total response time. The conclusion was that it was better to give no estimate rather than one which is likely to be misleading. 

> Spark Ui should report estimated time remaining for each stage.
> ---------------------------------------------------------------
>
>                 Key: SPARK-5216
>                 URL: https://issues.apache.org/jira/browse/SPARK-5216
>             Project: Spark
>          Issue Type: Wish
>          Components: Spark Core, Web UI
>    Affects Versions: 1.3.0
>            Reporter: Prashant Sharma
>            Assignee: Prashant Sharma
>
> Per stage feedback on estimated remaining time can help user get a grasp on how much time the job is going to take. This will only require changes on the UI/JobProgressListener side of code since we already have most of the information needed. 
> In the initial cut, plan is to estimate time based on statistics of running job i.e. average time taken by each task and number of task per stage. This will makes sense when jobs are long. And then if this makes sense, then more heuristics can be added like projected time saved if the rdd is cached and so on. 
> More precise details will come as this evolves. In the meantime thoughts on alternate ways and suggestion on usefulness are welcome.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org