[jira] [Resolved] (SPARK-5023) In Web UI job history, the total job duration is incorrect (much smaller than the sum of its stages)

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-5023.
------------------------------
    Resolution: Duplicate

Absent any follow-up to the contrary, this does indeed sound like the same issue as SPARK-4836

> In Web UI job history, the total job duration is incorrect (much smaller than the sum of its stages)
> ----------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-5023
>                 URL: https://issues.apache.org/jira/browse/SPARK-5023
>             Project: Spark
>          Issue Type: Bug
>          Components: Web UI
>    Affects Versions: 1.1.1, 1.2.0
>         Environment: Amazon EC2 AMI r3.2xlarge, cluster of 20 to 50 nodes, running the ec2 provided scripts to create. 
>            Reporter: Eran Medan
>
> I'm running a long process using Spark + Graph and things look good on the 4040 job status UI, but when the job is done, when going to the history then the job total duration is much, much smaller than the total of its stages.
> The way I set logs up is this:
>   val homeDir = sys.props("user.home")
>   val logsPath = new File(homeDir,"sparkEventLogs")
>   val conf = new SparkConf().setAppName("...")
>   conf.set("spark.eventLog.enabled", "true").set("spark.eventLog.dir", logsPath.getCanonicalPath)
> for example job ID X - duration 0.2 s, but when I click the job and look at its stages, the sum of their duration is more than 15 minutes!
> (before the job was over, in the 4040 job status, the job duration was correct, it is only incorrect when its done and going to the logs) 
> I hope I didn't configure something because I was very surprised no one reported it yet (I searched, but perhaps I missed it) 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org