You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Olivier Toupin <ol...@gmail.com> on 2015/01/05 21:25:12 UTC
Spark UI history job duration is wrong
Hello,
I'm using Spark 1.2.0 and when running an application, if I go into the UI
and then in the job tab ("/jobs/") the jobs duration are relevant and the
posted durations looks ok.
However when I open the history ("history/app-<xyz>/jobs/") for that job,
the duration are wrong showing milliseconds instead of the relevant job
time. The submitted time for each job (except maybe the first) is different
also.
The stage tab is unaffected and show the correct duration for each stages in
both mode.
Should I open a bug?
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-UI-history-job-duration-is-wrong-tp10010.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Spark UI history job duration is wrong
Posted by Patrick Wendell <pw...@gmail.com>.
Thanks for reporting this - it definitely sounds like a bug. Please
open a JIRA for it. My guess is that we define the start or end time
of the job based on the current time instead of looking at data
encoded in the underlying event stream. That would cause it to not
work properly when loading from historical data.
- Patrick
On Mon, Jan 5, 2015 at 12:25 PM, Olivier Toupin
<ol...@gmail.com> wrote:
> Hello,
>
> I'm using Spark 1.2.0 and when running an application, if I go into the UI
> and then in the job tab ("/jobs/") the jobs duration are relevant and the
> posted durations looks ok.
>
> However when I open the history ("history/app-<xyz>/jobs/") for that job,
> the duration are wrong showing milliseconds instead of the relevant job
> time. The submitted time for each job (except maybe the first) is different
> also.
>
> The stage tab is unaffected and show the correct duration for each stages in
> both mode.
>
> Should I open a bug?
>
>
>
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-UI-history-job-duration-is-wrong-tp10010.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org