Web UI and History Server are Inconsistent; Web UI sometimes cannot process logs

[jcai <jo...@yale.edu>]

I am running a very simple program (WordCount) on Spark stand-alone mode. I
find that when I examine the
web UI, a couple bugs arise:

1. There is a discrepancy between the number denoting the duration of the
application when I run the history server and the number given by the web UI
(default address is master:8080). I checked more specific details, including
task and stage durations (when clicking on the application), and these
appear to be the same for both avenues.

2. Sometimes the web UI on master:8080 is unable to display more specific
information for an application that has finished (when clicking on the
application), even when there is a log file in the appropriate directory.
But when the history server is opened, it is able to read this file and
output information.

Any ideas on how to approach these?

I am trying to do accurate performance measurements on Spark workloads. If
the application durations on the web UI aren't accurate (or consistent) for
such simple workloads, 
that seems problematic.

I also opened up a  JIRA report
<https://issues.apache.org/jira/browse/SPARK-8512?jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Major%20ORDER%20BY%20key%20DESC>  
on the issue.

-- Jonathon





--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Web-UI-and-History-Server-are-Inconsistent-Web-UI-sometimes-cannot-process-logs-tp12821.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org