You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:23:21 UTC
[jira] [Updated] (SPARK-10187) Sometimes Web UI reports Application
history not found
[ https://issues.apache.org/jira/browse/SPARK-10187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-10187:
---------------------------------
Labels: bulk-closed (was: )
> Sometimes Web UI reports Application history not found
> ------------------------------------------------------
>
> Key: SPARK-10187
> URL: https://issues.apache.org/jira/browse/SPARK-10187
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 1.4.1
> Environment: CentOS Linux release 7.1.1503 (Core)
> Linux 3.10.0-229.4.2.el7.x86_64 #1 SMP Wed May 13 10:06:09 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
> Reporter: Rares Vernica
> Priority: Minor
> Labels: bulk-closed
>
> On the Web UI home page {{http://localhost:8080/}} in the list of Completed Applications clicking on the Name of a job shows the history of that job. This works for most of the time. For some jobs, Spark returns:
> {quote}
> Application history not found (app-20150824104953-0018)
> No event logs found for application bid-00001-ph-1 in file:///data/disk1/spark/events. Did you specify the correct logging directory?
> {quote}
> All these jobs have the FINISHED state. {{spark-defaults.conf}} contains
> {quote}
> spark.eventLog.dir file:///data/disk1/spark/events
> spark.history.fs.logDirectory file:///data/disk1/spark/events
> {quote}
> {{/data/disk1/spark/events}} is a normal directory on the disk. This works fine for some jobs but not others (even jobs on the same Spark session).
> The Spark History UI shows correctly the history of the jobs that the Web UI cannot find the logs for. So, I believe the log directory is fine.
> Here is the HTML snippet from the Web UI home page, under the Completed Applications section. Notice how the UI finds the history for some jobs, but not others:
> {quote}
> ...
> <a href="app?appId=app-20150824110804-0019">app-20150824110804-0019</a>
> ...
> <a href="/history/app-20150824110804-0019">bid-00001-ph-2</a>
> ...
> <a href="app?appId=app-20150824104953-0018">app-20150824104953-0018</a>
> ...
> <a href="/history/not-found?msg=No+event+logs+found+for+application+bid-00001-ph-1+in+file%3A%2F%2F%2Fdata%2Fdisk1%2Fspark%2Fevents.+Did+you+specify+the+correct+logging+directory%3F&title=Application history not found (app-20150824104953-0018)">bid-00001-ph-1</a>
> ...
> <a href="app?appId=app-20150824104907-0017">app-20150824104907-0017</a>
> ...
> <a href="/history/not-found?msg=No+event+logs+found+for+application+bid-00001-ph-2+in+file%3A%2F%2F%2Fdata%2Fdisk1%2Fspark%2Fevents.+Did+you+specify+the+correct+logging+directory%3F&title=Application history not found (app-20150824104907-0017)">bid-00001-ph-2</a>
> ...
> <a href="app?appId=app-20150824103131-0016">app-20150824103131-0016</a>
> ...
> <a href="/history/app-20150824103131-0016">bid-00001-ph-1</a>
> {quote}
> These are sequential jobs and all the necessary files and directories in {{/data/disk1/spark/events}} and present and accessible and Spark History UI shows the history for all these jobs.
> I looked around JIRA for similar issues, I think this is related to SPARK-6107 and SPARK-6950 but not a duplicate. The jobs for which these manifests are all FINISHED for some time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org