[jira] [Resolved] (SPARK-14106) history server application cache doesn't detect that apps are completed

["Steve Loughran (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-14106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran resolved SPARK-14106.
------------------------------------
    Resolution: Not A Bug

Understanding this more, its coming from the history provider's view of completeness may not be that of the UI. That is: the UI knows its finished if there's an app end event processed. If not: incomplete.

However, the AppHistoryInfo structure's completed bit may, in some back ends, use the state of the running scheduler to determine complete/incomplete. That is: if its not running or not in the history, it is implicitly "complete". In such a situation the history UI will keep probing the app on every GET to its UI, as it doesn't consider it as finished...there will be checks.

what to do: make sure your probe is lightweight and doesn't over-report stage changes.

> history server application cache doesn't detect that apps are completed
> -----------------------------------------------------------------------
>
>                 Key: SPARK-14106
>                 URL: https://issues.apache.org/jira/browse/SPARK-14106
>             Project: Spark
>          Issue Type: Bug
>          Components: Web UI
>    Affects Versions: 2.0.0
>            Reporter: Steve Loughran
>            Priority: Minor
>
> The cache in SPARK-7889 hooks up a filter to the web UI of incomplete apps, triggering a probe to see if the app is out of date.
> It looks like the filter is being hooked up to completed apps too, that is: the {{ApplicationCache}} isn't correctly detecting when apps are finished.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org