You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "wy (Jira)" <ji...@apache.org> on 2022/07/08 04:57:00 UTC
[jira] [Updated] (SPARK-39620) History server page and API are using inconsistent conditions to filter running applications
[ https://issues.apache.org/jira/browse/SPARK-39620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
wy updated SPARK-39620:
-----------------------
Fix Version/s: (was: 3.3.0)
(was: 3.2.1)
> History server page and API are using inconsistent conditions to filter running applications
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-39620
> URL: https://issues.apache.org/jira/browse/SPARK-39620
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 3.2.1, 3.3.0
> Reporter: wy
> Priority: Major
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> When opening summary page, history server follows this logic:
> * If there's completed/incomplete application, page will add script in response, using AJAX to call the REST API to get the filtered list.
> * If there's no such application, page will only return a message telling nothing found.
> Issue is that page and REST API are using different conditions to filter applications. In HistoryPage, an application is considered as completed as long as the last attempt is completed. But in ApplicationListResource, all attempts should be completed. This brings inconsistency and will cause issue in a corner case.
> In driver, event queues have capacity to protect memory. When there's too many events, some of them will be dropped and the event log file will be incomplete. For an application with multiple attempts, there's possibility that the last attempt is completed, but the previous attempts is considered as incomplete due to loss of application end event.
> For this type of application, page thinks it is completed, but the API thinks it is still running. When opening summary page:
> * When checking completed applications, page will call script, but API returns nothing.
> * When checking incomplete applications, page returns nothing.
> So the user won't be able to see this app in history server.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org