You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/06/27 15:20:38 UTC
[GitHub] [spark] kuwii opened a new pull request, #37007: [SPARK-38056][Web UI][WIP] Use same condition in history server page and API to filter applications
kuwii opened a new pull request, #37007:
URL: https://github.com/apache/spark/pull/37007
### What changes were proposed in this pull request?
Updated REST API `/api/v1/applications`, to use the same condition as history server page to filter completed/incomplete applications.
### Why are the changes needed?
When opening summary page, history server follows this logic:
If there's completed/incomplete application, page will add script in response, using AJAX to call the REST API to get the filtered list.
- If there's no such application, page will only return a message telling nothing found.
- Issue is that page and REST API are using different conditions to filter applications. In `HistoryPage`, an application is considered as completed as long as the last attempt is completed. But in `ApplicationListResource`, all attempts should be completed. This brings inconsistency and will cause issue in a corner case.
In driver, event queues have capacity to protect memory. When there's too many events, some of them will be dropped and the event log file will be incomplete. For an application with multiple attempts, there's possibility that the last attempt is completed, but the previous attempts is considered as incomplete due to loss of application end event.
For this type of application, page thinks it is completed, but the API thinks it is still running. When opening summary page:
- When checking completed applications, page will call script, but API returns nothing.
- When checking incomplete applications, page returns nothing.
So the user won't be able to see this app in history server.
### Does this PR introduce _any_ user-facing change?
Yes, there will be a change on `/api/v1/applications` API.
For application mentioned above, previously it is considered as running. After the change it is considered as completed. So the result will be different using same filter.
But I think this change should be OK. Because attempts are executed sequentially and incrementally. So if an attempt with bigger ID is completed, the previous attempts should also be completed.
### How was this patch tested?
WIP
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kuwii commented on pull request #37007: [SPARK-39620][Web UI][WIP] Use same condition in history server page and API to filter applications
Posted by GitBox <gi...@apache.org>.
kuwii commented on PR #37007:
URL: https://github.com/apache/spark/pull/37007#issuecomment-1167531301
Linked to wrong JIRA. Will recreate a new one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kuwii closed pull request #37007: [SPARK-39620][Web UI][WIP] Use same condition in history server page and API to filter applications
Posted by GitBox <gi...@apache.org>.
kuwii closed pull request #37007: [SPARK-39620][Web UI][WIP] Use same condition in history server page and API to filter applications
URL: https://github.com/apache/spark/pull/37007
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org