You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Bibin A Chundatt (JIRA)" <ji...@apache.org> on 2018/09/08 14:21:00 UTC
[jira] [Comment Edited] (YARN-7599) [GPG] ApplicationCleaner in
Global Policy Generator
[ https://issues.apache.org/jira/browse/YARN-7599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16608065#comment-16608065 ]
Bibin A Chundatt edited comment on YARN-7599 at 9/8/18 2:20 PM:
----------------------------------------------------------------
Thank you [~botong] for working on this patch
IIUC as per the current implementation applicationCleaner depends on only expiry time + finished state.I think its better to have a Policy based implementation which defines how store is cleaned.
Expiry time can be one type of policy.
If we look at Resource manager side we have 2 clean configuration, one for in-memory and state store.
Both are based on finish order or apps and count.
Expiry time based implementation can create a race condition , were application is available in RM but not in Router and appReport could fail.
Thoughts??
was (Author: bibinchundatt):
Thank you [~botong] for working on this patch
IIUC as per the current implementation applicationCleaner depends on only expiry time + finished state.
At cleaner since, i think its better to have a Policy based implementation which defines how store is cleaned.
Expiry time can be one type of policy.
If we look at Resource manager side we have 2 clean configuration, one for in-memory and state store.
Both are based on finish order or apps and count.
Expiry time based implementation can create a race condition , were application is available in RM but not in Router and appReport could fail.
Thoughts??
> [GPG] ApplicationCleaner in Global Policy Generator
> ---------------------------------------------------
>
> Key: YARN-7599
> URL: https://issues.apache.org/jira/browse/YARN-7599
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Botong Huang
> Assignee: Botong Huang
> Priority: Minor
> Labels: federation, gpg
> Attachments: YARN-7599-YARN-7402.v1.patch, YARN-7599-YARN-7402.v2.patch
>
>
> In Federation, we need a cleanup service for StateStore as well as Yarn Registry. For the former, we need to remove old application records. For the latter, failed and killed applications might leave records in the Yarn Registry (see YARN-6128). We plan to do both cleanup work in ApplicationCleaner in GPG
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org