You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Tao Yang (JIRA)" <ji...@apache.org> on 2019/03/27 04:13:00 UTC
[jira] [Updated] (YARN-9413) Queue resource leak after app fail for
CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tao Yang updated YARN-9413:
---------------------------
Attachment: YARN-9413.001.patch
> Queue resource leak after app fail for CapacityScheduler
> --------------------------------------------------------
>
> Key: YARN-9413
> URL: https://issues.apache.org/jira/browse/YARN-9413
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
> Affects Versions: 3.1.2
> Reporter: Tao Yang
> Assignee: Tao Yang
> Priority: Major
> Attachments: YARN-9413.001.patch
>
>
> To reproduce this problem:
> # Submit an app which is configured to keep containers across app attempts and should fail after AM finished at first time (am-max-attempts=1).
> # App is started with 2 containers running on NM1 node.
> # Fail the AM of the application with PREEMPTED exit status which should not count towards max attempt retry but app will fail immediately.
> # Used resource of this queue leaks after app fail.
> The root cause is the inconsistency of handling app attempt failure between RMAppAttemptImpl$BaseFinalTransition#transition and RMAppImpl$AttemptFailedTransition#transition:
> # After app fail, RMAppFailedAttemptEvent will be sent in RMAppAttemptImpl$BaseFinalTransition#transition, if exit status of AM container is PREEMPTED/ABORTED/DISKS_FAILED/KILLED_BY_RESOURCEMANAGER, it will not count towards max attempt retry, so that it will send AppAttemptRemovedSchedulerEvent with keepContainersAcrossAppAttempts=true and RMAppFailedAttemptEvent with transferStateFromPreviousAttempt=true.
> # RMAppImpl$AttemptFailedTransition#transition handle RMAppFailedAttemptEvent and will fail the app if its max app attempts is 1.
> # CapacityScheduler handles AppAttemptRemovedSchedulerEvent in CapcityScheduler#doneApplicationAttempt, it will skip killing and calling completion process for containers belong to this app, so that queue resource leak happens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org