You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Rayman (JIRA)" <ji...@apache.org> on 2019/04/01 21:34:00 UTC
[jira] [Comment Edited] (YARN-9192) Deletion Taks will be picked up
to delete running containers
[ https://issues.apache.org/jira/browse/YARN-9192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807102#comment-16807102 ]
Rayman edited comment on YARN-9192 at 4/1/19 9:33 PM:
------------------------------------------------------
I'm observing a similar issue, when running Samza over YARN.
When bouncing an NM, the NM being killed writes LevelDB state for the deletion-service to act on.
The "new" NM reads it and acts upon it, but ends up deleting directories for running containers.
This happens when containers are long-running, and are placed on a fixed host.
I also observed this in the log
*[INFO] [shutdown-hook-0] containermanager.ContainerManagerImpl.cleanUpApplicationsOnNMShutDown(ContainerManagerImpl.java:718) - Waiting for Applications to be Finished*
was (Author: rayman7718):
I'm observing a similar issue, when running Samza over YARN.
When bouncing an NM, the NM being killed writes LevelDB state for the deletion-service to act on.
The "new" NM reads it and acts upon it, but ends up deleting directories for running containers.
This happens when containers are long-running, and are placed on a fixed host.
> Deletion Taks will be picked up to delete running containers
> ------------------------------------------------------------
>
> Key: YARN-9192
> URL: https://issues.apache.org/jira/browse/YARN-9192
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications
> Affects Versions: 2.9.1
> Reporter: Sihai Ke
> Priority: Major
>
> I suspect there is a bug in Yarn deletion task service, below is my repo steps:
> # First let's set yarn.nodemanager.delete.debug-delay-sec=3600, that means when the app finished, the Binary/container folder will be deleted after 3600 seconds.
> # when the application App1 (long running service) is running on machine machine1, and machine1 shutdown, ContainerManagerImpl#serviceStop() will be called -> ContainerManagerImpl#cleanUpApplicationsOnNMShutDown, and ApplicationFinishEvent will be sent, and then some delection tasks will be created, but be stored in DB and will be picked up to execute 3600 seconds.
> # 100 seconds later, machine1 comes back, and the same app is assigned to run this this machine, container created and works well.
> # then deleting task created in step 2 will be picked up to delete containers created in step 3 later.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org