You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/08/18 23:51:00 UTC

[jira] [Commented] (YARN-11856) DOWNLOADING resources unlock and cleanup is interrupted when killing a container that is localizing

    [ https://issues.apache.org/jira/browse/YARN-11856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18105722#comment-18105722 ] 

ASF GitHub Bot commented on YARN-11856:
---------------------------------------

eubnara opened a new pull request, #8691:
URL: https://github.com/apache/hadoop/pull/8691

   
   <!--
     Thanks for sending a pull request!
       1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
       2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
   -->
   
   ### Description of PR
   
   When a container is killed while it is localizing, the `LocalizerRunner`
   thread is interrupted. In the `finally` block of `LocalizerRunner.run()`,
   dispatching the `ContainerResourceFailedEvent` then throws
   `YarnRuntimeException(InterruptedException)`, which skips everything after
   it: resources left in DOWNLOADING state are never unlocked, and the
   deletion tasks for the localization dirs, the `_tmp` download dirs and the
   nmPrivate token file are never scheduled.
   
   This PR revives the fix from #7893 by @zheng-weihao (stale-closed after
   100 days of inactivity), unchanged: the event dispatch is wrapped in
   try/catch so the cleanup below it always runs. The remaining review
   comment on #7893 (dropping `FSError` from the catch list) had already been
   addressed in its final revision. Credit to the original author is kept in
   the commit message.
   
   What this PR adds on top of #7893 is a unit test, which was the other
   blocker ("no new or modified tests").
   
   We hit this in production on a ~600 NodeManager cluster running
   DefaultContainerExecutor: because DCE runs the localizer inside the NM
   JVM, every kill-during-localization also leaked DFS block reader sockets
   into the long-lived NM process, leaving DataNodes with FIN_WAIT1
   connections whose send queues never drain. The HDFS side of that leak is
   tracked separately in HDFS-17965 (#8690); this issue is the YARN-side
   trigger and also leaves resources stuck in DOWNLOADING state regardless of
   the container executor in use.
   
   
   ### How was this patch tested?
   
   New unit test
   `TestResourceLocalizationService#testDownloadingResourcesCleanedUpWhenDispatchFails`:
   a mocked dispatcher throws `YarnRuntimeException(InterruptedException)`
   when the `ContainerResourceFailedEvent` is dispatched, simulating the
   kill-during-localization interrupt. Without the fix, `run()` propagates
   the exception and the DOWNLOADING resource is never unlocked (test fails).
   With the fix, the resource is unlocked and the deletion tasks are
   scheduled (test passes).
   
   ### For code changes:
   
   - [x] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
   - [ ] Object storage: Have the integration tests been executed and the endpoint
         declared according to the connector-specific documentation? *Note: Automated CI
         testing doesn't cover all cases so manual testing with cloud storage is still
         required.*
   - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
   
   ### AI Tooling
   
   If an AI tool was used:
   
   - [x] The PR includes the phrase "Contains content generated by <tool>"
         where <tool> is the name of the AI tool used.
   Contains content generated by Claude Code (Anthropic Claude): the
         new unit test. The fix itself is unchanged from #7893. All content
         was human-reviewed and complies with the ASF Generative Tooling
         Guidance (https://www.apache.org/legal/generative-tooling.html).
   - [x] My use of AI contributions follows the ASF legal policy
         https://www.apache.org/legal/generative-tooling.html
   




> DOWNLOADING resources unlock and cleanup is interrupted when killing a container that is localizing
> ---------------------------------------------------------------------------------------------------
>
>                 Key: YARN-11856
>                 URL: https://issues.apache.org/jira/browse/YARN-11856
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>            Reporter: Weihao Zheng
>            Priority: Major
>              Labels: pull-request-available
>
> Currently, LocalizerRunner tries to send a ContainerResourceFailedEvent to dispatcher before unlocking and cleaning up downloading resource when failed by interruption. This handle invoking will throw uncaught exception, which causes other containers depend on the same private resource stuck at localizing.
>  
> Related logs:
> {quote}2025-MM-DD HH:04:59,573 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_e02_XXXXXXXXXXXXX_XXXXXXX_XX_XXXXXX
> ... ...
> 2025-MM-DD HH:04:59,573 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e02_XXXXXXXXXXXXX_XXXXXXX_XX_XXXXXX transitioned from LOCALIZING to KILLING
> ... ...
> 2025-MM-DD HH:04:59,627 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Unknown localizer with localizerId container_e02_XXXXXXXXXXXXX_XXXXXXX_XX_XXXXXX is sending heartbeat. Ordering it to DIE
> 2025-MM-DD HH:04:59,628 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed for container_e02_XXXXXXXXXXXXX_XXXXXXX_XX_XXXXXX
> java.io.IOException: java.io.InterruptedIOException: Interrupted waiting to send RPC request to server
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:200)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:180)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1247)
> Caused by: java.io.InterruptedIOException: Interrupted waiting to send RPC request to server
> at org.apache.hadoop.ipc.Client.call(Client.java:1446)
> at org.apache.hadoop.ipc.Client.call(Client.java:1388)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> at com.sun.proxy.$Proxy82.heartbeat(Unknown Source)
> at org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:63)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:306)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:198)
> ... 2 more
> Caused by: java.lang.InterruptedException
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1158)
> at org.apache.hadoop.ipc.Client.call(Client.java:1441)
> ... 9 more
> 2025-MM-DD HH:04:59,629 WARN org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher thread interrupted
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
> at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339)
> at org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:304)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1271)
> 2025-MM-DD HH:04:59,629 ERROR org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread[LocalizerRunner for container_e02_XXXXXXXXXXXXX_XXXXXXX_XX_XXXXXX,5,main] threw an Exception.
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.InterruptedException
> at org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:312)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1271)
> Caused by: java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
> at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339)
> at org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:304)
> ... 1 more
> {quote}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org