You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/08/18 23:51:00 UTC
[jira] [Commented] (YARN-11856) DOWNLOADING resources unlock and cleanup is interrupted when killing a container that is localizing
[ https://issues.apache.org/jira/browse/YARN-11856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18105722#comment-18105722 ]
ASF GitHub Bot commented on YARN-11856:
---------------------------------------
eubnara opened a new pull request, #8691:
URL: https://github.com/apache/hadoop/pull/8691
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
When a container is killed while it is localizing, the `LocalizerRunner`
thread is interrupted. In the `finally` block of `LocalizerRunner.run()`,
dispatching the `ContainerResourceFailedEvent` then throws
`YarnRuntimeException(InterruptedException)`, which skips everything after
it: resources left in DOWNLOADING state are never unlocked, and the
deletion tasks for the localization dirs, the `_tmp` download dirs and the
nmPrivate token file are never scheduled.
This PR revives the fix from #7893 by @zheng-weihao (stale-closed after
100 days of inactivity), unchanged: the event dispatch is wrapped in
try/catch so the cleanup below it always runs. The remaining review
comment on #7893 (dropping `FSError` from the catch list) had already been
addressed in its final revision. Credit to the original author is kept in
the commit message.
What this PR adds on top of #7893 is a unit test, which was the other
blocker ("no new or modified tests").
We hit this in production on a ~600 NodeManager cluster running
DefaultContainerExecutor: because DCE runs the localizer inside the NM
JVM, every kill-during-localization also leaked DFS block reader sockets
into the long-lived NM process, leaving DataNodes with FIN_WAIT1
connections whose send queues never drain. The HDFS side of that leak is
tracked separately in HDFS-17965 (#8690); this issue is the YARN-side
trigger and also leaves resources stuck in DOWNLOADING state regardless of
the container executor in use.
### How was this patch tested?
New unit test
`TestResourceLocalizationService#testDownloadingResourcesCleanedUpWhenDispatchFails`:
a mocked dispatcher throws `YarnRuntimeException(InterruptedException)`
when the `ContainerResourceFailedEvent` is dispatched, simulating the
kill-during-localization interrupt. Without the fix, `run()` propagates
the exception and the DOWNLOADING resource is never unlocked (test fails).
With the fix, the resource is unlocked and the deletion tasks are
scheduled (test passes).
### For code changes:
- [x] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the endpoint
declared according to the connector-specific documentation? *Note: Automated CI
testing doesn't cover all cases so manual testing with cloud storage is still
required.*
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [x] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
Contains content generated by Claude Code (Anthropic Claude): the
new unit test. The fix itself is unchanged from #7893. All content
was human-reviewed and complies with the ASF Generative Tooling
Guidance (https://www.apache.org/legal/generative-tooling.html).
- [x] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
> DOWNLOADING resources unlock and cleanup is interrupted when killing a container that is localizing
> ---------------------------------------------------------------------------------------------------
>
> Key: YARN-11856
> URL: https://issues.apache.org/jira/browse/YARN-11856
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Reporter: Weihao Zheng
> Priority: Major
> Labels: pull-request-available
>
> Currently, LocalizerRunner tries to send a ContainerResourceFailedEvent to dispatcher before unlocking and cleaning up downloading resource when failed by interruption. This handle invoking will throw uncaught exception, which causes other containers depend on the same private resource stuck at localizing.
>
> Related logs:
> {quote}2025-MM-DD HH:04:59,573 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_e02_XXXXXXXXXXXXX_XXXXXXX_XX_XXXXXX
> ... ...
> 2025-MM-DD HH:04:59,573 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e02_XXXXXXXXXXXXX_XXXXXXX_XX_XXXXXX transitioned from LOCALIZING to KILLING
> ... ...
> 2025-MM-DD HH:04:59,627 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Unknown localizer with localizerId container_e02_XXXXXXXXXXXXX_XXXXXXX_XX_XXXXXX is sending heartbeat. Ordering it to DIE
> 2025-MM-DD HH:04:59,628 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer failed for container_e02_XXXXXXXXXXXXX_XXXXXXX_XX_XXXXXX
> java.io.IOException: java.io.InterruptedIOException: Interrupted waiting to send RPC request to server
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:200)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:180)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1247)
> Caused by: java.io.InterruptedIOException: Interrupted waiting to send RPC request to server
> at org.apache.hadoop.ipc.Client.call(Client.java:1446)
> at org.apache.hadoop.ipc.Client.call(Client.java:1388)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> at com.sun.proxy.$Proxy82.heartbeat(Unknown Source)
> at org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:63)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:306)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:198)
> ... 2 more
> Caused by: java.lang.InterruptedException
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1158)
> at org.apache.hadoop.ipc.Client.call(Client.java:1441)
> ... 9 more
> 2025-MM-DD HH:04:59,629 WARN org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher thread interrupted
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
> at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339)
> at org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:304)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1271)
> 2025-MM-DD HH:04:59,629 ERROR org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread[LocalizerRunner for container_e02_XXXXXXXXXXXXX_XXXXXXX_XX_XXXXXX,5,main] threw an Exception.
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.InterruptedException
> at org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:312)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1271)
> Caused by: java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
> at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339)
> at org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:304)
> ... 1 more
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org