You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "chhinlinghean (Jira)" <ji...@apache.org> on 2026/10/07 02:31:00 UTC
[jira] [Updated] (YARN-12009) NodeManager disk health check falsely marks local dirs as failed when files are deleted or renamed during the content accessibility validation
[ https://issues.apache.org/jira/browse/YARN-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
chhinlinghean updated YARN-12009:
---------------------------------
Description:
{{DirectoryCollection#validateSubsAccessibility}} ([code|https://github.com/apache/hadoop/blob/997fa7c12a407c441855f1a409767f198db68cd3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java#L592]) first collects the *full list of paths* under the dir with {{Files.walk(...).collect(...)}} then checks each path. The NodeManager keeps changing this tree the whole time:
# Localization unpacks archives into {{filecache/_tmp}} and then renames the directory to {{{}filecache/{}}},
# Cache cleanup deletes old {{filecache}} entries,
# Application cleanup and log aggregation remove application directories.
Any entry removed between listing and checking is reported as an error. {{Files.isReadable()}} returns {{false}} for a missing file, so the log message looks like a permission problem:
{code:java}
WARN DirectoryCollection: Directory /data1/yarn/local error, Can not read /data1/yarn/local/filecache/12345_tmp/env.tar.gz/bin/ld, removing from list of valid directories
INFO LocalDirsHandlerService: Disk(s) failed: 1/8 local-dirs have errors: [ /data1/yarn/local : Can not read /data1/yarn/local/filecache/12345_tmp/env.tar.gz/bin/ld ] {noformat} {code}
was:
{\{DirectoryCollection#validateSubsAccessibility}} ([code|https://github.com/apache/hadoop/blob/997fa7c12a407c441855f1a409767f198db68cd3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java#L592]) first collects the full list of paths under the dir with \{{Files.walk(...).collect(...)}}, then checks each path. The NodeManager keeps changing this tree the whole time:
# Localization unpacks archives into \{{filecache/_tmp}} and then renames the directory to \{{filecache/}},
# Cache cleanup deletes old \{{filecache}} entries,
# Application cleanup and log aggregation remove application directories.
Any entry removed between listing and checking is reported as an error. \{{Files.isReadable()}} returns \{{false}} for a missing file, so the log message looks like a permission problem:
```
WARN DirectoryCollection: Directory /data1/yarn/local error, Can not read /data1/yarn/local/filecache/12345_tmp/env.tar.gz/bin/ld, removing from list of valid directories
INFO LocalDirsHandlerService: Disk(s) failed: 1/8 local-dirs have errors: [ /data1/yarn/local : Can not read /data1/yarn/local/filecache/12345_tmp/env.tar.gz/bin/ld ] \{noformat}
```
Depending on timing, the walk itself can also fail with \{{UncheckedIOException(NoSuchFileException)}}
> NodeManager disk health check falsely marks local dirs as failed when files are deleted or renamed during the content accessibility validation
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-12009
> URL: https://issues.apache.org/jira/browse/YARN-12009
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
> Affects Versions: 3.5.0
> Reporter: chhinlinghean
> Assignee: chhinlinghean
> Priority: Major
>
> {{DirectoryCollection#validateSubsAccessibility}} ([code|https://github.com/apache/hadoop/blob/997fa7c12a407c441855f1a409767f198db68cd3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java#L592]) first collects the *full list of paths* under the dir with {{Files.walk(...).collect(...)}} then checks each path. The NodeManager keeps changing this tree the whole time:
> # Localization unpacks archives into {{filecache/_tmp}} and then renames the directory to {{{}filecache/{}}},
> # Cache cleanup deletes old {{filecache}} entries,
> # Application cleanup and log aggregation remove application directories.
> Any entry removed between listing and checking is reported as an error. {{Files.isReadable()}} returns {{false}} for a missing file, so the log message looks like a permission problem:
> {code:java}
> WARN DirectoryCollection: Directory /data1/yarn/local error, Can not read /data1/yarn/local/filecache/12345_tmp/env.tar.gz/bin/ld, removing from list of valid directories
> INFO LocalDirsHandlerService: Disk(s) failed: 1/8 local-dirs have errors: [ /data1/yarn/local : Can not read /data1/yarn/local/filecache/12345_tmp/env.tar.gz/bin/ld ] {noformat} {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org