You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "William Montaz (Jira)" <ji...@apache.org> on 2025/12/08 13:58:00 UTC
[jira] [Updated] (YARN-11906) Nodemanager broken when using ReadWriteDiskValidator
[ https://issues.apache.org/jira/browse/YARN-11906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
William Montaz updated YARN-11906:
----------------------------------
Description:
YARN-7300 introduces DiskValidator in LocalDirAllocator. It replaces calls to DiskChecker.checkDir() by DiskValidator.checkStatus().
The problem this brings is that DiskChecker is creating directories in its check. This is directly reproduced by BasicDiskValidator. But ReadWriteDiskValidator does not create the dir, it check if dir exists prior to calling DiskChecker.checkDir() and fails directly if the dir is absent.
ContainerLaunch will try to create paths that do not exist (example, the launch_container.sh file), expecting DiskChecker to create them. Since YARN-7300 using ReadWriteDiskValidator will thus fail on any good disk when the dir do not exist. We end with such exceptions at any container launch phase
{noformat}
Could not find any valid local directory for nmPrivate/application_1765167412015_56041/container_e1150_1765167412015_56041_02_000001//launch_container.sh with requested size -1 as the max capacity in any directory is 0{noformat}
was:
YARN-7300 introduces DiskValidator in LocalDirAllocator. In details, it replaces calls to DiskChecker.checkDir() by DiskValidator.checkStatus().
The problem this brings is that DiskChecker is creating directories in its check. This is directly reproduced by BasicDiskValidator. But ReadWriteDiskValidator does not create the dir, it check if dir exists prior to calling DiskChecker.checkDir() and fails directly if the dir is absent.
ContainerLaunch will try to create paths that do not exist (example, the launch_container.sh file), expecting DiskChecker to create them. Since YARN-7300 using ReadWriteDiskValidator will thus fail on any good disk when the dir do not exist. We end with such exceptions at any container launch phase
{noformat}
Could not find any valid local directory for nmPrivate/application_1765167412015_56041/container_e1150_1765167412015_56041_02_000001//launch_container.sh with requested size -1 as the max capacity in any directory is 0{noformat}
> Nodemanager broken when using ReadWriteDiskValidator
> ----------------------------------------------------
>
> Key: YARN-11906
> URL: https://issues.apache.org/jira/browse/YARN-11906
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 3.3.6
> Reporter: William Montaz
> Priority: Major
>
> YARN-7300 introduces DiskValidator in LocalDirAllocator. It replaces calls to DiskChecker.checkDir() by DiskValidator.checkStatus().
> The problem this brings is that DiskChecker is creating directories in its check. This is directly reproduced by BasicDiskValidator. But ReadWriteDiskValidator does not create the dir, it check if dir exists prior to calling DiskChecker.checkDir() and fails directly if the dir is absent.
> ContainerLaunch will try to create paths that do not exist (example, the launch_container.sh file), expecting DiskChecker to create them. Since YARN-7300 using ReadWriteDiskValidator will thus fail on any good disk when the dir do not exist. We end with such exceptions at any container launch phase
> {noformat}
> Could not find any valid local directory for nmPrivate/application_1765167412015_56041/container_e1150_1765167412015_56041_02_000001//launch_container.sh with requested size -1 as the max capacity in any directory is 0{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org