You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2023/02/17 11:41:00 UTC
[jira] [Created] (HADOOP-18636) LocalDirAllocator cannot recover from directory tree deletion during the life of a filesystem client
Steve Loughran created HADOOP-18636:
---------------------------------------
Summary: LocalDirAllocator cannot recover from directory tree deletion during the life of a filesystem client
Key: HADOOP-18636
URL: https://issues.apache.org/jira/browse/HADOOP-18636
Project: Hadoop Common
Issue Type: Bug
Components: fs, fs/azure, fs/s3
Affects Versions: 3.3.4
Reporter: Steve Loughran
Assignee: Steve Loughran
The s3a and abfs clients use LocalDirAllocator for allocating files in local (temporary) storage for buffering blocks to write, and, for the s3a staging committer, files being staged.
When initialized (or when the configuration key value is updated) LocalDirAllocator enumerates all directories in the list and calls {{mkdirs()}} to create them.
when you ask actually for a file, it will look for the parent dir, but it calls {{mkdir()}}, rather than {{mkdirs()}}
This means it will recreate a missing parent file but cannot recover from a missing grandparent. If during the life of an application the temp directory is cleaned up, it can result in the failure of the application.
Fix add an "s" to the right place in the production code, plus a new test.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org