You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2023/03/01 13:31:00 UTC
[jira] [Resolved] (HADOOP-18636) LocalDirAllocator cannot recover from directory tree deletion during the life of a filesystem client
[ https://issues.apache.org/jira/browse/HADOOP-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran resolved HADOOP-18636.
-------------------------------------
Fix Version/s: 3.4.0
3.3.9
Resolution: Fixed
> LocalDirAllocator cannot recover from directory tree deletion during the life of a filesystem client
> ----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-18636
> URL: https://issues.apache.org/jira/browse/HADOOP-18636
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, fs/azure, fs/s3
> Affects Versions: 3.3.4
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> The s3a and abfs clients use LocalDirAllocator for allocating files in local (temporary) storage for buffering blocks to write, and, for the s3a staging committer, files being staged.
> When initialized (or when the configuration key value is updated) LocalDirAllocator enumerates all directories in the list and calls {{mkdirs()}} to create them.
> when you ask actually for a file, it will look for the parent dir, and will again call {{mkdirs()}}.
> But before it does that, it looks to see if the dir has any space...if not it is excluded from the list of directories with room for data.
> And guess what: directories which don't exist report as having no space. So they get excluded -the recreation code doesn't get a chance to run.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org