[jira] [Created] (HADOOP-19937) When a folder is empty or renamed, fs.s3a.directory.marker.retention=delete should delete the parent markers

["Jayesh Lalwani (Jira)" <ji...@apache.org>]

Jayesh Lalwani created HADOOP-19937:
---------------------------------------

             Summary: When a folder is empty or renamed, fs.s3a.directory.marker.retention=delete should delete the parent markers
                 Key: HADOOP-19937
                 URL: https://issues.apache.org/jira/browse/HADOOP-19937
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs/s3
    Affects Versions: 3.4.1, 3.5.0
            Reporter: Jayesh Lalwani


Right now, when a folder is deleted or renamed, S3A creates the parent markers for the empty folders. This makes sense when fs.s3a.directory.marker.retention is set to keep.

 

 However, when fs.s3a.directory.marker.retention is set to delete, it should skip creation of parent markers to maintain backward compatibility.

 

{*}Code references (branch hadoop-3.4.2):{*}{*}{*}

 

 - S3AFileSystem.deleteWithoutCloseCheck() (line 4022) unconditionally calls maybeCreateFakeParentDirectory(path) after delete

 - OperationCallbacksImpl.finishRename() (line 3091) unconditionally calls maybeCreateFakeParentDirectory(sourceRenamed) after rename

 - Neither call checks keepDirectoryMarkers() before creating the marker

 

{*}Impact:{*}{*}{*}

 

 Customers migrating from EMRFS to S3A (e.g., EMR 7.8 → 7.10) see orphan zero-byte directory markers accumulating after FileSystem.rename() operations. These markers cause applications (e.g., Spark Structured Streaming) to issue nnecessary S3 LIST calls on empty paths, increasing S3 costs. Setting fs.s3a.directory.marker.retention=delete does not prevent this behavior.

 

{*}Proposed fix:{*}{*}{*}

 

 Guard the maybeCreateFakeParentDirectory() calls with the marker retention policy:

 

{{// In finishRename():}}

{{if (keepDirectoryMarkers(sourceRenamed.getParent())) {}}

{{     maybeCreateFakeParentDirectory(sourceRenamed);}}

{{ }}}

 

{{// In deleteWithoutCloseCheck():}}

{{if (keepDirectoryMarkers(path.getParent())) {}}

{{     maybeCreateFakeParentDirectory(path);}}

{{ }}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org