[jira] [Created] (HADOOP-18948) S3A. Add option fs.s3a.directory.operations.purge.uploads to do that on rename/delete

["Steve Loughran (Jira)" <ji...@apache.org>]

Steve Loughran created HADOOP-18948:
---------------------------------------

             Summary: S3A. Add option fs.s3a.directory.operations.purge.uploads to do that on rename/delete
                 Key: HADOOP-18948
                 URL: https://issues.apache.org/jira/browse/HADOOP-18948
             Project: Hadoop Common
          Issue Type: Sub-task
          Components: fs/s3
    Affects Versions: 3.4.0
            Reporter: Steve Loughran
            Assignee: Steve Loughran



On third-party stores without lifecycle rules its possible to accrue many GB of pending multipart uploads, including from
* magic committer jobs where spark driver/MR AM failed before commit/abort
* distcp jobs which timeout and get aborted
* any client code writing datasets which are interrupted before close.

Although there's a purge pending uploads option, that's dangerous because if any fs is instantiated with it, it can destroy in-flight work

otherwise, the "hadoop s3guard uploads" command does work but needs scheduling/manual execution

proposed: add a new property {{fs.s3a.directory.operations.purge.uploads}} which will automatically cancel all pending uploads under a path
* delete: everything under the dir
* rename: all under the source dir

This will be done in parallel to the normal operation, but no attempt to post abortMultipartUploads in different threads. The assumption here is that this is rare. And it'll be off by default as in AWS people should have rules for these things.


+ doc (third_party?)
+ add new counter/metric for abort operations, count and duration
+ test to include cost assertions






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org