[jira] [Commented] (HADOOP-19589) Support atomic rename() for S3 express

["Steve Loughran (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17983676#comment-17983676 ] 

Steve Loughran commented on HADOOP-19589:
-----------------------------------------

* we can bypass xfer manager and just issue rename commands to the thread pool, with some throttling
* source etag should be passed in
* auditing to attach referrer header


I see the comment about idempotency and want to know what it means if the request works, but we get a network error on the response, so don't see the 200

s3a will have to retry, and the source will have been copied. What then?

If the dest etag == source then s3 should be able to consider this a no-op. does it?

otherwise we repeat, catch whatever error comes back from missing source, probe dest for having etag and the interpret as success.

* Lots of tests of failure and race conditions, with the fault injection stuff we've added for other tests.


note also the manifest committer could then run on s3, org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestStoreOperationsThroughFileSystem
no tangible benefit over the magic committer, just an interesting fact. Not worth the effort to support.

Be more relevant to hive commits, FWIW.



> Support atomic rename() for S3 express
> --------------------------------------
>
>                 Key: HADOOP-19589
>                 URL: https://issues.apache.org/jira/browse/HADOOP-19589
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Ahmar Suhail
>            Priority: Major
>
> S3 Express now supports atomic renames: [https://aws.amazon.com/about-aws/whats-new/2025/06/amazon-s3-express-one-zone-atomic-renaming-objects-api/]
>  
> Rename API is available as of SDK version 2.31.66. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org