You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "benesch (via GitHub)" <gi...@apache.org> on 2024/11/26 05:30:31 UTC

[PR] Support real S3's If-Match semantics [arrow-rs]

benesch opened a new pull request, #6801:
URL: https://github.com/apache/arrow-rs/pull/6801

   As of today [0] S3 now supports the If-Match for in-place conditional writes. This commit adjusts the existing support for S3ConditionalPut::Etag mode for compatibility with real S3's particular semantics, which vary slightly from MinIO and R2. Specifically:
   
     * Real S3 can occasionally return 409 Conflict when concurrent If-Match requests are in progress. These requests need to be retried.
   
     * Real S3 returns 404 Not Found instead of 412 Precondition Failed when issuing an If-Match request against an object that does not exist.
   
   Fix #6799.
   
   [0]: https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-functionality-conditional-writes/
   
   # Which issue does this PR close?
   
   #6799
   
   # Are there any user-facing changes?
   
   None beyond what's described in the PR description.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] object-store: support real S3's If-Match semantics [arrow-rs]

Posted by "benesch (via GitHub)" <gi...@apache.org>.
benesch commented on PR #6801:
URL: https://github.com/apache/arrow-rs/pull/6801#issuecomment-2501360107

   Sounds good.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] object-store: support real S3's If-Match semantics [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #6801:
URL: https://github.com/apache/arrow-rs/pull/6801#issuecomment-2501275038

   No, I think this looks good, but given the localstack update may land shortly I was going to hold this until then just to ensure there aren't any gremlins lurking


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] object-store: support real S3's If-Match semantics [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #6801:
URL: https://github.com/apache/arrow-rs/pull/6801#discussion_r1858029921


##########
object_store/src/aws/mod.rs:
##########
@@ -199,7 +199,25 @@ impl ObjectStore for AmazonS3 {
                 match put {
                     S3ConditionalPut::ETagPutIfNotExists => Err(Error::NotImplemented),
                     S3ConditionalPut::ETagMatch => {
-                        request.header(&IF_MATCH, etag.as_str()).do_put().await
+                        match request
+                            .header(&IF_MATCH, etag.as_str())
+                            // Real S3 will occasionally report 409 Conflict
+                            // if there are concurrent `If-Match` requests
+                            // in flight, so we need to be prepared to retry
+                            // 409 responses.
+                            .retry_on_conflict(true)
+                            .do_put()
+                            .await
+                        {
+                            // Real S3 reports NotFound rather than PreconditionFailed when the
+                            // object doesn't exist. Convert to PreconditionFailed for
+                            // consistency with R2. This also matches what the HTTP spec

Review Comment:
   😅



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] object-store: support real S3's If-Match semantics [arrow-rs]

Posted by "benesch (via GitHub)" <gi...@apache.org>.
benesch commented on PR #6801:
URL: https://github.com/apache/arrow-rs/pull/6801#issuecomment-2501283269

   > No, I think this looks good, but given the localstack update may land shortly I was going to hold this until then just to ensure there aren't any gremlins lurking
   
   Makes sense! In that case would you prefer if I closed this PR in favor of https://github.com/apache/arrow-rs/pull/6802, so there's just one PR to review and merge after the new localstack release comes out? Or is it still useful to have both PRs?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] object-store: support real S3's If-Match semantics [arrow-rs]

Posted by "benesch (via GitHub)" <gi...@apache.org>.
benesch commented on PR #6801:
URL: https://github.com/apache/arrow-rs/pull/6801#issuecomment-2506694138

   Since [all seems to be working](https://github.com/apache/arrow-rs/pull/6802#issuecomment-2506693683) with the latest version of localstack, I'm going to go ahead and close this in favor of #6802. Can always reopen this if necessary!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] object-store: support real S3's If-Match semantics [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #6801:
URL: https://github.com/apache/arrow-rs/pull/6801#issuecomment-2501343483

   Let's keep it for now just in case there's some unforeseen hickup with the localstack release, but yes presuming localstack lands we'll probably just merge the other PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] object-store: support real S3's If-Match semantics [arrow-rs]

Posted by "benesch (via GitHub)" <gi...@apache.org>.
benesch closed pull request #6801: object-store: support real S3's If-Match semantics
URL: https://github.com/apache/arrow-rs/pull/6801


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] object-store: support real S3's If-Match semantics [arrow-rs]

Posted by "benesch (via GitHub)" <gi...@apache.org>.
benesch commented on PR #6801:
URL: https://github.com/apache/arrow-rs/pull/6801#issuecomment-2501193851

   > Makes sense to me, thank you.
   
   You bet! Anything else you need from me to merge this? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org