You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "sksingh2005 (via GitHub)" <gi...@apache.org> on 2025/12/31 23:42:49 UTC
[PR] feat(s3): add conditional headers (If-Match, If-None-Match) for copy/… [opendal]
sksingh2005 opened a new pull request, #7115:
URL: https://github.com/apache/opendal/pull/7115
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #7090 .
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
AWS S3 now supports conditional headers (If-Match and If-None-Match) for CopyObject and DeleteObject operations, enabling atomic operations and preventing race conditions in concurrent scenarios. This PR adds support for these headers to OpenDAL's S3 service, bringing it to feature parity with AWS S3's latest capabilities and matching the existing implementation in the Azure Blob service.
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
This PR implements three new conditional operation capabilities for S3:
1. `copy_with_if_not_exists`
- Copy operations that only succeed if the target doesn't exist (using `If-None-Match: *`)
2. `copy_with_if_match`
- Copy operations that only succeed if the source's ETag matches (using `If-Match: <etag>`)
3. `delete_with_if_match`
- Delete operations that only succeed if the file's ETag matches (using` If-Match: <etag>`)
### Core Changes:
- Added [copy_with_if_match](cci:1://file://wsl.localhost/Ubuntu-22.04/home/sayshank/projects/opendal/core/tests/behavior/async_copy.rs:361:0-387:1) and [delete_with_if_match](cci:1://file://wsl.localhost/Ubuntu-22.04/home/sayshank/projects/opendal/core/tests/behavior/async_delete.rs:414:0-439:1) capability flags
- Extended [OpCopy](cci:2://file://wsl.localhost/Ubuntu-22.04/home/sayshank/projects/opendal/core/core/src/raw/ops.rs:891:0-894:1) and [OpDelete](cci:2://file://wsl.localhost/Ubuntu-22.04/home/sayshank/projects/opendal/core/core/src/raw/ops.rs:43:0-47:1) with [if_match](cci:1://file://wsl.localhost/Ubuntu-22.04/home/sayshank/projects/opendal/core/core/src/raw/ops.rs:85:4-88:5) fields
- Added [if_match](cci:1://file://wsl.localhost/Ubuntu-22.04/home/sayshank/projects/opendal/core/core/src/raw/ops.rs:85:4-88:5) to [CopyOptions](cci:2://file://wsl.localhost/Ubuntu-22.04/home/sayshank/projects/opendal/core/core/src/types/options.rs:525:0-549:1) and [DeleteOptions](cci:2://file://wsl.localhost/Ubuntu-22.04/home/sayshank/projects/opendal/core/core/src/types/options.rs:24:0-40:1)
- Exposed [if_match()](cci:1://file://wsl.localhost/Ubuntu-22.04/home/sayshank/projects/opendal/core/core/src/raw/ops.rs:85:4-88:5) methods in `FutureCopy` and `FutureDelete` for user-facing API
**S3 Service Implementation:**
- Updated S3 backend to advertise new capabilities
- Implemented conditional headers in [s3_copy_object()](cci:1://file://wsl.localhost/Ubuntu-22.04/home/sayshank/projects/opendal/core/services/s3/src/core.rs:632:4-701:5) and [s3_delete_object()](cci:1://file://wsl.localhost/Ubuntu-22.04/home/sayshank/projects/opendal/core/services/s3/src/core.rs:594:4-630:5)
- Added proper header handling for `If-Match` and `If-None-Match`
**Testing:**
- Added 4 new behavior tests covering success and failure scenarios
- Tests follow existing OpenDAL patterns and are registered in the test suite
# Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
**Yes**, this PR adds new user-facing methods:
```rust
// Copy with If-Match header
op.copy_with("source", "target")
.if_match("\"etag-value\"")
.await?;
// Delete with If-Match header
op.delete_with("path")
.if_match("\"etag-value\"")
.await?;
```
<!---
If there are any breaking changes to public APIs, please add the `breaking-changes` label.
-->
# AI Usage Statement
<!--
If you are using AI tools to build this PR, please include a statement specifying the tool and models you are using.
-->
This PR was implemented with assistance from **Claude Sonnet 4.5** through the Antigravity coding assistant. The AI helped with:
- Test case design based on existing behavior tests
All code was reviewed, tested, and validated by the human contributor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(s3): add conditional headers (If-Match, If-None-Match) for copy/… [opendal]
Posted by "dentiny (via GitHub)" <gi...@apache.org>.
dentiny commented on PR #7115:
URL: https://github.com/apache/opendal/pull/7115#issuecomment-4286674138
Hi @sksingh2005 I think this PR is valuable, I'm wondering if you want to resume? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(s3): add conditional headers (If-Match, If-None-Match) for copy/… [opendal]
Posted by "sksingh2005 (via GitHub)" <gi...@apache.org>.
sksingh2005 commented on code in PR #7115:
URL: https://github.com/apache/opendal/pull/7115#discussion_r2659715227
##########
core/services/s3/src/backend.rs:
##########
@@ -922,12 +922,15 @@ impl Builder for S3Builder {
},
delete: true,
+ delete_with_if_match: true,
delete_max_size: Some(delete_max_size),
delete_with_version: config.enable_versioning,
copy: true,
+ copy_with_if_not_exists: true,
+ copy_with_if_match: true,
- list: true,
+ rename: false,
Review Comment:
My apologies, it seems `list: true` was accidentally removed during the `rename `capability update. I have restored it. We need `rename: false` to force the `Copy `+ `Delete` polyfill because S3 doesn't support native rename, but l`ist: true` is definitely required i guess sir ??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(s3): add conditional headers (If-Match, If-None-Match) for copy/… [opendal]
Posted by "sksingh2005 (via GitHub)" <gi...@apache.org>.
sksingh2005 commented on PR #7115:
URL: https://github.com/apache/opendal/pull/7115#issuecomment-3756425817
My apologies, it seems `list: true` was accidentally removed during the `rename ` capability update. I have restored it. We need `rename: false` to force the Copy + Delete polyfill because S3 doesn't support native rename, but `list: true` is definitely required i guess sir ?? Review needed @Xuanwo sir
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(s3): add conditional headers (If-Match, If-None-Match) for copy/… [opendal]
Posted by "sksingh2005 (via GitHub)" <gi...@apache.org>.
sksingh2005 commented on PR #7115:
URL: https://github.com/apache/opendal/pull/7115#issuecomment-3772540540
Thank you @Xuanwo sir for the review and for confirming the root cause.
The implementation is correct for AWS S3. The test failures are expected since MinIO doesn't support `If-Match`/`If-None-Match` headers for CopyObject and DeleteObject yet.
Please let me know if you'd like me to add a configuration flag to disable these features for S3-compatible services, or if the current implementation is acceptable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(s3): add conditional headers (If-Match, If-None-Match) for copy/… [opendal]
Posted by "sksingh2005 (via GitHub)" <gi...@apache.org>.
sksingh2005 closed pull request #7115: feat(s3): add conditional headers (If-Match, If-None-Match) for copy/…
URL: https://github.com/apache/opendal/pull/7115
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(s3): add conditional headers (If-Match, If-None-Match) for copy/… [opendal]
Posted by "dentiny (via GitHub)" <gi...@apache.org>.
dentiny commented on PR #7115:
URL: https://github.com/apache/opendal/pull/7115#issuecomment-4190385494
Seems env vars will be parsed at test start: https://github.com/apache/opendal/blob/ab9274cf197287239132ef02f6f73327c05a59bb/core/testkit/src/utils.rs#L55-L61
So I guess we could add an optional to disable
```diff
diff --git a/.github/services/s3/0_minio_s3/action.yml b/.github/services/s3/0_minio_s3/action.yml
index 2ee8bc8bb..479522ddc 100644
--- a/.github/services/s3/0_minio_s3/action.yml
+++ b/.github/services/s3/0_minio_s3/action.yml
@@ -43,4 +43,5 @@ runs:
OPENDAL_S3_ACCESS_KEY_ID=minioadmin
OPENDAL_S3_SECRET_ACCESS_KEY=minioadmin
OPENDAL_S3_REGION=us-east-1
+ OPENDAL_S3_DISABLE_COPY_WITH_IF_MATCH=true
EOF
diff --git a/.github/services/s3/minio_s3_with_anonymous/action.yml b/.github/services/s3/minio_s3_with_anonymous/action.yml
index 31c5e0851..114bf7780 100644
--- a/.github/services/s3/minio_s3_with_anonymous/action.yml
+++ b/.github/services/s3/minio_s3_with_anonymous/action.yml
@@ -48,4 +48,5 @@ runs:
OPENDAL_S3_REGION=us-east-1
OPENDAL_S3_ALLOW_ANONYMOUS=on
OPENDAL_S3_DISABLE_EC2_METADATA=on
+ OPENDAL_S3_DISABLE_COPY_WITH_IF_MATCH=true
EOF
diff --git a/.github/services/s3/minio_s3_with_list_objects_v1/action.yml b/.github/services/s3/minio_s3_with_list_objects_v1/action.yml
index 46904d7aa..ef209c79c 100644
--- a/.github/services/s3/minio_s3_with_list_objects_v1/action.yml
+++ b/.github/services/s3/minio_s3_with_list_objects_v1/action.yml
@@ -44,4 +44,5 @@ runs:
OPENDAL_S3_SECRET_ACCESS_KEY=minioadmin
OPENDAL_S3_REGION=us-east-1
OPENDAL_S3_DISABLE_LIST_OBJECTS_V2=true
+ OPENDAL_S3_DISABLE_COPY_WITH_IF_MATCH=true
EOF
diff --git a/.github/services/s3/minio_s3_with_versioning/action.yml b/.github/services/s3/minio_s3_with_versioning/action.yml
index e2db993a8..8143df537 100644
--- a/.github/services/s3/minio_s3_with_versioning/action.yml
+++ b/.github/services/s3/minio_s3_with_versioning/action.yml
@@ -45,4 +45,5 @@ runs:
OPENDAL_S3_SECRET_ACCESS_KEY=minioadmin
OPENDAL_S3_REGION=us-east-1
OPENDAL_S3_ENABLE_VERSIONING=true
+ OPENDAL_S3_DISABLE_COPY_WITH_IF_MATCH=true
EOF
diff --git a/core/services/s3/src/backend.rs b/core/services/s3/src/backend.rs
index 3fd62ee69..36e17dfe2 100644
--- a/core/services/s3/src/backend.rs
+++ b/core/services/s3/src/backend.rs
@@ -555,6 +555,14 @@ impl S3Builder {
self
}
+ /// Disable copy with if match so that opendal will not send copy request with if match headers.
+ ///
+ /// For example, MinIO doesn't support copy with if match.
+ pub fn disable_copy_with_if_match(mut self) -> Self {
+ self.config.disable_copy_with_if_match = true;
+ self
+ }
+
/// Enable write with append so that opendal will send write request with append headers.
pub fn enable_write_with_append(mut self) -> Self {
self.config.enable_write_with_append = true;
diff --git a/core/services/s3/src/config.rs b/core/services/s3/src/config.rs
index 8c219de6c..cba6491b5 100644
--- a/core/services/s3/src/config.rs
+++ b/core/services/s3/src/config.rs
@@ -210,6 +210,11 @@ pub struct S3Config {
/// For example, Ceph RADOS S3 doesn't support write with if matched.
pub disable_write_with_if_match: bool,
+ /// Disable copy with if match so that opendal will not send copy request with if match headers.
+ ///
+ /// For example, MinIO doesn't support copy with if match.
+ pub disable_copy_with_if_match: bool,
+
/// Enable write with append so that opendal will send write request with append headers.
pub enable_write_with_append: bool,
diff --git a/core/services/s3/src/docs.md b/core/services/s3/src/docs.md
index df279f35b..e0f47efd3 100644
--- a/core/services/s3/src/docs.md
+++ b/core/services/s3/src/docs.md
@@ -30,6 +30,7 @@ This service can be used to:
- `disable_config_load`: Disable aws config load from env.
- `enable_virtual_host_style`: Enable virtual host style.
- `disable_write_with_if_match`: Disable write with if match.
+- `disable_copy_with_if_match`: Disable copy with if match / if none match. Some S3-compatible services like MinIO don't support conditional copy.
- `enable_request_payer`: Enable the request payer for backend.
- `default_acl`: Define the default access control list (ACL) when creating a new object. Note that some s3 services like minio do not support this option.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(s3): add conditional headers (If-Match, If-None-Match) for copy/… [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #7115:
URL: https://github.com/apache/opendal/pull/7115#issuecomment-3766827711
Thank you @sksingh2005 for working on this. I believe most of the changes in this PR are correct. The root cause here is that minio does not support `copy with if-match & if-none-match` or `delete with if-match`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(s3): add conditional headers (If-Match, If-None-Match) for copy/… [opendal]
Posted by "sksingh2005 (via GitHub)" <gi...@apache.org>.
sksingh2005 commented on code in PR #7115:
URL: https://github.com/apache/opendal/pull/7115#discussion_r2659715227
##########
core/services/s3/src/backend.rs:
##########
@@ -922,12 +922,15 @@ impl Builder for S3Builder {
},
delete: true,
+ delete_with_if_match: true,
delete_max_size: Some(delete_max_size),
delete_with_version: config.enable_versioning,
copy: true,
+ copy_with_if_not_exists: true,
+ copy_with_if_match: true,
- list: true,
+ rename: false,
Review Comment:
My apologies, it seems `list: true` was accidentally removed during the `rename `capability update. I have restored it. We need `rename: false` to force the `Copy `+ `Delete` polyfill because S3 doesn't support native rename, but l`ist: true` is definitely required i guess sir ??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(s3): add conditional headers (If-Match, If-None-Match) for copy/… [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #7115:
URL: https://github.com/apache/opendal/pull/7115#discussion_r2659479574
##########
core/services/s3/src/backend.rs:
##########
@@ -922,12 +922,15 @@ impl Builder for S3Builder {
},
delete: true,
+ delete_with_if_match: true,
delete_max_size: Some(delete_max_size),
delete_with_version: config.enable_versioning,
copy: true,
+ copy_with_if_not_exists: true,
+ copy_with_if_match: true,
- list: true,
+ rename: false,
Review Comment:
Seems we changed `list: true` to `rename: false`? Why this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org