Re: [PR] feat(s3): add conditional headers (If-Match, If-None-Match) for copy/… [opendal]

["dentiny (via GitHub)" <gi...@apache.org>]

dentiny commented on PR #7115:
URL: https://github.com/apache/opendal/pull/7115#issuecomment-4190385494

   Seems env vars will be parsed at test start: https://github.com/apache/opendal/blob/ab9274cf197287239132ef02f6f73327c05a59bb/core/testkit/src/utils.rs#L55-L61
   
   So I guess we could add an optional to disable
   ```diff
   diff --git a/.github/services/s3/0_minio_s3/action.yml b/.github/services/s3/0_minio_s3/action.yml
   index 2ee8bc8bb..479522ddc 100644
   --- a/.github/services/s3/0_minio_s3/action.yml
   +++ b/.github/services/s3/0_minio_s3/action.yml
   @@ -43,4 +43,5 @@ runs:
            OPENDAL_S3_ACCESS_KEY_ID=minioadmin
            OPENDAL_S3_SECRET_ACCESS_KEY=minioadmin
            OPENDAL_S3_REGION=us-east-1
   +        OPENDAL_S3_DISABLE_COPY_WITH_IF_MATCH=true
            EOF
   diff --git a/.github/services/s3/minio_s3_with_anonymous/action.yml b/.github/services/s3/minio_s3_with_anonymous/action.yml
   index 31c5e0851..114bf7780 100644
   --- a/.github/services/s3/minio_s3_with_anonymous/action.yml
   +++ b/.github/services/s3/minio_s3_with_anonymous/action.yml
   @@ -48,4 +48,5 @@ runs:
            OPENDAL_S3_REGION=us-east-1
            OPENDAL_S3_ALLOW_ANONYMOUS=on
            OPENDAL_S3_DISABLE_EC2_METADATA=on
   +        OPENDAL_S3_DISABLE_COPY_WITH_IF_MATCH=true
            EOF
   diff --git a/.github/services/s3/minio_s3_with_list_objects_v1/action.yml b/.github/services/s3/minio_s3_with_list_objects_v1/action.yml
   index 46904d7aa..ef209c79c 100644
   --- a/.github/services/s3/minio_s3_with_list_objects_v1/action.yml
   +++ b/.github/services/s3/minio_s3_with_list_objects_v1/action.yml
   @@ -44,4 +44,5 @@ runs:
            OPENDAL_S3_SECRET_ACCESS_KEY=minioadmin
            OPENDAL_S3_REGION=us-east-1
            OPENDAL_S3_DISABLE_LIST_OBJECTS_V2=true
   +        OPENDAL_S3_DISABLE_COPY_WITH_IF_MATCH=true
            EOF
   diff --git a/.github/services/s3/minio_s3_with_versioning/action.yml b/.github/services/s3/minio_s3_with_versioning/action.yml
   index e2db993a8..8143df537 100644
   --- a/.github/services/s3/minio_s3_with_versioning/action.yml
   +++ b/.github/services/s3/minio_s3_with_versioning/action.yml
   @@ -45,4 +45,5 @@ runs:
            OPENDAL_S3_SECRET_ACCESS_KEY=minioadmin
            OPENDAL_S3_REGION=us-east-1
            OPENDAL_S3_ENABLE_VERSIONING=true
   +        OPENDAL_S3_DISABLE_COPY_WITH_IF_MATCH=true
            EOF
   diff --git a/core/services/s3/src/backend.rs b/core/services/s3/src/backend.rs
   index 3fd62ee69..36e17dfe2 100644
   --- a/core/services/s3/src/backend.rs
   +++ b/core/services/s3/src/backend.rs
   @@ -555,6 +555,14 @@ impl S3Builder {
            self
        }
    
   +    /// Disable copy with if match so that opendal will not send copy request with if match headers.
   +    ///
   +    /// For example, MinIO doesn't support copy with if match.
   +    pub fn disable_copy_with_if_match(mut self) -> Self {
   +        self.config.disable_copy_with_if_match = true;
   +        self
   +    }
   +
        /// Enable write with append so that opendal will send write request with append headers.
        pub fn enable_write_with_append(mut self) -> Self {
            self.config.enable_write_with_append = true;
   diff --git a/core/services/s3/src/config.rs b/core/services/s3/src/config.rs
   index 8c219de6c..cba6491b5 100644
   --- a/core/services/s3/src/config.rs
   +++ b/core/services/s3/src/config.rs
   @@ -210,6 +210,11 @@ pub struct S3Config {
        /// For example, Ceph RADOS S3 doesn't support write with if matched.
        pub disable_write_with_if_match: bool,
    
   +    /// Disable copy with if match so that opendal will not send copy request with if match headers.
   +    ///
   +    /// For example, MinIO doesn't support copy with if match.
   +    pub disable_copy_with_if_match: bool,
   +
        /// Enable write with append so that opendal will send write request with append headers.
        pub enable_write_with_append: bool,
    
   diff --git a/core/services/s3/src/docs.md b/core/services/s3/src/docs.md
   index df279f35b..e0f47efd3 100644
   --- a/core/services/s3/src/docs.md
   +++ b/core/services/s3/src/docs.md
   @@ -30,6 +30,7 @@ This service can be used to:
    - `disable_config_load`: Disable aws config load from env.
    - `enable_virtual_host_style`: Enable virtual host style.
    - `disable_write_with_if_match`: Disable write with if match.
   +- `disable_copy_with_if_match`: Disable copy with if match / if none match. Some S3-compatible services like MinIO don't support conditional copy.
    - `enable_request_payer`: Enable the request payer for backend.
    - `default_acl`: Define the default access control list (ACL) when creating a new object. Note that some s3 services like minio do not support this option.
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org