[I] new feature: copy_with should be able to replace user metadata and content type (S3 x-amz-metadata-directive REPLACE) [opendal]

["dmxlol (via GitHub)" <gi...@apache.org>]

dmxlol opened a new issue, #8430:
URL: https://github.com/apache/opendal/issues/8430

   ### Feature Description
   
   Add two options to `CopyOptions`, with matching `copy_with` / `copier_with` builder methods and `OpCopy` getters:
   
   ```rust
   pub struct CopyOptions {
       // existing fields ...
       pub user_metadata: Option<HashMap<String, String>>,
       pub content_type: Option<String>,
   }
   
   op.copy_with("a.txt", "a.txt")
       .user_metadata([("mtime".to_string(), "1700000000".to_string())])
       .content_type("text/plain")
       .await?;
   ```
   
   Semantics, following S3's CopyObject:
   
   - When either option is set, the destination gets exactly the given values and does not inherit metadata from the source. Anything left unset takes the service default.
   - With either option set, `from == to` is allowed and updates the metadata in place.
   - Two new capability flags, `copy_with_user_metadata` and `copy_with_content_type`. A service that does not advertise a flag returns `Unsupported`, as the conditional copy options do today; the options are never dropped silently.
   
   For the s3 service the options add `x-amz-metadata-directive: REPLACE`, `Content-Type` and the `x-amz-meta-*` headers to the CopyObject request, and the same headers to CreateMultipartUpload for a multipart copy.
   
   ### Problem and Solution
   
   I am building a self-hosted file server (WebDAV, SFTP, FTPS) on top of the s3 service. Every file is a plain object and its mtime lives in `x-amz-meta-mtime`. Protocol clients set the mtime after the upload is done: WebDAV PROPPATCH, SFTP SETSTAT, `rsync -t` and `scp -p` do it for every file.
   
   S3 can change an object's metadata with one zero-body request: CopyObject from the key to itself with `x-amz-metadata-directive: REPLACE`. The S3 API reference documents this on the CopyObject page:
   
   - `x-amz-metadata-directive`: "Specifies whether the metadata is copied from the source object or replaced with metadata that's provided in the request." Valid values `COPY | REPLACE`. <https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html#API_CopyObject_RequestSyntax>
   - The same page: "You can use the CopyObject action to change the metadata of an object" by copying it to itself with the directive set to REPLACE, and user metadata is sent as `x-amz-meta-*` request headers. <https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html>
   - User-defined metadata and how CopyObject treats it: <https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html#UserMetadata>
   - Multipart copies do not inherit metadata from the source; it is set on CreateMultipartUpload: <https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html>
   
   OpenDAL cannot send that request today:
   
   - `CopyOptions` has no metadata fields, so the s3 copier always sends COPY semantics (the source metadata is carried over, new headers are ignored).
   - `copy(from, to)` with `from == to` returns `IsSameFile` unless `source_version` is set.
   
   So the only way to change metadata through OpenDAL is to read the object and write it back. For a large file, setting its mtime costs a full download and re-upload.
   
   The proposed change follows the shape of the existing copy options: an options field, a builder method, an `OpCopy` getter, a capability flag checked by `CorrectnessCheckLayer`, and the s3 request builder.
   
   ### Additional Context
   
   - Precedent: #7115 added the conditional copy headers along the same path.
   - Related: #7825 discusses copy semantics and notes that S3 copy can either copy or replace metadata.
   - `ComposeOptions` already carries `user_metadata` and `content_type` for the destination; this brings copy in line with it.
   
   ### Are you willing to contribute to the development of this feature?
   
   - [x] Yes, I am willing to contribute to the development of this feature.
   
   I have a draft with unit tests and a behavior test. It passes the s3 behavior suite against Silo and Garage, and the in-place metadata test against AWS S3. I will open the PR once this issue is accepted.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org