You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "haoqixu (via GitHub)" <gi...@apache.org> on 2024/08/22 04:11:35 UTC
[PR] feat(services-s3): support user defined metadata [opendal]
haoqixu opened a new pull request, #5030:
URL: https://github.com/apache/opendal/pull/5030
# Which issue does this PR close?
Support user defined metadata for `services-s3`.
Part of #4842.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(services-s3): support user defined metadata [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #5030:
URL: https://github.com/apache/opendal/pull/5030#discussion_r1726534537
##########
core/src/services/s3/backend.rs:
##########
@@ -1116,6 +1118,25 @@ impl Access for S3Backend {
let headers = resp.headers();
let mut meta = parse_into_metadata(path, headers)?;
+ let user_meta: HashMap<String, String> = headers
+ .iter()
+ .filter(|(name, _)| name.as_str().starts_with(constants::X_AMZ_META_PREFIX))
Review Comment:
How about merging this filter in the next `filter_map`?
##########
core/src/services/s3/backend.rs:
##########
@@ -1116,6 +1118,25 @@ impl Access for S3Backend {
let headers = resp.headers();
let mut meta = parse_into_metadata(path, headers)?;
+ let user_meta: HashMap<String, String> = headers
+ .iter()
+ .filter(|(name, _)| name.as_str().starts_with(constants::X_AMZ_META_PREFIX))
+ .filter_map(|(name, _)| {
+ parse_header_to_str(headers, name)
+ .unwrap_or(None)
+ .map(|val| {
+ let stripped = name
Review Comment:
How about using:
```rust
let Some(stripped) = name.as_str().strip_prefix(constants::X_AMZ_META_PREFIX) {
return None;
}
parse_header_to_str(headers, name).balabala()
```
We can save an extra check of prefix.
##########
core/src/services/s3/core.rs:
##########
@@ -192,6 +194,51 @@ impl S3Core {
self.client.send(req).await
}
+ pub fn insert_user_metadata_headers(
Review Comment:
I perfer filling the header directly with `user_metadata` without check.
##########
core/src/services/s3/core.rs:
##########
@@ -456,6 +503,9 @@ impl S3Core {
req = req.header(HeaderName::from_static(constants::X_AMZ_STORAGE_CLASS), v);
}
+ // Set user metadata headers.
+ req = self.insert_user_metadata_headers(req, args)?;
Review Comment:
We can insert in header here directly.
##########
core/src/services/s3/core.rs:
##########
@@ -192,6 +194,51 @@ impl S3Core {
self.client.send(req).await
}
+ pub fn insert_user_metadata_headers(
+ &self,
+ mut req: http::request::Builder,
+ args: &OpWrite,
+ ) -> Result<http::request::Builder> {
+ if let Some(user_metadata) = args.user_metadata() {
+ for (key, value) in user_metadata {
+ // before insert user defined metadata header, add prefix to the header name
+ if !self.check_user_metadata_key(key) {
+ return Err(Error::new(
+ ErrorKind::Unsupported,
+ "the format of the user metadata key is invalid, please refer the document",
+ ));
+ }
+
+ req = req.header(format!("{}{}", constants::X_AMZ_META_PREFIX, key), value)
+ }
+ }
+ Ok(req)
+ }
+
+ // The AWS S3 documentation does not cover valid characters for the metadata key.
+ // But it should at least be restricted to valid characters in HTTP Header Name
+ // as per https://www.rfc-editor.org/rfc/rfc9110.html#name-tokens.
+ fn check_user_metadata_key(&self, key: &str) -> bool {
Review Comment:
Let's simplify things removing this check for now and let users perform the check. I believe most users won't use `*` as a metadata key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(services-s3): support user defined metadata [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #5030:
URL: https://github.com/apache/opendal/pull/5030#discussion_r1726482175
##########
core/Cargo.toml:
##########
@@ -380,6 +382,8 @@ prometheus-client = { version = "0.22.2", optional = true }
tracing = { version = "0.1", optional = true }
# for layers-dtrace
probe = { version = "0.5.1", optional = true }
+rfc2047-decoder = { version = "1.0.5", optional = true }
+email-encoding = { version = "0.3.0", optional = true }
Review Comment:
Thank you very much for your effort. However, I discovered that even AWS's own Go SDK doesn't handle this: https://github.com/aws/aws-sdk-go-v2/blob/38fe812f7abf976b6b1a5122c0c3bf303502cfc5/service/s3/deserializers.go#L7167-L7175
I prefer to keep things simple since it's quite rare for users to include non-ASCII characters in metadata. We can also make it clear in our API that users should encode those values themselves.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(services-s3): support user defined metadata [opendal]
Posted by "haoqixu (via GitHub)" <gi...@apache.org>.
haoqixu commented on code in PR #5030:
URL: https://github.com/apache/opendal/pull/5030#discussion_r1726532946
##########
core/Cargo.toml:
##########
@@ -380,6 +382,8 @@ prometheus-client = { version = "0.22.2", optional = true }
tracing = { version = "0.1", optional = true }
# for layers-dtrace
probe = { version = "0.5.1", optional = true }
+rfc2047-decoder = { version = "1.0.5", optional = true }
+email-encoding = { version = "0.3.0", optional = true }
Review Comment:
That sounds reasonable. Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core/services-s3): support user defined metadata [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo merged PR #5030:
URL: https://github.com/apache/opendal/pull/5030
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org