You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "YuangGao (via GitHub)" <gi...@apache.org> on 2026/05/14 04:52:24 UTC
[PR] feat(services/azdls): Add user defined metadata support [opendal]
YuangGao opened a new pull request, #7528:
URL: https://github.com/apache/opendal/pull/7528
# Which issue does this PR close?
Part of #4842.
# Rationale for this change
PR #6842 added user-defined metadata support for `azdls` using the `x-ms-meta-*` header convention from Azure Blob Storage, but was reverted on the same day because ADLS Gen2 silently ignores those headers. ADLS Gen2 instead exposes user properties via a single `x-ms-properties` header formatted as comma-separated `name=base64(value)` pairs ([Path Create REST API](https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/create)). This PR uses the correct protocol so `write_with_user_metadata` and `stat_with_user_metadata` actually round-trip on a real ADLS Gen2 backend.
# What changes are included in this PR?
- Add `X_MS_PROPERTIES` constant and `encode_user_metadata` / `decode_user_metadata` helpers in `core.rs`.
- Inject the encoded header in `azdls_create`.
- Parse the header in `azdls_stat_metadata`, with explicit errors on malformed pairs, invalid base64, or non-UTF-8 values.
- Enable `write_with_user_metadata: true` in the capability declaration.
- Add 7 unit tests covering round-trip, special characters (`=`, `,`), UTF-8, malformed input, and the three error paths — the encoding-layer regression coverage that #6842 lacked.
# Are there any user-facing changes?
Yes — `write_with().user_metadata(...)` is now persisted on ADLS Gen2 and round-trips through `stat()`. Not a breaking change: the capability is opt-in via the existing `OpWrite` API.
# AI Usage Statement
AI-assisted implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(services/azdls): Add user defined metadata support [opendal]
Posted by "YuangGao (via GitHub)" <gi...@apache.org>.
YuangGao commented on code in PR #7528:
URL: https://github.com/apache/opendal/pull/7528#discussion_r3251949378
##########
core/services/azdls/src/core.rs:
##########
@@ -481,3 +500,106 @@ impl AzdlsCore {
}
}
}
+
+fn encode_user_metadata(metadata: &HashMap<String, String>) -> String {
+ metadata
+ .iter()
+ .map(|(k, v)| format!("{k}={}", BASE64_STANDARD.encode(v)))
+ .collect::<Vec<_>>()
+ .join(",")
+}
+
+/// Empty pairs (e.g. trailing commas) are skipped; everything else is fatal.
+fn decode_user_metadata(value: &str) -> Result<HashMap<String, String>> {
Review Comment:
that makes sense, updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(services/azdls): Add user defined metadata support [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo merged PR #7528:
URL: https://github.com/apache/opendal/pull/7528
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(services/azdls): Add user defined metadata support [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #7528:
URL: https://github.com/apache/opendal/pull/7528#discussion_r3248618291
##########
core/services/azdls/src/core.rs:
##########
@@ -481,3 +500,106 @@ impl AzdlsCore {
}
}
}
+
+fn encode_user_metadata(metadata: &HashMap<String, String>) -> String {
+ metadata
+ .iter()
+ .map(|(k, v)| format!("{k}={}", BASE64_STANDARD.encode(v)))
+ .collect::<Vec<_>>()
+ .join(",")
+}
+
+/// Empty pairs (e.g. trailing commas) are skipped; everything else is fatal.
+fn decode_user_metadata(value: &str) -> Result<HashMap<String, String>> {
Review Comment:
Maybe we just let user to do this. I remember that other services also didn't handle this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(services/azdls): Add user defined metadata support [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #7528:
URL: https://github.com/apache/opendal/pull/7528#issuecomment-4467426565
Hi @YuangGao, test failed after this change: https://github.com/apache/opendal/actions/runs/25964199525/job/76324672823
Would you like to take a look and fix it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(services/azdls): Add user defined metadata support [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #7528:
URL: https://github.com/apache/opendal/pull/7528#issuecomment-4467500172
The post-merge Behavior Test failure was caused by `azdls` stat still using `action=getStatus`. Azure Data Lake Storage Gen2 only returns system-defined properties for Get Status, so `x-ms-properties` was not returned even though this PR wrote it successfully.
I opened #7532 to switch `azdls` stat back to the default Get Properties API so user metadata is returned by `stat`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(services/azdls): Add user defined metadata support [opendal]
Posted by "YuangGao (via GitHub)" <gi...@apache.org>.
YuangGao commented on PR #7528:
URL: https://github.com/apache/opendal/pull/7528#issuecomment-4467512471
@Xuanwo yes i will take a look and fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org