You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "yinzixin (via GitHub)" <gi...@apache.org> on 2026/05/19 12:00:37 UTC
[PR] [feature](s3) Support S3 Express One Zone (AWS SDK 1.11.400 + CRC32C checksum) [doris]
yinzixin opened a new pull request, #63409:
URL: https://github.com/apache/doris/pull/63409
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Doris currently cannot read from or write to **Amazon S3 Express One Zone** buckets. Two distinct gaps block this:
1. **AWS SDK is too old / wrong client builder.** The bundled SDK is `aws-sdk-cpp 1.11.219`, which predates S3 Express support. The S3 client is also constructed with the legacy `Aws::Client::ClientConfiguration` path, which bypasses the SDK's endpoint-rules resolver — so even on a newer SDK, the `--x-s3` bucket suffix would not trigger `CreateSession` or the `s3express` SigV4 service name.
2. **`Content-MD5` is unconditionally attached to every PutObject and UploadPart.** S3 Express One Zone rejects `Content-MD5` with `501 NotImplemented` and instead requires a flexible checksum (CRC32 / CRC32C / SHA1 / SHA256).
This PR fixes both:
**Commit 1 — `feat: add support for CRC32C checksum in S3 uploads and disable Content-MD5 for S3 Express`**
- Adds a new BE config `s3_disable_content_md5` (default `false`) and auto-enables it when the endpoint string contains `s3express`.
- When disabled, `PutObject` / `UploadPart` send `x-amz-checksum-crc32c` (computed via the existing `crc32c` thirdparty lib) instead of `Content-MD5`. Behavior for non-S3-Express endpoints is unchanged.
- Threads the endpoint string into `S3ObjStorageClient` so the per-client flag can be set at construction time.
**Commit 2 — `Upgrade AWS SDK and use new create client to support S3 express`**
- Bumps `aws-sdk-cpp` from `1.11.219` to `1.11.400` in `thirdparty/vars.sh` (with refreshed MD5 / URL).
- Switches `S3ClientFactory::_create_s3_client` to `Aws::S3::S3ClientConfiguration` + `S3EndpointProvider`, which activates the endpoint-rules resolver required for S3 Express (`CreateSession`, `s3express` SigV4 service name).
- Detects S3 Express buckets by the `--x-s3` suffix or `s3express` endpoint substring; for those buckets we skip `endpointOverride` (so the SDK resolves the bucket-specific endpoint) and clear `disableS3ExpressAuth`. For all other buckets the behavior is unchanged.
- Adds the `vaLog` override in `DorisAWSLogger` that 1.11.400's `LogSystemInterface` now requires.
### Release note
Support reading from / writing to Amazon S3 Express One Zone buckets. A new BE config `s3_disable_content_md5` is added to send a CRC32C checksum instead of `Content-MD5` on uploads; it is auto-enabled for endpoints whose hostname contains `s3express`.
### Check List (For Author)
- Test
- [x] Manual test (add detailed scripts or steps below)
- Verified PutObject / multipart upload against an `--x-s3` (S3 Express One Zone) bucket in `us-east-1`: `CreateSession` is issued, requests are signed as `s3express`, and uploads succeed with CRC32C in place of `Content-MD5`.
- Verified regression-free behavior against a regular S3 bucket (with and without `endpointOverride`) and against an S3-compatible MinIO endpoint — `Content-MD5` is still sent and uploads succeed.
- Behavior changed:
- [x] Yes.
- `s3_disable_content_md5=true` (or any endpoint matching `s3express`) replaces `Content-MD5` with CRC32C on `PutObject` / `UploadPart`. Default is unchanged for non-S3-Express endpoints.
- AWS SDK upgraded from 1.11.219 → 1.11.400; the BE links a rebuilt thirdparty bundle.
- Does this need documentation?
- [ ] Yes — a short note for the new `s3_disable_content_md5` BE config and an S3 Express One Zone usage example. Will open the doris-website PR once this is reviewed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](s3) Support S3 Express One Zone (AWS SDK 1.11.400 + CRC32C checksum) [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #63409:
URL: https://github.com/apache/doris/pull/63409#issuecomment-4764329753
Hi @yinzixin, any update?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](s3) Support S3 Express One Zone (AWS SDK 1.11.400 + CRC32C checksum) [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63409:
URL: https://github.com/apache/doris/pull/63409#issuecomment-4487502035
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](s3) Support S3 Express One Zone (AWS SDK 1.11.400 + CRC32C checksum) [doris]
Posted by "yinzixin (via GitHub)" <gi...@apache.org>.
yinzixin commented on PR #63409:
URL: https://github.com/apache/doris/pull/63409#issuecomment-4527174470
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](s3) Support S3 Express One Zone (AWS SDK 1.11.400 + CRC32C checksum) [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63409:
URL: https://github.com/apache/doris/pull/63409#issuecomment-4490727162
# BE Regression && UT Coverage Report
Increment line coverage `72.73% (32/44)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63409_c2174ff23bface06bf664f1dc17f45c20788bc5e_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63409_c2174ff23bface06bf664f1dc17f45c20788bc5e_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.84% (27238/37916) |
| Line Coverage | 55.21% (290256/525775) |
| Region Coverage | 52.46% (242718/462702) |
| Branch Coverage | 53.64% (104293/194417) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](s3) Support S3 Express One Zone (AWS SDK 1.11.400 + CRC32C checksum) [doris]
Posted by "yinzixin (via GitHub)" <gi...@apache.org>.
yinzixin commented on PR #63409:
URL: https://github.com/apache/doris/pull/63409#issuecomment-4487548340
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](s3) Support S3 Express One Zone (AWS SDK 1.11.400 + CRC32C checksum) [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #63409:
URL: https://github.com/apache/doris/pull/63409#issuecomment-4590395098
@yinzixin Please fix code formatter error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](s3) Support S3 Express One Zone (AWS SDK 1.11.400 + CRC32C checksum) [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #63409:
URL: https://github.com/apache/doris/pull/63409#discussion_r3333277413
##########
be/src/common/config.cpp:
##########
@@ -1497,6 +1497,8 @@ DEFINE_mInt32(s3_read_max_wait_time_ms, "800");
DEFINE_mBool(enable_s3_object_check_after_upload, "true");
DEFINE_mInt32(aws_client_request_timeout_ms, "30000");
+DEFINE_mBool(s3_disable_content_md5, "false");
Review Comment:
suggest naming
s3_disable_content_md5 -> enable_s3_content_md5
default to true.
##########
be/src/io/fs/s3_obj_storage_client.cpp:
##########
@@ -116,6 +120,31 @@ using namespace Aws::S3::Model;
static constexpr int S3_REQUEST_THRESHOLD_MS = 5000;
+namespace {
+// S3 Express One Zone endpoints follow the pattern
+// "*.s3express-<zone>.<region>.amazonaws.com" — substring match is sufficient
+// for both the gateway and the s3express subdomain forms.
+bool is_s3_express_endpoint(const std::string& endpoint) {
Review Comment:
what's the difference to
```
const bool is_s3_express = s3_conf.endpoi
```
in s3_util.cpp
it seems better to abstract a static/global function/utility to check if the given thing/context is s3 express, something like.
```
is_s3_express(endpoint, bucket_name, ...) {
if (one of the input is in s3 express context) return true;
else return false;
}
```
##########
be/src/util/s3_util.cpp:
##########
@@ -528,12 +530,23 @@ std::shared_ptr<io::ObjStorageClient> S3ClientFactory::_create_s3_client(
aws_config.retryStrategy = std::make_shared<S3CustomRetryStrategy>(
config::max_s3_client_retry /*scaleFactor = 25*/);
- std::shared_ptr<Aws::S3::S3Client> new_client = std::make_shared<Aws::S3::S3Client>(
- get_aws_credentials_provider(s3_conf), std::move(aws_config),
- Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never,
- s3_conf.use_virtual_addressing);
+ // S3 Express buckets are identified by the --x-s3 suffix or an s3express endpoint.
+ // Skip endpointOverride for these so the SDK resolves the bucket-specific endpoint
+ // and manages CreateSession automatically. For all other buckets, keep the override.
+ const bool is_s3_express = s3_conf.endpoint.find("s3express") != std::string::npos ||
Review Comment:
this is not robust, e.g.
"--x-s3" is the bucket name suffix, however, `s3_conf.bucket.find("--x-s3") != std::string::npos` does not reflect the truth.
BTW, what is the formal way to support s3 express, better to find some official references.
##########
be/src/common/config.cpp:
##########
@@ -1497,6 +1497,8 @@ DEFINE_mInt32(s3_read_max_wait_time_ms, "800");
DEFINE_mBool(enable_s3_object_check_after_upload, "true");
DEFINE_mInt32(aws_client_request_timeout_ms, "30000");
+DEFINE_mBool(s3_disable_content_md5, "false");
Review Comment:
or this configuration should be something like
`s3_content_checksum_method = md5`
should change to crc32
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](s3) Support S3 Express One Zone (AWS SDK 1.11.400 + CRC32C checksum) [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63409:
URL: https://github.com/apache/doris/pull/63409#issuecomment-4490586553
# BE UT Coverage Report
Increment line coverage `72.73% (32/44)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/c2174ff23bface06bf664f1dc17f45c20788bc5e_c2174ff23bface06bf664f1dc17f45c20788bc5e/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/c2174ff23bface06bf664f1dc17f45c20788bc5e_c2174ff23bface06bf664f1dc17f45c20788bc5e/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.56% (20735/38714) |
| Line Coverage | 37.21% (196154/527128) |
| Region Coverage | 33.51% (153557/458281) |
| Branch Coverage | 34.59% (66994/193690) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](s3) Support S3 Express One Zone (AWS SDK 1.11.400 + CRC32C checksum) [doris]
Posted by "0AyanamiRei (via GitHub)" <gi...@apache.org>.
0AyanamiRei commented on PR #63409:
URL: https://github.com/apache/doris/pull/63409#issuecomment-5339440968
Hi @yinzixin, thanks for your work on this! We’ll take over the remaining work in #65504, so you don’t need to continue following up on this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org