You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "andyhuangdev (via GitHub)" <gi...@apache.org> on 2026/10/16 08:14:23 UTC
[PR] HDDS-17055. Reject malformed x-amz-content-sha256 before writing the … [ozone]
andyhuangdev opened a new pull request, #11562:
URL: https://github.com/apache/ozone/pull/11562
## What changes were proposed in this pull request?
S3 Gateway currently reads and writes the request body before rejecting a malformed x-amz-content-sha256 value with XAmzContentSHA256Mismatch. Such values cannot match a valid SHA-256 digest, so processing the body wastes I/O and network bandwidth.
This patch adds format validation to the shared header validation method. Hash values that are not exactly 64 hexadecimal characters are rejected with InvalidArgument before the body is read or a write stream is opened. Existing unsigned and streaming payload handling remains unchanged. Well-formed hashes that do not match the body continue to return XAmzContentSHA256Mismatch.
Existing test fixtures using malformed placeholder hashes are updated to use well-formed values so those tests continue exercising their intended behavior.
## What is the link to the Apache JIRA
[HDDS-17055](https://issues.apache.org/jira/browse/HDDS-17055)
## How was this patch tested?
- Confirmed that eight new regression cases failed before the fix.
- Added header validation tests and tests confirming that malformed headers are rejected without reading the body for regular, datastream, and multipart uploads.
- Verified successful uploads with matching hashes and existing mismatch behavior.
- Ran 674 tests covering the new behavior and affected test classes; all passed.
- Ran the project checkstyle script and git diff --check; both passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17055. Reject malformed x-amz-content-sha256 before writing the body [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #11562:
URL: https://github.com/apache/ozone/pull/11562#issuecomment-6481835572
> will (hopefully) allow skipping most integration tests for S3-only changes.
It turns out there is another prerequisite: #11830.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17055. Reject malformed x-amz-content-sha256 before writing the body [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #11562:
URL: https://github.com/apache/ozone/pull/11562#discussion_r4394927392
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3Utils.java:
##########
@@ -189,7 +190,17 @@ public static String validateSignatureHeader(HttpHeaders headers, String resourc
throw ex;
}
- return xAmzContentSha256Header;
+ S3ContentSha256Type payloadType = S3ContentSha256Type.fromHeaderValue(xAmzContentSha256Header);
+ if (payloadType.isUnsigned() || payloadType.isStreaming()) {
+ return xAmzContentSha256Header;
+ }
+ if (!xAmzContentSha256Header.matches("[0-9a-fA-F]{64}")) {
Review Comment:
nit: `String.matches` compiles the regex on every call, and a regular PUT calls `validateSignatureHeader` twice. Use a precompiled constant:
```suggestion
if (!SHA256_HEX_PATTERN.matcher(xAmzContentSha256Header).matches()) {
```
```diff
+ private static final Pattern SHA256_HEX_PATTERN = Pattern.compile("[0-9a-fA-F]{64}");
```
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3Utils.java:
##########
@@ -189,7 +190,17 @@ public static String validateSignatureHeader(HttpHeaders headers, String resourc
throw ex;
}
- return xAmzContentSha256Header;
+ S3ContentSha256Type payloadType = S3ContentSha256Type.fromHeaderValue(xAmzContentSha256Header);
+ if (payloadType.isUnsigned() || payloadType.isStreaming()) {
+ return xAmzContentSha256Header;
+ }
+ if (!xAmzContentSha256Header.matches("[0-9a-fA-F]{64}")) {
+ OS3Exception ex = S3ErrorTable.newError(S3ErrorTable.INVALID_ARGUMENT, resource);
+ ex.setErrorMessage("The " + X_AMZ_CONTENT_SHA256 + " header must contain 64 hexadecimal characters");
Review Comment:
nit: the other two errors in this method start with `"An error occurred (InvalidArgument): "`. Keep the same format:
```suggestion
ex.setErrorMessage("An error occurred (InvalidArgument): " +
"The " + X_AMZ_CONTENT_SHA256 + " header must contain 64 hexadecimal characters");
```
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3Utils.java:
##########
@@ -189,7 +190,17 @@ public static String validateSignatureHeader(HttpHeaders headers, String resourc
throw ex;
}
- return xAmzContentSha256Header;
+ S3ContentSha256Type payloadType = S3ContentSha256Type.fromHeaderValue(xAmzContentSha256Header);
+ if (payloadType.isUnsigned() || payloadType.isStreaming()) {
+ return xAmzContentSha256Header;
+ }
+ if (!xAmzContentSha256Header.matches("[0-9a-fA-F]{64}")) {
+ OS3Exception ex = S3ErrorTable.newError(S3ErrorTable.INVALID_ARGUMENT, resource);
+ ex.setErrorMessage("The " + X_AMZ_CONTENT_SHA256 + " header must contain 64 hexadecimal characters");
+ throw ex;
+ }
+
+ return xAmzContentSha256Header.toLowerCase(Locale.ROOT);
Review Comment:
nit: the uppercase case is handled twice. `S3ObjectWriteGuard.checkContentSha256` now uses `equalsIgnoreCase`, and that alone covers all four call sites, while this lowercasing does not reach the two multipart paths anyway (`ObjectEndpoint` part upload and `ObjectEndpointStreaming.createMultipartKey` pass the raw header to `checkContentSha256`).
Return the header unchanged and drop the `Locale` import; `testNormalizeContentSha256` can go too, since the uppercase upload tests in `TestObjectPut` and `TestPartUpload` already cover it end to end.
```suggestion
return xAmzContentSha256Header;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17055. Reject malformed x-amz-content-sha256 before writing the body [ozone]
Posted by "adoroszlai (via GitHub) via issues" <is...@ozone.apache.org>.
adoroszlai commented on PR #11562:
URL: https://github.com/apache/ozone/pull/11562#issuecomment-6533820981
Thanks @andyhuangdev for the patch, @smengcl for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17055. Reject malformed x-amz-content-sha256 before writing the body [ozone]
Posted by "adoroszlai (via GitHub) via issues" <is...@ozone.apache.org>.
adoroszlai merged PR #11562:
URL: https://github.com/apache/ozone/pull/11562
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17055. Reject malformed x-amz-content-sha256 before writing the body [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #11562:
URL: https://github.com/apache/ozone/pull/11562#discussion_r4301717937
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3Utils.java:
##########
@@ -217,6 +217,13 @@ public static String validateSignatureHeader(HttpHeaders headers, String resourc
throw ex;
}
+ if (!hasUnsignedPayload(xAmzContentSha256Header) && !hasMultiChunksPayload(xAmzContentSha256Header)
+ && !xAmzContentSha256Header.matches("[0-9a-fA-F]{64}")) {
Review Comment:
The new format check deliberately accepts uppercase hex, and the added test covers it, but the upload paths still compare the header with the lowercase digest using String.equals (for example S3ObjectWriteGuard.checkContentSha256).
Thus a header containing the uppercase form of the correct SHA-256 passes this validation and is then rejected as XAmzContentSHA256Mismatch, so the accepted format is not actually usable.
Normalize the expected value (or compare case-insensitively) before returning/validating it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17055. Reject malformed x-amz-content-sha256 before writing the body [ozone]
Posted by "andyhuangdev (via GitHub)" <gi...@apache.org>.
andyhuangdev commented on code in PR #11562:
URL: https://github.com/apache/ozone/pull/11562#discussion_r4301761324
##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3Utils.java:
##########
@@ -217,6 +217,13 @@ public static String validateSignatureHeader(HttpHeaders headers, String resourc
throw ex;
}
+ if (!hasUnsignedPayload(xAmzContentSha256Header) && !hasMultiChunksPayload(xAmzContentSha256Header)
+ && !xAmzContentSha256Header.matches("[0-9a-fA-F]{64}")) {
Review Comment:
Agreed. I’ll normalize validated hex digests to lowercase in validateSignatureHeader, preserving the unsigned and streaming constants, and add upload tests for uppercase and mixed-case digests across regular, datastream, and multipart uploads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17055. Reject malformed x-amz-content-sha256 before writing the body [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #11562:
URL: https://github.com/apache/ozone/pull/11562#issuecomment-6383940634
Thanks for the patch. I would like to defer merging this after #11735, which will (hopefully) allow skipping most integration tests for S3-only changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org