You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Wei-Chiu Chuang (Jira)" <ji...@apache.org> on 2026/08/04 16:58:00 UTC
[jira] [Updated] (HDDS-12935) Support unsigned chunked upload with STREAMING-UNSIGNED-PAYLOAD-TRAILER
[ https://issues.apache.org/jira/browse/HDDS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wei-Chiu Chuang updated HDDS-12935:
-----------------------------------
Release Note: Fixed S3 Gateway signature validation for HTTPS chunked uploads using STREAMING-UNSIGNED-PAYLOAD-TRAILER, which had caused PutObject and other AWS SDK v2 requests to fail with signature mismatch errors.
> Support unsigned chunked upload with STREAMING-UNSIGNED-PAYLOAD-TRAILER
> -----------------------------------------------------------------------
>
> Key: HDDS-12935
> URL: https://issues.apache.org/jira/browse/HDDS-12935
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Critical
> Labels: pull-request-available
> Fix For: 2.1.0, 2.0.1
>
>
> HDDS-12488 fixes the issue where the AWS checksum trailer data is silently appended to the final payload. This works for http endpoint which will by default provide with signed payload (i.e. x-amz-content-sha256 is set to the payload signature).
> When the request is sent against https endpoint, the x-amz-content-sha256 is set to STREAMING-UNSIGNED-PAYLOAD-TRAILER , probably because https protocol already encrypts the HTTP payload and there is no need to further encryption using signature. HDDS-12488 handled it by using string "UNSIGNED-PAYLOAD: when building the canonical request (StringToSignProducer#buildCanonicalRequest) as specified in [https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html]. However, this causes the mismatch between the AWS SDK canonical request and the calculated canonical request which results in signature mismatch and all PutObject requests for the newer AWS SDK version to fail.
> Did manual testing
> AWS CLI canonical request
> {code:java}
> PUT
> /ivan-bucket/test4.txt
> content-encoding:aws-chunked
> host:<redacted>
> x-amz-content-sha256:STREAMING-UNSIGNED-PAYLOAD-TRAILER
> x-amz-date:20250430T071131Z
> x-amz-decoded-content-length:5
> x-amz-sdk-checksum-algorithm:CRC64NVME
> x-amz-trailer:x-amz-checksum-crc64nvme
> content-encoding;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-trailer
> STREAMING-UNSIGNED-PAYLOAD-TRAILER
> {code}
> Ozone canonical request
> {code:java}
> PUT
> /ivan-bucket/test4.txt
> content-encoding:aws-chunked
> host:<redacted>
> x-amz-content-sha256:STREAMING-UNSIGNED-PAYLOAD-TRAILER
> x-amz-date:20250430T071131Z
> x-amz-decoded-content-length:5
> x-amz-sdk-checksum-algorithm:CRC64NVME
> x-amz-trailer:x-amz-checksum-crc64nvme
> content-encoding;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-trailer
> UNSIGNED-PAYLOAD
> {code}
> AWS CLI StrToSign
> {code:java}
> AWS4-HMAC-SHA256
> 20250430T065600Z
> 20250430/us-east-1/s3/aws4_request
> a3b489c10cf8e4878c7d7b870d3eda15c9c17f52541bd4742361a538c2b458fe
> {code}
> Ozone StrToSign
> {code:java}
> AWS4-HMAC-SHA256
> 20250430T065600Z
> 20250430/us-east-1/s3/aws4_request
> cb995d592067f026dc30bb4416340afa7e9d38abee57e8e8a44a5604380a22ab
> {code}
> The correct behavior seems to use the "STREAMING-UNSIGNED-PAYLOAD-TRAILER" instead of "UNSIGNED-PAYLOAD" when building the canonical request.
> Ref: [https://github.com/aws/aws-sdk-java-v2/blob/f6adeaa5b24a4c203106d122647537f83a5ecfbc/core/auth/src/main/java/software/amazon/awssdk/auth/signer/internal/AbstractAwsS3V4Signer.java#L206]
> Additionally, unsigned chunked upload (STREAMING-UNSIGNED-PAYLOAD-TRAILER) will omit the "chunk-signature". For example:
> {code:java}
> 5
> test
> 0
> x-amz-checksum-crc64nvme:2wstOANdZ/o=
> {code}
> We have SignedChunksInputStream, but it only works for signed chunked upload. This might requires us to create a new InputStream to handle unsigned chunk uploads. Additionally, currently STREAMING-UNSIGNED-PAYLOAD-TRAILER is handled as normal single chunk input stream which causes unexpected key content.
> Note that some of the behaviors are not well-documented, and might not catch some edge cases.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org