You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "chungen0126 (via GitHub)" <gi...@apache.org> on 2026/06/24 01:00:35 UTC
[PR] HDDS-15652. Support S3 signed multi chunks payload verification [ozone]
chungen0126 opened a new pull request, #10006:
URL: https://github.com/apache/ozone/pull/10006
## What changes were proposed in this pull request?
This is a design doc for supporting s3 multi chunks upload verification.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15652
## How was this patch tested?
None.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3196028251
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
Review Comment:
> Just to clarify, the verification process require the secret key. The official AWS documentation explains how the signing key is generated. See https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html.
At step 2.
Yes you're right we do need the secret key. Thanks for the clarification.
> One-way Hashing: The key S3G receives is a derived key computed through multiple iterations of HMAC-SHA256. Since HMAC is a one-way cryptographic function, even if this derived key is compromised, it is impossible to get the original secret key
Yes, I think we can use the signing key to mask the secret key (since it's hashed). The main security point is that the response returned by the OM should not include a plain text secret key.
> For a multi-chunk streaming upload, this only adds one additional RPC per request. I believe the impact on overall performance will be small. Or maybe we could piggyback the derived key onto the metadata returned by the createKey call. This would eliminate the extra RPC entirely by providing the key upfront for all subsequent chunk uploads.
Hm, to be consistent (handle secret revocation change), S3G need to send an additional RPC per PutObject. However, I think this might have some performance implication since each PutObject will generate around 5+ RPC request (S3 secret fetch, get volume, get bucket, open key, commit key). Additionally, the S3GetSecretRequest is implemented as a Ratis transaction so the latency should be higher than normal read request (without Ratis). One idea is to cache the signing key and refresh periodically, but this has consistency and security implication since the secret key might already be revoked but we might still allow the requests, etc.
The piggyback logic might work to amortize this secret key. We can think further on this.
Our cluster uses a different strategy so I'm not that well-versed about this. We might need to get people more familiar with this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3194767183
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
Review Comment:
I don't think we need to get AWS Secret key at all, AFAIK the information sent in the S3 request is enough for S3G to calculate the chunk signature.
Also querying OM for each write will affect performance so it should not be done. Transmitting the secret key securely from OM to S3G is also another security concern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3195032539
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
Review Comment:
In symmetric cryptography like HMAC, both the client and the server independently hold the shared secret key to prevent:
1. Key interception and unauthorized writes: The user's secret key is never transmitted over the network. If it were, any illegal actor intercepting the traffic could steal the key and perform unauthorized write operations.
2. Spoofed verification and untraceable access: The server must never rely on a secret key provided within the request itself for verification. If it did, an attacker could simply sign a payload with a fake secret key and send it along, making it impossible for the server to detect the unauthorized access.
AFAIK, S3G currently does not hold any secret key state locally. If I missed something and there is already a mechanism where S3G stores this, could you please point me to it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10006:
URL: https://github.com/apache/ozone/pull/10006#issuecomment-4617890208
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3194767183
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
Review Comment:
I don't think we need to get AWS Secret key at all, AFAIK the information sent in the S3 request is enough for S3G to calculate the chunk signature (I might be wrong).
Also querying OM for each write will affect performance so it should not be done. Transmitting the secret key securely from OM to S3G is also another security concern. So is there a way to do some chunk signature validation without knowing the user secret key?
Since this will affect the existing S3 chunked uploads, we need to think about the carefully think about the side-effect. IMO if we need to get the secret key, this feature might be more difficult than expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3196161798
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
+
+From a security perspective, this new API **will not expose the raw AWS Secret Key** to the S3G. Instead, S3G will provide the request context (Date, Region, Service), and OM will compute and return the **Derived Key**. This architectural choice provides significant security benefits:
+- **Defense in Depth:** Even in the unlikely event that an S3G instance is compromised or the internal network is intercepted, the Secret Key remains safely isolated within OM.
+- **Limited Blast Radius:** The exposed Derived Key is strictly scoped to a specific date, region, and service. An attacker cannot use it to forge arbitrary requests for other days or different services.
+
+## HMAC-SHA256 Implementation
+
+We add `ChunksValidator` to handle related verification tasks, such as updating hashing, building `strToSign`, and verifying signatures. To achieve this with minimal overhead, we will extract the reusable SigV4 HMAC logic currently embedded in `AWSV4AuthValidator` into a shared utility module that both `ozone-manager` and `s3gateway` can depend on. The current `AWSV4AuthValidator` class is package-private and lives in the `ozone-manager` module, so `SignedChunksInputStream` must not depend on it directly. Instead, the refactoring will move common operations such as signing-key derivation and signature calculation into a buildable shared component, while `AWSV4AuthValidator` and `SignedChunksInputStream` each call that shared code from their respective modules. This still allows `SignedChunksInputStream` to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and preserving reuse of the existing highly-optimized `ThreadLocal Mac`-based i
mplementation.
+
+## ECDSA-SHA256 Implementation
+
+Currently, the Ozone S3 Gateway exclusively supports HMAC for AWS Signature V4. To implement ECDSA (SigV4a), we will extend the authentication flow across the Ozone Manager (OM) and S3 Gateway (S3G) through three key updates:
+
+- Public Key Provisioning (OM): Introduce a new OM API to set/get an ECDSA AccessID and Public Key. This endpoint will integrate with the existing Kerberos security model to ensure all access is verified.
+- HTTP Request Authentication (S3G): Update the verification flow for HTTP requests from S3. OM will verify the HTTP request signature when requests are coming from S3G.
+- Chunked Upload Validation (S3G): Same as HMAC-SHA256, the SignedChunksInputStream will be enhanced to support ECDSA signature verification for chunked uploads.
+
+## SignedChunksInputStream flow
+
+```mermaid
+graph TD
+ classDef state fill:#f9f9fa,stroke:#333,stroke-width:2px;
+ classDef action fill:#e1f5fe,stroke:#0288d1,stroke-width:1px;
+ classDef check fill:#fff3e0,stroke:#f57c00,stroke-width:2px;
+ classDef error fill:#ffebee,stroke:#c62828,stroke-width:2px;
+ classDef endpoint fill:#e8f5e9,stroke:#388e3c,stroke-width:2px;
+
+ Start((Start InputStream)):::endpoint --> GetDerivedKey[Get Derived Key from OM]:::action
+ GetDerivedKey --> Init[Initiate Validator with Derived Key & SignatureInfo]:::action
+ Init --> ReadHeader[Read Chunk Header until CRLF]:::action
+
+ ReadHeader --> ParseHeader{Extract Payload Size <br>& Expected Signature}:::check
+ ParseHeader -- Invalid Format --> ThrowError((Throw IOException)):::error
+
+ ParseHeader -- Valid --> CheckSize{Is Size == 0?}:::check
+
+ %% Final Chunk Flow
+ CheckSize -- "Yes (Final Chunk)" --> ValidateFinal[Validate Final Signature]:::action
+ ValidateFinal -- Valid --> CheckTailer{Is Trailer}:::check
+
+
+ CheckTailer -- "Yes (Trailer?)" --> ValidateTrailer[Validate Trailer]:::action
+ ValidateTrailer --> ReturnEOF((End of the read)):::endpoint
+
+ CheckTailer -- "No (Trailer?)" --> ReturnEOF((End of the read)):::endpoint
+
+ %% Data Chunk Flow
+ CheckSize -- "No (Data Chunk)" --> ReadPayload[Read Payload Bytes]:::action
+ ReadPayload --> UpdateHash[Incrementally update <br> SHA-256 Hash]:::action
+
+ UpdateHash --> CheckChunkEnd{Is Chunk fully read?}:::check
+ CheckChunkEnd -- "No" --> ReadPayload
+
+ CheckChunkEnd -- "Yes" --> ValidateChunk[Verify Chunk Signature via ECDSA]:::action
+
+ ValidateChunk -- "Signature Mismatch" --> ThrowError
+ ValidateChunk -- "Signature Matches" --> ConsumeCRLF[Consume trailing CRLF]:::action
+ ConsumeCRLF --> ReadHeader
+```
+# Trade-offs
+
+## Verification in S3 Gateway (Fail-Fast vs. Backend Processing)
+
+Rather than offloading the verification to the backend Ozone cluster or introducing complex asynchronous pre-fetching, we decided to maintain the current stream-based architecture and execute the verification process entirely within the S3 Gateway (S3G). The primary reasons for this architectural choice are:
+
+- Fail-Fast: This allows us to immediately reject requests with invalid signatures at the edge, preventing malformed data from consuming DataNode I/O, network bandwidth, and storage capacity.
+- Stateless Scalability: Signature verification is a CPU-intensive task. Since the S3 Gateway is stateless and horizontally scalable, offloading the verification computations to S3G prevents the backend Ozone Managers (OM) or DataNodes (DN) from becoming CPU bottlenecks. S3G instances can be scaled out independently as compute demands increase.
+
+## Incremental Hashing
+
+To maintain a low memory footprint during the continuous buffering process, the system utilizes incremental hashing (e.g., MessageDigest.update()) on the incoming byte streams to calculate the payload digest on the fly. This prevents allocating massive temporary byte arrays and avoids Garbage Collection (GC) spikes during large multi-gigabyte uploads. The computed digest is then used to construct the required StringToSign, which dictates the final signature calculation.
Review Comment:
The current `MessageDigest` is simplistic since it does not rely on the previous data chunks so might not be appropriate. You might want to implement your own `MessageDigest`
From below
<img width="956" height="688" alt="image" src="https://github.com/user-attachments/assets/23385c97-dc3e-4dff-b489-633ec3095c62" />
The next chunk digest depends on the previous signature, so that is the only state you can store.
The implementation can be left up to @chungen0126. The main point (as mentioned in the design doc) is that we should not buffer anything than necessary in S3G (the data needs to be streamed seamlessly).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3232248976
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
+
+From a security perspective, this new API **will not expose the raw AWS Secret Key** to the S3G. Instead, S3G will provide the request context (Date, Region, Service), and OM will compute and return the **Derived Key**. This architectural choice provides significant security benefits:
+- **Defense in Depth:** Even in the unlikely event that an S3G instance is compromised or the internal network is intercepted, the Secret Key remains safely isolated within OM.
+- **Limited Blast Radius:** The exposed Derived Key is strictly scoped to a specific date, region, and service. An attacker cannot use it to forge arbitrary requests for other days or different services.
+
+## HMAC-SHA256 Implementation
+
+We add `ChunksValidator` to handle related verification tasks, such as updating hashing, building `strToSign`, and verifying signatures. To achieve this with minimal overhead, we will extract the reusable SigV4 HMAC logic currently embedded in `AWSV4AuthValidator` into a shared utility module that both `ozone-manager` and `s3gateway` can depend on. The current `AWSV4AuthValidator` class is package-private and lives in the `ozone-manager` module, so `SignedChunksInputStream` must not depend on it directly. Instead, the refactoring will move common operations such as signing-key derivation and signature calculation into a buildable shared component, while `AWSV4AuthValidator` and `SignedChunksInputStream` each call that shared code from their respective modules. This still allows `SignedChunksInputStream` to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and preserving reuse of the existing highly-optimized `ThreadLocal Mac`-based i
mplementation.
+
+## ECDSA-SHA256 Implementation
+
+Currently, the Ozone S3 Gateway exclusively supports HMAC for AWS Signature V4. To implement ECDSA (SigV4a), we will extend the authentication flow across the Ozone Manager (OM) and S3 Gateway (S3G) through three key updates:
Review Comment:
I'm not sure if we need to implement ECDSA in Sigv4a. ECDSA is slow compared with sha256. AWS introduced Sigv4a for some special endpoints. We don't have this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3442470562
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
+
+From a security perspective, this new API **will not expose the raw AWS Secret Key** to the S3G. Instead, S3G will provide the request context (Date, Region, Service), and OM will compute and return the **Derived Key**. This architectural choice provides significant security benefits:
+- **Defense in Depth:** Even in the unlikely event that an S3G instance is compromised or the internal network is intercepted, the Secret Key remains safely isolated within OM.
+- **Limited Blast Radius:** The exposed Derived Key is strictly scoped to a specific date, region, and service. An attacker cannot use it to forge arbitrary requests for other days or different services.
+
+## HMAC-SHA256 Implementation
+
+We add `ChunksValidator` to handle related verification tasks, such as updating hashing, building `strToSign`, and verifying signatures. To achieve this with minimal overhead, we will extract the reusable SigV4 HMAC logic currently embedded in `AWSV4AuthValidator` into a shared utility module that both `ozone-manager` and `s3gateway` can depend on. The current `AWSV4AuthValidator` class is package-private and lives in the `ozone-manager` module, so `SignedChunksInputStream` must not depend on it directly. Instead, the refactoring will move common operations such as signing-key derivation and signature calculation into a buildable shared component, while `AWSV4AuthValidator` and `SignedChunksInputStream` each call that shared code from their respective modules. This still allows `SignedChunksInputStream` to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and preserving reuse of the existing highly-optimized `ThreadLocal Mac`-based i
mplementation.
+
+## ECDSA-SHA256 Implementation
+
+Currently, the Ozone S3 Gateway exclusively supports HMAC for AWS Signature V4. To implement ECDSA (SigV4a), we will extend the authentication flow across the Ozone Manager (OM) and S3 Gateway (S3G) through three key updates:
Review Comment:
Maybe I'll move ECDSA to future work. I agree that there is no use case for it in Ozone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3160299931
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,146 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
+
+From a security perspective, this new API **will not expose the raw AWS Secret Key** to the S3G. Instead, S3G will provide the request context (Date, Region, Service), and OM will compute and return the **Derived Key**. This architectural choice provides significant security benefits:
+- **Defense in Depth:** Even in the unlikely event that an S3G instance is compromised or the internal network is intercepted, the Secret Key remains safely isolated within OM.
+- **Limited Blast Radius:** The exposed Derived Key is strictly scoped to a specific date, region, and service. An attacker cannot use it to forge arbitrary requests for other days or different services.
+
+## HMAC-SHA256 Implementation
+
+We add `ChunksValidator` to handle related verification tasks, such as updating hashing, building `strToSign`, and verifying signatures. To achieve this with minimal overhead, we will extract the reusable SigV4 HMAC logic currently embedded in `AWSV4AuthValidator` into a shared utility module that both `ozone-manager` and `s3gateway` can depend on. The current `AWSV4AuthValidator` class is package-private and lives in the `ozone-manager` module, so `SignedChunksInputStream` must not depend on it directly. Instead, the refactoring will move common operations such as signing-key derivation and signature calculation into a buildable shared component, while `AWSV4AuthValidator` and `SignedChunksInputStream` each call that shared code from their respective modules. This still allows `SignedChunksInputStream` to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and preserving reuse of the existing highly-optimized `ThreadLocal Mac`-based i
mplementation.
+
+## ECDSA-SHA256 Implementation
+
+Currently, the Ozone S3 Gateway exclusively supports HMAC for AWS Signature V4. To implement ECDSA (SigV4a), we will extend the authentication flow across the Ozone Manager (OM) and S3 Gateway (S3G) through three key updates:
+
+- Public Key Provisioning (OM): Introduce a new OM API to set/get an ECDSA AccessID and Public Key. This endpoint will integrate with the existing Kerberos security model to ensure all access is verified.
+- HTTP Request Authentication (S3G): Update the verification flow for HTTP requests from S3. OM will verify the HTTP request signature when requests are coming from S3G.
+- Chunked Upload Validation (S3G): Same as HMAC-SHA256, the SignedChunksInputStream will be enhanced to support ECDSA signature verification for chunked uploads.
+
+## SignedChunksInputStream flow
+
+```mermaid
+graph TD
+ classDef state fill:#f9f9fa,stroke:#333,stroke-width:2px;
+ classDef action fill:#e1f5fe,stroke:#0288d1,stroke-width:1px;
+ classDef check fill:#fff3e0,stroke:#f57c00,stroke-width:2px;
+ classDef error fill:#ffebee,stroke:#c62828,stroke-width:2px;
+ classDef endpoint fill:#e8f5e9,stroke:#388e3c,stroke-width:2px;
+
+ Start((Start InputStream)):::endpoint --> GetDerivedKey[Get Derived Key from OM]:::action
+ GetDerivedKey --> Init[Initiate Validator with Derived Key & SignatureInfo]:::action
+ Init --> ReadHeader[Read Chunk Header until CRLF]:::action
+
+ ReadHeader --> ParseHeader{Extract Payload Size <br>& Expected Signature}:::check
+ ParseHeader -- Invalid Format --> ThrowError((Throw IOException)):::error
+
+ ParseHeader -- Valid --> CheckSize{Is Size == 0?}:::check
+
+ %% Final Chunk Flow
+ CheckSize -- "Yes (Final Chunk)" --> ValidateFinal[Validate Final Signature]:::action
+ ValidateFinal -- Valid --> CheckTailer{Is Trailer}:::check
+
+
+ CheckTailer -- "Yes (Trailer?)" --> ValidateTrailer[Validate Trailer]:::action
+ ValidateTrailer --> ReturnEOF((End of the read)):::endpoint
+
+ CheckTailer -- "No (Trailer?)" --> ReturnEOF((End of the read)):::endpoint
+
+ %% Data Chunk Flow
+ CheckSize -- "No (Data Chunk)" --> ReadPayload[Read Payload Bytes]:::action
+ ReadPayload --> UpdateHash[Incrementally update <br> SHA-256 Hash]:::action
+
+ UpdateHash --> CheckChunkEnd{Is Chunk fully read?}:::check
+ CheckChunkEnd -- "No" --> ReadPayload
+
+ CheckChunkEnd -- "Yes" --> ValidateChunk[Verify Chunk Signature via ECDSA]:::action
+
+ ValidateChunk -- "Signature Mismatch" --> ThrowError
+ ValidateChunk -- "Signature Matches" --> ConsumeCRLF[Consume trailing CRLF]:::action
+ ConsumeCRLF --> ReadHeader
+```
+# Trade-offs
+
+## Verification in S3 Gateway (Fail-Fast vs. Backend Processing)
+
+Rather than offloading the verification to the backend Ozone cluster or introducing complex asynchronous pre-fetching, we decided to maintain the current stream-based architecture and execute the verification process entirely within the S3 Gateway (S3G). The primary reasons for this architectural choice are:
+
+- Fail-Fast: This allows us to immediately reject requests with invalid signatures at the edge, preventing malformed data from consuming DataNode I/O, network bandwidth, and storage capacity.
+- Stateless Scalability: Signature verification is a CPU-intensive task. Since the S3 Gateway is stateless and horizontally scalable, offloading the verification computations to S3G prevents the backend Ozone Managers (OM) or DataNodes (DN) from becoming CPU bottlenecks. S3G instances can be scaled out independently as compute demands increase.
+
+## Accumulative Buffering
+
+The chunk size defined by AWS S3 Signature V4 (e.g., 64KB) is fundamentally different from—and typically much smaller than—Ozone's optimal internal transmission chunk size (e.g., 4MB).
+To bridge this impedance mismatch, we introduce an internal buffering mechanism within the SignedChunksInputStream. The stream will aggressively read, verify, and accumulate multiple small S3 chunks in its buffer until the data reaches Ozone's preferred transmission length. This significantly reduces the number of micro-writes and system calls made to the downstream OzoneOutputStream, maximizing Ratis pipeline throughput.
Review Comment:
> we introduce an internal buffering mechanism within the SignedChunksInputStream. The stream will aggressively read, verify, and accumulate multiple small S3 chunks in its buffer until the data reaches Ozone's preferred transmission length.
Could you explain how would this buffer guarantee a ack chunk write success means it's durable?
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,146 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
+
+From a security perspective, this new API **will not expose the raw AWS Secret Key** to the S3G. Instead, S3G will provide the request context (Date, Region, Service), and OM will compute and return the **Derived Key**. This architectural choice provides significant security benefits:
+- **Defense in Depth:** Even in the unlikely event that an S3G instance is compromised or the internal network is intercepted, the Secret Key remains safely isolated within OM.
+- **Limited Blast Radius:** The exposed Derived Key is strictly scoped to a specific date, region, and service. An attacker cannot use it to forge arbitrary requests for other days or different services.
+
+## HMAC-SHA256 Implementation
+
+We add `ChunksValidator` to handle related verification tasks, such as updating hashing, building `strToSign`, and verifying signatures. To achieve this with minimal overhead, we will extract the reusable SigV4 HMAC logic currently embedded in `AWSV4AuthValidator` into a shared utility module that both `ozone-manager` and `s3gateway` can depend on. The current `AWSV4AuthValidator` class is package-private and lives in the `ozone-manager` module, so `SignedChunksInputStream` must not depend on it directly. Instead, the refactoring will move common operations such as signing-key derivation and signature calculation into a buildable shared component, while `AWSV4AuthValidator` and `SignedChunksInputStream` each call that shared code from their respective modules. This still allows `SignedChunksInputStream` to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and preserving reuse of the existing highly-optimized `ThreadLocal Mac`-based i
mplementation.
+
+## ECDSA-SHA256 Implementation
+
+Currently, the Ozone S3 Gateway exclusively supports HMAC for AWS Signature V4. To implement ECDSA (SigV4a), we will extend the authentication flow across the Ozone Manager (OM) and S3 Gateway (S3G) through three key updates:
+
+- Public Key Provisioning (OM): Introduce a new OM API to set/get an ECDSA AccessID and Public Key. This endpoint will integrate with the existing Kerberos security model to ensure all access is verified.
+- HTTP Request Authentication (S3G): Update the verification flow for HTTP requests from S3. OM will verify the HTTP request signature when requests are coming from S3G.
+- Chunked Upload Validation (S3G): Same as HMAC-SHA256, the SignedChunksInputStream will be enhanced to support ECDSA signature verification for chunked uploads.
+
+## SignedChunksInputStream flow
+
+```mermaid
+graph TD
+ classDef state fill:#f9f9fa,stroke:#333,stroke-width:2px;
+ classDef action fill:#e1f5fe,stroke:#0288d1,stroke-width:1px;
+ classDef check fill:#fff3e0,stroke:#f57c00,stroke-width:2px;
+ classDef error fill:#ffebee,stroke:#c62828,stroke-width:2px;
+ classDef endpoint fill:#e8f5e9,stroke:#388e3c,stroke-width:2px;
+
+ Start((Start InputStream)):::endpoint --> GetDerivedKey[Get Derived Key from OM]:::action
+ GetDerivedKey --> Init[Initiate Validator with Derived Key & SignatureInfo]:::action
+ Init --> ReadHeader[Read Chunk Header until CRLF]:::action
+
+ ReadHeader --> ParseHeader{Extract Payload Size <br>& Expected Signature}:::check
+ ParseHeader -- Invalid Format --> ThrowError((Throw IOException)):::error
+
+ ParseHeader -- Valid --> CheckSize{Is Size == 0?}:::check
+
+ %% Final Chunk Flow
+ CheckSize -- "Yes (Final Chunk)" --> ValidateFinal[Validate Final Signature]:::action
+ ValidateFinal -- Valid --> CheckTailer{Is Trailer}:::check
+
+
+ CheckTailer -- "Yes (Trailer?)" --> ValidateTrailer[Validate Trailer]:::action
+ ValidateTrailer --> ReturnEOF((End of the read)):::endpoint
+
+ CheckTailer -- "No (Trailer?)" --> ReturnEOF((End of the read)):::endpoint
+
+ %% Data Chunk Flow
+ CheckSize -- "No (Data Chunk)" --> ReadPayload[Read Payload Bytes]:::action
+ ReadPayload --> UpdateHash[Incrementally update <br> SHA-256 Hash]:::action
+
+ UpdateHash --> CheckChunkEnd{Is Chunk fully read?}:::check
+ CheckChunkEnd -- "No" --> ReadPayload
+
+ CheckChunkEnd -- "Yes" --> ValidateChunk[Verify Chunk Signature via ECDSA]:::action
+
+ ValidateChunk -- "Signature Mismatch" --> ThrowError
+ ValidateChunk -- "Signature Matches" --> ConsumeCRLF[Consume trailing CRLF]:::action
+ ConsumeCRLF --> ReadHeader
+```
+# Trade-offs
+
+## Verification in S3 Gateway (Fail-Fast vs. Backend Processing)
+
+Rather than offloading the verification to the backend Ozone cluster or introducing complex asynchronous pre-fetching, we decided to maintain the current stream-based architecture and execute the verification process entirely within the S3 Gateway (S3G). The primary reasons for this architectural choice are:
+
+- Fail-Fast: This allows us to immediately reject requests with invalid signatures at the edge, preventing malformed data from consuming DataNode I/O, network bandwidth, and storage capacity.
+- Stateless Scalability: Signature verification is a CPU-intensive task. Since the S3 Gateway is stateless and horizontally scalable, offloading the verification computations to S3G prevents the backend Ozone Managers (OM) or DataNodes (DN) from becoming CPU bottlenecks. S3G instances can be scaled out independently as compute demands increase.
+
+## Accumulative Buffering
+
+The chunk size defined by AWS S3 Signature V4 (e.g., 64KB) is fundamentally different from—and typically much smaller than—Ozone's optimal internal transmission chunk size (e.g., 4MB).
+To bridge this impedance mismatch, we introduce an internal buffering mechanism within the SignedChunksInputStream. The stream will aggressively read, verify, and accumulate multiple small S3 chunks in its buffer until the data reaches Ozone's preferred transmission length. This significantly reduces the number of micro-writes and system calls made to the downstream OzoneOutputStream, maximizing Ratis pipeline throughput.
+
+## Incremental Hashing
+
+To maintain a low memory footprint during the continuous buffering process, the system utilizes incremental hashing (e.g., MessageDigest.update()) on the incoming byte streams to calculate the payload digest on the fly. This prevents allocating massive temporary byte arrays and avoids Garbage Collection (GC) spikes during large multi-gigabyte uploads. The computed digest is then used to construct the required StringToSign, which dictates the final signature calculation.
Review Comment:
Doesn't the signed multi chunks payload verification check per chunk indenpently? or could you explain what is the incremental hashing for?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on PR #10006:
URL: https://github.com/apache/ozone/pull/10006#issuecomment-4380095932
> I think the only thing we need to add is, make the SignedChunksInputStream support materialised the chunked content if it bytes in the inputBody reach the length set in the x-amz-decoded-content-length header, reaming hash (each chunk signature depending on the previous chunk / prior state)
Thank you for the catch. You are right. The buffers work well now. I'll update the design doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3194767183
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
Review Comment:
I don't think we need to get AWS Secret key at all, AFAIK the information sent in the S3 request is enough for S3G to calculate the chunk signature (I might be wrong).
Also querying OM for each write will affect performance so it should not be done. Transmitting the secret key securely from OM to S3G is also another security concern.
Since this will affect the existing S3 uploads, we need to think about the carefully think about the side-effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3193864309
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
Review Comment:
Thanks for the review. However, under the AWS Signature V4 protocol, the client never transmits the Secret Key over the network. They only send the Access Key and the calculated Signature.
Since S3G never actually receives the user's Secret Key in the request, it is impossible for S3G to calculate the derived key locally. It must query Ozone Manager (OM), which securely holds the master keys, to perform the derivation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3195032539
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
Review Comment:
In symmetric cryptography like HMAC, both the client and the server independently hold the shared secret key to prevent:
1. Key interception and unauthorized writes: The user's secret key is never transmitted over the network. If it were, any illegal actor intercepting the traffic could steal the key and perform unauthorized write operations.
2. Spoofed verification and untraceable access: The server must never rely on a secret key provided within the request itself for verification. If it did, an attacker could simply sign a payload with a fake secret key and send it along, making it impossible for the server to detect the unauthorized access.
AFAIK, S3G currently does not hold any secret key state locally. If I missed something and there is already a mechanism where S3G stores this, could you please point me to it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3194767183
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
Review Comment:
I don't think we need to get AWS Secret key at all, AFAIK the information sent in the S3 request is enough for S3G to calculate the chunk signature (I might be wrong).
Also querying OM for each write will affect performance so it should not be done. Transmitting the secret key securely from OM to S3G is also another security concern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3076604622
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,91 @@
+---
+title: S3 Multi Chunks Verification
+date: 2026-03-30
+jira: HDDS-12542
+status: design
Review Comment:
The YAML front-matter is missing the `summary:` field, which appears to be required/standard for design docs in this directory (e.g., `s3-conditional-requests.md` and others). Also `status: design` doesn’t match any existing status values used elsewhere (`draft`, `proposed`, `implementing`, etc.); consider using one of the existing statuses so the docs site metadata stays consistent.
```suggestion
summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
date: 2026-03-30
jira: HDDS-12542
status: proposed
```
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,91 @@
+---
+title: S3 Multi Chunks Verification
+date: 2026-03-30
+jira: HDDS-12542
+status: design
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual cryptographic verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## HMAC-SHA256 Implementation
+
+To achieve this with minimal overhead, we will reuse and refactor AWSV4AuthValidator to expose getSigningKey and calculateSignature separately. This allows the SignedChunksInputStream to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and reusing the existing highly-optimized ThreadLocal Mac instances.
+
+## ECDSA-SHA256 Implementation
+
+While this proposal strictly focuses on HMAC-SHA256 verification (Goals), supporting AWS Signature V4a (ECDSA) for multi-region access is a recognized future enhancement.
+AWS SDK relies on the AWS Common Runtime (CRT), a C-based native library, to perform the complex Key Derivation Function (NIST SP 800-108) required for SigV4a. To implement this in Ozone while adhering to Apache's strict supply chain security and avoiding the risks of pre-compiled third-party JNI binaries, we propose the following roadmap for a future Jira ticket:
Review Comment:
This section says the proposal “strictly focuses on HMAC-SHA256 verification (Goals)”, but the Goals section above includes ECDSA streaming algorithms as in-scope. Please reconcile the scope (either remove ECDSA from Goals or describe how ECDSA support will be delivered in this design).
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,91 @@
+---
+title: S3 Multi Chunks Verification
+date: 2026-03-30
+jira: HDDS-12542
+status: design
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual cryptographic verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## HMAC-SHA256 Implementation
+
+To achieve this with minimal overhead, we will reuse and refactor AWSV4AuthValidator to expose getSigningKey and calculateSignature separately. This allows the SignedChunksInputStream to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and reusing the existing highly-optimized ThreadLocal Mac instances.
+
+## ECDSA-SHA256 Implementation
+
+While this proposal strictly focuses on HMAC-SHA256 verification (Goals), supporting AWS Signature V4a (ECDSA) for multi-region access is a recognized future enhancement.
+AWS SDK relies on the AWS Common Runtime (CRT), a C-based native library, to perform the complex Key Derivation Function (NIST SP 800-108) required for SigV4a. To implement this in Ozone while adhering to Apache's strict supply chain security and avoiding the risks of pre-compiled third-party JNI binaries, we propose the following roadmap for a future Jira ticket:
+
+- Native Module Compilation: Similar to Ozone's existing rocks-native module, we will introduce a dedicated module (e.g., hdds-aws-crt-native). This module will compile the AWS CRT C/C++ source code using CMake directly during the Ozone Maven build process.
+- Dynamic Loading: We will leverage Ozone's existing NativeLibraryLoader infrastructure to safely extract and load the compiled dynamic libraries (.so, .dylib) at runtime.
+- JNI Integration: A Java wrapper will be implemented to pass the secret key to the native KDF function to derive the ECDSA public key, which will then be used by a new ECDSAChunkSignatureVerifier.
+
+# Trade-offs
+
+## Verification in S3 Gateway (Fail-Fast vs. Backend Processing)
+
+Rather than offloading the cryptographic verification to the backend Ozone cluster or introducing complex asynchronous pre-fetching, we decided to maintain the current stream-based architecture and execute the verification process entirely within the S3 Gateway (S3G). The primary reasons for this architectural choice are:
+
+- Fail-Fast: This allows us to immediately reject requests with invalid signatures at the edge, preventing malformed data from consuming DataNode I/O, network bandwidth, and storage capacity.
+- Stateless Scalability: Signature verification is a CPU-intensive task. Since the S3 Gateway is stateless and horizontally scalable, offloading the cryptographic computations to S3G prevents the backend Ozone Managers (OM) or DataNodes (DN) from becoming CPU bottlenecks. S3G instances can be scaled out independently as compute demands increase.
+
+## Accumulative Buffering
+
+The chunk size defined by AWS S3 Signature V4 (e.g., 64KB) is fundamentally different from—and typically much smaller than—Ozone's optimal internal transmission chunk size (e.g., 4MB).
+To bridge this impedance mismatch, we introduce an internal buffering mechanism within the SignedChunksInputStream. The stream will aggressively read, verify, and accumulate multiple small S3 chunks in its buffer until the data reaches Ozone's preferred transmission length. This significantly reduces the number of micro-writes and system calls made to the downstream OzoneOutputStream, maximizing Ratis pipeline throughput.
+
+## Incremental Hashing
+
+To maintain a low memory footprint during the continuous buffering process, the signature calculation utilizes Mac.update() (Incremental Hashing) directly on the incoming byte streams. This ensures that we validate the payload on the fly without allocating massive temporary byte arrays, avoiding Garbage Collection (GC) spikes during large multi-gigabyte uploads.
+
+# Performance
+
+To ensure that introducing the real-time signature verification process does not significantly degrade the overall upload throughput, the architecture is designed with the following optimizations in mind. Furthermore, we plan to conduct simple benchmarks in the future to validate these performance expectations:
+
+- Constant Memory: Incremental hashing processes byte streams on the fly. This prevents large memory allocations and avoids GC spikes during massive uploads.
+- CPU Offloading & Scalability: Cryptographic computation is isolated in the stateless, horizontally scalable S3G instances. This allows verification throughput to scale easily by adding more S3G nodes, protecting backend OM and DataNodes from CPU bottlenecks.
+
+# Compatibility
+
+A core principle of this design is to introduce robust security enhancements without breaking existing workflows or requiring modifications from end-users. The proposed architecture ensures seamless integration with current S3 clients and minimal impact on Ozone's internal backend components.
+
+- Client Compatibility: Standard S3 clients (e.g., AWS SDK) require no changes to use the signature verification.
+- Backend Compatibility: No changes to existing Ozone data layouts or core RPC protocols. Only a lightweight OM API is added for S3G to retrieve the key.
Review Comment:
“Only a lightweight OM API is added for S3G to retrieve the key” needs more detail because retrieving the user’s AWS secret access key (or equivalent material) into S3G has significant security implications. Please clarify what key material is returned (raw secret vs derived per-request signing key), how it’s protected (authorization, auditing, rotation/TTL), and consider returning only derived signing keys to minimize exposure.
```suggestion
- Backend Compatibility: No changes to existing Ozone data layouts or core RPC protocols. Only a lightweight internal OM API is added for S3G to obtain the minimum signing material required for chunk verification.
- OM MUST NOT return the user's raw AWS secret access key (or any equivalent long-lived credential) to S3G.
- The API should return only a derived SigV4 signing key scoped to the request context (for example, date/region/service) or an equivalent short-lived verifier that is sufficient to validate the streaming signatures for that request.
- Access to this API must be restricted to trusted S3G-to-OM communication only, enforced with existing internal service authentication/authorization controls.
- Each retrieval of signing material must be audit logged by OM, including the requesting S3G identity and the target access key/user, without logging the secret or derived key itself.
- Returned signing material should be short-lived, tied to the request scope, and discarded by S3G after verification completes; rotation of the underlying user secret in OM must naturally invalidate newly derived material.
```
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,91 @@
+---
+title: S3 Multi Chunks Verification
+date: 2026-03-30
+jira: HDDS-12542
+status: design
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
Review Comment:
There’s an extra space in “complete the”, which reads like a typo and can be fixed for readability.
```suggestion
Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
```
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,91 @@
+---
+title: S3 Multi Chunks Verification
+date: 2026-03-30
+jira: HDDS-12542
+status: design
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual cryptographic verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## HMAC-SHA256 Implementation
+
+To achieve this with minimal overhead, we will reuse and refactor AWSV4AuthValidator to expose getSigningKey and calculateSignature separately. This allows the SignedChunksInputStream to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and reusing the existing highly-optimized ThreadLocal Mac instances.
Review Comment:
The design proposes reusing/refactoring `AWSV4AuthValidator` for chunk verification, but that class currently lives in the `ozone-manager` module and is package-private (`final class AWSV4AuthValidator`), while `SignedChunksInputStream` is in `s3gateway`. As written, S3G can’t depend on or access this code without an explicit module move/shared utility or duplication; please capture the intended approach (e.g., move shared SigV4 utilities into a common module) to avoid an unbuildable plan.
```suggestion
To achieve this with minimal overhead, we will extract the reusable SigV4 HMAC logic currently embedded in `AWSV4AuthValidator` into a shared utility module that both `ozone-manager` and `s3gateway` can depend on. The current `AWSV4AuthValidator` class is package-private and lives in the `ozone-manager` module, so `SignedChunksInputStream` must not depend on it directly. Instead, the refactoring will move common operations such as signing-key derivation and signature calculation into a buildable shared component, while `AWSV4AuthValidator` and `SignedChunksInputStream` each call that shared code from their respective modules. This still allows `SignedChunksInputStream` to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and preserving reuse of the existing highly-optimized `ThreadLocal Mac`-based implementation.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3160422355
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,146 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
+
+From a security perspective, this new API **will not expose the raw AWS Secret Key** to the S3G. Instead, S3G will provide the request context (Date, Region, Service), and OM will compute and return the **Derived Key**. This architectural choice provides significant security benefits:
+- **Defense in Depth:** Even in the unlikely event that an S3G instance is compromised or the internal network is intercepted, the Secret Key remains safely isolated within OM.
+- **Limited Blast Radius:** The exposed Derived Key is strictly scoped to a specific date, region, and service. An attacker cannot use it to forge arbitrary requests for other days or different services.
+
+## HMAC-SHA256 Implementation
+
+We add `ChunksValidator` to handle related verification tasks, such as updating hashing, building `strToSign`, and verifying signatures. To achieve this with minimal overhead, we will extract the reusable SigV4 HMAC logic currently embedded in `AWSV4AuthValidator` into a shared utility module that both `ozone-manager` and `s3gateway` can depend on. The current `AWSV4AuthValidator` class is package-private and lives in the `ozone-manager` module, so `SignedChunksInputStream` must not depend on it directly. Instead, the refactoring will move common operations such as signing-key derivation and signature calculation into a buildable shared component, while `AWSV4AuthValidator` and `SignedChunksInputStream` each call that shared code from their respective modules. This still allows `SignedChunksInputStream` to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and preserving reuse of the existing highly-optimized `ThreadLocal Mac`-based i
mplementation.
+
+## ECDSA-SHA256 Implementation
+
+Currently, the Ozone S3 Gateway exclusively supports HMAC for AWS Signature V4. To implement ECDSA (SigV4a), we will extend the authentication flow across the Ozone Manager (OM) and S3 Gateway (S3G) through three key updates:
+
+- Public Key Provisioning (OM): Introduce a new OM API to set/get an ECDSA AccessID and Public Key. This endpoint will integrate with the existing Kerberos security model to ensure all access is verified.
+- HTTP Request Authentication (S3G): Update the verification flow for HTTP requests from S3. OM will verify the HTTP request signature when requests are coming from S3G.
+- Chunked Upload Validation (S3G): Same as HMAC-SHA256, the SignedChunksInputStream will be enhanced to support ECDSA signature verification for chunked uploads.
+
+## SignedChunksInputStream flow
+
+```mermaid
+graph TD
+ classDef state fill:#f9f9fa,stroke:#333,stroke-width:2px;
+ classDef action fill:#e1f5fe,stroke:#0288d1,stroke-width:1px;
+ classDef check fill:#fff3e0,stroke:#f57c00,stroke-width:2px;
+ classDef error fill:#ffebee,stroke:#c62828,stroke-width:2px;
+ classDef endpoint fill:#e8f5e9,stroke:#388e3c,stroke-width:2px;
+
+ Start((Start InputStream)):::endpoint --> GetDerivedKey[Get Derived Key from OM]:::action
+ GetDerivedKey --> Init[Initiate Validator with Derived Key & SignatureInfo]:::action
+ Init --> ReadHeader[Read Chunk Header until CRLF]:::action
+
+ ReadHeader --> ParseHeader{Extract Payload Size <br>& Expected Signature}:::check
+ ParseHeader -- Invalid Format --> ThrowError((Throw IOException)):::error
+
+ ParseHeader -- Valid --> CheckSize{Is Size == 0?}:::check
+
+ %% Final Chunk Flow
+ CheckSize -- "Yes (Final Chunk)" --> ValidateFinal[Validate Final Signature]:::action
+ ValidateFinal -- Valid --> CheckTailer{Is Trailer}:::check
+
+
+ CheckTailer -- "Yes (Trailer?)" --> ValidateTrailer[Validate Trailer]:::action
+ ValidateTrailer --> ReturnEOF((End of the read)):::endpoint
+
+ CheckTailer -- "No (Trailer?)" --> ReturnEOF((End of the read)):::endpoint
+
+ %% Data Chunk Flow
+ CheckSize -- "No (Data Chunk)" --> ReadPayload[Read Payload Bytes]:::action
+ ReadPayload --> UpdateHash[Incrementally update <br> SHA-256 Hash]:::action
+
+ UpdateHash --> CheckChunkEnd{Is Chunk fully read?}:::check
+ CheckChunkEnd -- "No" --> ReadPayload
+
+ CheckChunkEnd -- "Yes" --> ValidateChunk[Verify Chunk Signature via ECDSA]:::action
+
+ ValidateChunk -- "Signature Mismatch" --> ThrowError
+ ValidateChunk -- "Signature Matches" --> ConsumeCRLF[Consume trailing CRLF]:::action
+ ConsumeCRLF --> ReadHeader
+```
+# Trade-offs
+
+## Verification in S3 Gateway (Fail-Fast vs. Backend Processing)
+
+Rather than offloading the verification to the backend Ozone cluster or introducing complex asynchronous pre-fetching, we decided to maintain the current stream-based architecture and execute the verification process entirely within the S3 Gateway (S3G). The primary reasons for this architectural choice are:
+
+- Fail-Fast: This allows us to immediately reject requests with invalid signatures at the edge, preventing malformed data from consuming DataNode I/O, network bandwidth, and storage capacity.
+- Stateless Scalability: Signature verification is a CPU-intensive task. Since the S3 Gateway is stateless and horizontally scalable, offloading the verification computations to S3G prevents the backend Ozone Managers (OM) or DataNodes (DN) from becoming CPU bottlenecks. S3G instances can be scaled out independently as compute demands increase.
+
+## Accumulative Buffering
+
+The chunk size defined by AWS S3 Signature V4 (e.g., 64KB) is fundamentally different from—and typically much smaller than—Ozone's optimal internal transmission chunk size (e.g., 4MB).
+To bridge this impedance mismatch, we introduce an internal buffering mechanism within the SignedChunksInputStream. The stream will aggressively read, verify, and accumulate multiple small S3 chunks in its buffer until the data reaches Ozone's preferred transmission length. This significantly reduces the number of micro-writes and system calls made to the downstream OzoneOutputStream, maximizing Ratis pipeline throughput.
+
+## Incremental Hashing
+
+To maintain a low memory footprint during the continuous buffering process, the system utilizes incremental hashing (e.g., MessageDigest.update()) on the incoming byte streams to calculate the payload digest on the fly. This prevents allocating massive temporary byte arrays and avoids Garbage Collection (GC) spikes during large multi-gigabyte uploads. The computed digest is then used to construct the required StringToSign, which dictates the final signature calculation.
Review Comment:
Doesn't the signed multi chunks payload verification check per chunk indenpently? or could you explain what is the incremental hashing for?
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,146 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
+
+From a security perspective, this new API **will not expose the raw AWS Secret Key** to the S3G. Instead, S3G will provide the request context (Date, Region, Service), and OM will compute and return the **Derived Key**. This architectural choice provides significant security benefits:
+- **Defense in Depth:** Even in the unlikely event that an S3G instance is compromised or the internal network is intercepted, the Secret Key remains safely isolated within OM.
+- **Limited Blast Radius:** The exposed Derived Key is strictly scoped to a specific date, region, and service. An attacker cannot use it to forge arbitrary requests for other days or different services.
+
+## HMAC-SHA256 Implementation
+
+We add `ChunksValidator` to handle related verification tasks, such as updating hashing, building `strToSign`, and verifying signatures. To achieve this with minimal overhead, we will extract the reusable SigV4 HMAC logic currently embedded in `AWSV4AuthValidator` into a shared utility module that both `ozone-manager` and `s3gateway` can depend on. The current `AWSV4AuthValidator` class is package-private and lives in the `ozone-manager` module, so `SignedChunksInputStream` must not depend on it directly. Instead, the refactoring will move common operations such as signing-key derivation and signature calculation into a buildable shared component, while `AWSV4AuthValidator` and `SignedChunksInputStream` each call that shared code from their respective modules. This still allows `SignedChunksInputStream` to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and preserving reuse of the existing highly-optimized `ThreadLocal Mac`-based i
mplementation.
+
+## ECDSA-SHA256 Implementation
+
+Currently, the Ozone S3 Gateway exclusively supports HMAC for AWS Signature V4. To implement ECDSA (SigV4a), we will extend the authentication flow across the Ozone Manager (OM) and S3 Gateway (S3G) through three key updates:
+
+- Public Key Provisioning (OM): Introduce a new OM API to set/get an ECDSA AccessID and Public Key. This endpoint will integrate with the existing Kerberos security model to ensure all access is verified.
+- HTTP Request Authentication (S3G): Update the verification flow for HTTP requests from S3. OM will verify the HTTP request signature when requests are coming from S3G.
+- Chunked Upload Validation (S3G): Same as HMAC-SHA256, the SignedChunksInputStream will be enhanced to support ECDSA signature verification for chunked uploads.
+
+## SignedChunksInputStream flow
+
+```mermaid
+graph TD
+ classDef state fill:#f9f9fa,stroke:#333,stroke-width:2px;
+ classDef action fill:#e1f5fe,stroke:#0288d1,stroke-width:1px;
+ classDef check fill:#fff3e0,stroke:#f57c00,stroke-width:2px;
+ classDef error fill:#ffebee,stroke:#c62828,stroke-width:2px;
+ classDef endpoint fill:#e8f5e9,stroke:#388e3c,stroke-width:2px;
+
+ Start((Start InputStream)):::endpoint --> GetDerivedKey[Get Derived Key from OM]:::action
+ GetDerivedKey --> Init[Initiate Validator with Derived Key & SignatureInfo]:::action
+ Init --> ReadHeader[Read Chunk Header until CRLF]:::action
+
+ ReadHeader --> ParseHeader{Extract Payload Size <br>& Expected Signature}:::check
+ ParseHeader -- Invalid Format --> ThrowError((Throw IOException)):::error
+
+ ParseHeader -- Valid --> CheckSize{Is Size == 0?}:::check
+
+ %% Final Chunk Flow
+ CheckSize -- "Yes (Final Chunk)" --> ValidateFinal[Validate Final Signature]:::action
+ ValidateFinal -- Valid --> CheckTailer{Is Trailer}:::check
+
+
+ CheckTailer -- "Yes (Trailer?)" --> ValidateTrailer[Validate Trailer]:::action
+ ValidateTrailer --> ReturnEOF((End of the read)):::endpoint
+
+ CheckTailer -- "No (Trailer?)" --> ReturnEOF((End of the read)):::endpoint
+
+ %% Data Chunk Flow
+ CheckSize -- "No (Data Chunk)" --> ReadPayload[Read Payload Bytes]:::action
+ ReadPayload --> UpdateHash[Incrementally update <br> SHA-256 Hash]:::action
+
+ UpdateHash --> CheckChunkEnd{Is Chunk fully read?}:::check
+ CheckChunkEnd -- "No" --> ReadPayload
+
+ CheckChunkEnd -- "Yes" --> ValidateChunk[Verify Chunk Signature via ECDSA]:::action
+
+ ValidateChunk -- "Signature Mismatch" --> ThrowError
+ ValidateChunk -- "Signature Matches" --> ConsumeCRLF[Consume trailing CRLF]:::action
+ ConsumeCRLF --> ReadHeader
+```
+# Trade-offs
+
+## Verification in S3 Gateway (Fail-Fast vs. Backend Processing)
+
+Rather than offloading the verification to the backend Ozone cluster or introducing complex asynchronous pre-fetching, we decided to maintain the current stream-based architecture and execute the verification process entirely within the S3 Gateway (S3G). The primary reasons for this architectural choice are:
+
+- Fail-Fast: This allows us to immediately reject requests with invalid signatures at the edge, preventing malformed data from consuming DataNode I/O, network bandwidth, and storage capacity.
+- Stateless Scalability: Signature verification is a CPU-intensive task. Since the S3 Gateway is stateless and horizontally scalable, offloading the verification computations to S3G prevents the backend Ozone Managers (OM) or DataNodes (DN) from becoming CPU bottlenecks. S3G instances can be scaled out independently as compute demands increase.
+
+## Accumulative Buffering
+
+The chunk size defined by AWS S3 Signature V4 (e.g., 64KB) is fundamentally different from—and typically much smaller than—Ozone's optimal internal transmission chunk size (e.g., 4MB).
+To bridge this impedance mismatch, we introduce an internal buffering mechanism within the SignedChunksInputStream. The stream will aggressively read, verify, and accumulate multiple small S3 chunks in its buffer until the data reaches Ozone's preferred transmission length. This significantly reduces the number of micro-writes and system calls made to the downstream OzoneOutputStream, maximizing Ratis pipeline throughput.
Review Comment:
> we introduce an internal buffering mechanism within the SignedChunksInputStream. The stream will aggressively read, verify, and accumulate multiple small S3 chunks in its buffer until the data reaches Ozone's preferred transmission length.
Could you explain how would this buffer guarantee a ack chunk write success means it's durable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3181998552
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,146 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
+
+From a security perspective, this new API **will not expose the raw AWS Secret Key** to the S3G. Instead, S3G will provide the request context (Date, Region, Service), and OM will compute and return the **Derived Key**. This architectural choice provides significant security benefits:
+- **Defense in Depth:** Even in the unlikely event that an S3G instance is compromised or the internal network is intercepted, the Secret Key remains safely isolated within OM.
+- **Limited Blast Radius:** The exposed Derived Key is strictly scoped to a specific date, region, and service. An attacker cannot use it to forge arbitrary requests for other days or different services.
+
+## HMAC-SHA256 Implementation
+
+We add `ChunksValidator` to handle related verification tasks, such as updating hashing, building `strToSign`, and verifying signatures. To achieve this with minimal overhead, we will extract the reusable SigV4 HMAC logic currently embedded in `AWSV4AuthValidator` into a shared utility module that both `ozone-manager` and `s3gateway` can depend on. The current `AWSV4AuthValidator` class is package-private and lives in the `ozone-manager` module, so `SignedChunksInputStream` must not depend on it directly. Instead, the refactoring will move common operations such as signing-key derivation and signature calculation into a buildable shared component, while `AWSV4AuthValidator` and `SignedChunksInputStream` each call that shared code from their respective modules. This still allows `SignedChunksInputStream` to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and preserving reuse of the existing highly-optimized `ThreadLocal Mac`-based i
mplementation.
+
+## ECDSA-SHA256 Implementation
+
+Currently, the Ozone S3 Gateway exclusively supports HMAC for AWS Signature V4. To implement ECDSA (SigV4a), we will extend the authentication flow across the Ozone Manager (OM) and S3 Gateway (S3G) through three key updates:
+
+- Public Key Provisioning (OM): Introduce a new OM API to set/get an ECDSA AccessID and Public Key. This endpoint will integrate with the existing Kerberos security model to ensure all access is verified.
+- HTTP Request Authentication (S3G): Update the verification flow for HTTP requests from S3. OM will verify the HTTP request signature when requests are coming from S3G.
+- Chunked Upload Validation (S3G): Same as HMAC-SHA256, the SignedChunksInputStream will be enhanced to support ECDSA signature verification for chunked uploads.
+
+## SignedChunksInputStream flow
+
+```mermaid
+graph TD
+ classDef state fill:#f9f9fa,stroke:#333,stroke-width:2px;
+ classDef action fill:#e1f5fe,stroke:#0288d1,stroke-width:1px;
+ classDef check fill:#fff3e0,stroke:#f57c00,stroke-width:2px;
+ classDef error fill:#ffebee,stroke:#c62828,stroke-width:2px;
+ classDef endpoint fill:#e8f5e9,stroke:#388e3c,stroke-width:2px;
+
+ Start((Start InputStream)):::endpoint --> GetDerivedKey[Get Derived Key from OM]:::action
+ GetDerivedKey --> Init[Initiate Validator with Derived Key & SignatureInfo]:::action
+ Init --> ReadHeader[Read Chunk Header until CRLF]:::action
+
+ ReadHeader --> ParseHeader{Extract Payload Size <br>& Expected Signature}:::check
+ ParseHeader -- Invalid Format --> ThrowError((Throw IOException)):::error
+
+ ParseHeader -- Valid --> CheckSize{Is Size == 0?}:::check
+
+ %% Final Chunk Flow
+ CheckSize -- "Yes (Final Chunk)" --> ValidateFinal[Validate Final Signature]:::action
+ ValidateFinal -- Valid --> CheckTailer{Is Trailer}:::check
+
+
+ CheckTailer -- "Yes (Trailer?)" --> ValidateTrailer[Validate Trailer]:::action
+ ValidateTrailer --> ReturnEOF((End of the read)):::endpoint
+
+ CheckTailer -- "No (Trailer?)" --> ReturnEOF((End of the read)):::endpoint
+
+ %% Data Chunk Flow
+ CheckSize -- "No (Data Chunk)" --> ReadPayload[Read Payload Bytes]:::action
+ ReadPayload --> UpdateHash[Incrementally update <br> SHA-256 Hash]:::action
+
+ UpdateHash --> CheckChunkEnd{Is Chunk fully read?}:::check
+ CheckChunkEnd -- "No" --> ReadPayload
+
+ CheckChunkEnd -- "Yes" --> ValidateChunk[Verify Chunk Signature via ECDSA]:::action
+
+ ValidateChunk -- "Signature Mismatch" --> ThrowError
+ ValidateChunk -- "Signature Matches" --> ConsumeCRLF[Consume trailing CRLF]:::action
+ ConsumeCRLF --> ReadHeader
+```
+# Trade-offs
+
+## Verification in S3 Gateway (Fail-Fast vs. Backend Processing)
+
+Rather than offloading the verification to the backend Ozone cluster or introducing complex asynchronous pre-fetching, we decided to maintain the current stream-based architecture and execute the verification process entirely within the S3 Gateway (S3G). The primary reasons for this architectural choice are:
+
+- Fail-Fast: This allows us to immediately reject requests with invalid signatures at the edge, preventing malformed data from consuming DataNode I/O, network bandwidth, and storage capacity.
+- Stateless Scalability: Signature verification is a CPU-intensive task. Since the S3 Gateway is stateless and horizontally scalable, offloading the verification computations to S3G prevents the backend Ozone Managers (OM) or DataNodes (DN) from becoming CPU bottlenecks. S3G instances can be scaled out independently as compute demands increase.
+
+## Accumulative Buffering
+
+The chunk size defined by AWS S3 Signature V4 (e.g., 64KB) is fundamentally different from—and typically much smaller than—Ozone's optimal internal transmission chunk size (e.g., 4MB).
+To bridge this impedance mismatch, we introduce an internal buffering mechanism within the SignedChunksInputStream. The stream will aggressively read, verify, and accumulate multiple small S3 chunks in its buffer until the data reaches Ozone's preferred transmission length. This significantly reduces the number of micro-writes and system calls made to the downstream OzoneOutputStream, maximizing Ratis pipeline throughput.
+
+## Incremental Hashing
+
+To maintain a low memory footprint during the continuous buffering process, the system utilizes incremental hashing (e.g., MessageDigest.update()) on the incoming byte streams to calculate the payload digest on the fly. This prevents allocating massive temporary byte arrays and avoids Garbage Collection (GC) spikes during large multi-gigabyte uploads. The computed digest is then used to construct the required StringToSign, which dictates the final signature calculation.
Review Comment:
While it's true that each chunk's signature is verified, incremental hashing is used here for performance and memory efficiency.
Since we are dealing with a stream, we don't want to buffer the entire chunk in memory just to compute the hash. Instead, we feed the data into the MD5/SHA256 digest incrementally as we read it from the network. Once the chunk boundary is reached, we finalize the hash to verify that specific chunk's signature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3194767183
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
Review Comment:
I don't think we need to get AWS Secret key at all, AFAIK the information sent in the S3 request is enough for S3G to calculate the chunk signature.
Also querying OM for each write will affect performance so it should not be done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3194767183
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
Review Comment:
I don't think we need to get AWS Secret key at all, AFAIK the information sent in the S3 request is enough for S3G to calculate the chunk signature (I might be wrong).
Also querying OM for each write will affect performance so it should not be done. Transmitting the secret key securely from OM to S3G is also another security concern. So is there a way to do some chunk signature validation without knowing the user secret key?
Since this will affect the existing S3 uploads, we need to think about the carefully think about the side-effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3232269534
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
+
+From a security perspective, this new API **will not expose the raw AWS Secret Key** to the S3G. Instead, S3G will provide the request context (Date, Region, Service), and OM will compute and return the **Derived Key**. This architectural choice provides significant security benefits:
Review Comment:
This "Devrived key" looks good to me. Can it be piggy backed when S3 sends RPC to OM to create the file/object?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3186424132
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,146 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
+
+From a security perspective, this new API **will not expose the raw AWS Secret Key** to the S3G. Instead, S3G will provide the request context (Date, Region, Service), and OM will compute and return the **Derived Key**. This architectural choice provides significant security benefits:
+- **Defense in Depth:** Even in the unlikely event that an S3G instance is compromised or the internal network is intercepted, the Secret Key remains safely isolated within OM.
+- **Limited Blast Radius:** The exposed Derived Key is strictly scoped to a specific date, region, and service. An attacker cannot use it to forge arbitrary requests for other days or different services.
+
+## HMAC-SHA256 Implementation
+
+We add `ChunksValidator` to handle related verification tasks, such as updating hashing, building `strToSign`, and verifying signatures. To achieve this with minimal overhead, we will extract the reusable SigV4 HMAC logic currently embedded in `AWSV4AuthValidator` into a shared utility module that both `ozone-manager` and `s3gateway` can depend on. The current `AWSV4AuthValidator` class is package-private and lives in the `ozone-manager` module, so `SignedChunksInputStream` must not depend on it directly. Instead, the refactoring will move common operations such as signing-key derivation and signature calculation into a buildable shared component, while `AWSV4AuthValidator` and `SignedChunksInputStream` each call that shared code from their respective modules. This still allows `SignedChunksInputStream` to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and preserving reuse of the existing highly-optimized `ThreadLocal Mac`-based i
mplementation.
+
+## ECDSA-SHA256 Implementation
+
+Currently, the Ozone S3 Gateway exclusively supports HMAC for AWS Signature V4. To implement ECDSA (SigV4a), we will extend the authentication flow across the Ozone Manager (OM) and S3 Gateway (S3G) through three key updates:
+
+- Public Key Provisioning (OM): Introduce a new OM API to set/get an ECDSA AccessID and Public Key. This endpoint will integrate with the existing Kerberos security model to ensure all access is verified.
+- HTTP Request Authentication (S3G): Update the verification flow for HTTP requests from S3. OM will verify the HTTP request signature when requests are coming from S3G.
+- Chunked Upload Validation (S3G): Same as HMAC-SHA256, the SignedChunksInputStream will be enhanced to support ECDSA signature verification for chunked uploads.
+
+## SignedChunksInputStream flow
+
+```mermaid
+graph TD
+ classDef state fill:#f9f9fa,stroke:#333,stroke-width:2px;
+ classDef action fill:#e1f5fe,stroke:#0288d1,stroke-width:1px;
+ classDef check fill:#fff3e0,stroke:#f57c00,stroke-width:2px;
+ classDef error fill:#ffebee,stroke:#c62828,stroke-width:2px;
+ classDef endpoint fill:#e8f5e9,stroke:#388e3c,stroke-width:2px;
+
+ Start((Start InputStream)):::endpoint --> GetDerivedKey[Get Derived Key from OM]:::action
+ GetDerivedKey --> Init[Initiate Validator with Derived Key & SignatureInfo]:::action
+ Init --> ReadHeader[Read Chunk Header until CRLF]:::action
+
+ ReadHeader --> ParseHeader{Extract Payload Size <br>& Expected Signature}:::check
+ ParseHeader -- Invalid Format --> ThrowError((Throw IOException)):::error
+
+ ParseHeader -- Valid --> CheckSize{Is Size == 0?}:::check
+
+ %% Final Chunk Flow
+ CheckSize -- "Yes (Final Chunk)" --> ValidateFinal[Validate Final Signature]:::action
+ ValidateFinal -- Valid --> CheckTailer{Is Trailer}:::check
+
+
+ CheckTailer -- "Yes (Trailer?)" --> ValidateTrailer[Validate Trailer]:::action
+ ValidateTrailer --> ReturnEOF((End of the read)):::endpoint
+
+ CheckTailer -- "No (Trailer?)" --> ReturnEOF((End of the read)):::endpoint
+
+ %% Data Chunk Flow
+ CheckSize -- "No (Data Chunk)" --> ReadPayload[Read Payload Bytes]:::action
+ ReadPayload --> UpdateHash[Incrementally update <br> SHA-256 Hash]:::action
+
+ UpdateHash --> CheckChunkEnd{Is Chunk fully read?}:::check
+ CheckChunkEnd -- "No" --> ReadPayload
+
+ CheckChunkEnd -- "Yes" --> ValidateChunk[Verify Chunk Signature via ECDSA]:::action
+
+ ValidateChunk -- "Signature Mismatch" --> ThrowError
+ ValidateChunk -- "Signature Matches" --> ConsumeCRLF[Consume trailing CRLF]:::action
+ ConsumeCRLF --> ReadHeader
+```
+# Trade-offs
+
+## Verification in S3 Gateway (Fail-Fast vs. Backend Processing)
+
+Rather than offloading the verification to the backend Ozone cluster or introducing complex asynchronous pre-fetching, we decided to maintain the current stream-based architecture and execute the verification process entirely within the S3 Gateway (S3G). The primary reasons for this architectural choice are:
+
+- Fail-Fast: This allows us to immediately reject requests with invalid signatures at the edge, preventing malformed data from consuming DataNode I/O, network bandwidth, and storage capacity.
+- Stateless Scalability: Signature verification is a CPU-intensive task. Since the S3 Gateway is stateless and horizontally scalable, offloading the verification computations to S3G prevents the backend Ozone Managers (OM) or DataNodes (DN) from becoming CPU bottlenecks. S3G instances can be scaled out independently as compute demands increase.
+
+## Accumulative Buffering
+
+The chunk size defined by AWS S3 Signature V4 (e.g., 64KB) is fundamentally different from—and typically much smaller than—Ozone's optimal internal transmission chunk size (e.g., 4MB).
+To bridge this impedance mismatch, we introduce an internal buffering mechanism within the SignedChunksInputStream. The stream will aggressively read, verify, and accumulate multiple small S3 chunks in its buffer until the data reaches Ozone's preferred transmission length. This significantly reduces the number of micro-writes and system calls made to the downstream OzoneOutputStream, maximizing Ratis pipeline throughput.
Review Comment:
I think the v1, v2 pipeline are already support buffering, the `SignedChunksInputStream` only needs to materialised the content if it bytes in the inputBody reach the lenght set in the `x-amz-decoded-content-length` header, then apply the chained SigV4 streaming hash (each chunk signature depending on the previous chunk / prior state)
v1:
https://github.com/apache/ozone/blob/576fcce9e6994a06481b364f164c6cba33e692a0/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java#L287-L294
v2:
https://github.com/apache/ozone/blob/576fcce9e6994a06481b364f164c6cba33e692a0/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java#L276-L283
https://github.com/apache/ozone/blob/576fcce9e6994a06481b364f164c6cba33e692a0/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpointStreaming.java#L205-L220
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3181921247
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,146 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
+
+From a security perspective, this new API **will not expose the raw AWS Secret Key** to the S3G. Instead, S3G will provide the request context (Date, Region, Service), and OM will compute and return the **Derived Key**. This architectural choice provides significant security benefits:
+- **Defense in Depth:** Even in the unlikely event that an S3G instance is compromised or the internal network is intercepted, the Secret Key remains safely isolated within OM.
+- **Limited Blast Radius:** The exposed Derived Key is strictly scoped to a specific date, region, and service. An attacker cannot use it to forge arbitrary requests for other days or different services.
+
+## HMAC-SHA256 Implementation
+
+We add `ChunksValidator` to handle related verification tasks, such as updating hashing, building `strToSign`, and verifying signatures. To achieve this with minimal overhead, we will extract the reusable SigV4 HMAC logic currently embedded in `AWSV4AuthValidator` into a shared utility module that both `ozone-manager` and `s3gateway` can depend on. The current `AWSV4AuthValidator` class is package-private and lives in the `ozone-manager` module, so `SignedChunksInputStream` must not depend on it directly. Instead, the refactoring will move common operations such as signing-key derivation and signature calculation into a buildable shared component, while `AWSV4AuthValidator` and `SignedChunksInputStream` each call that shared code from their respective modules. This still allows `SignedChunksInputStream` to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and preserving reuse of the existing highly-optimized `ThreadLocal Mac`-based i
mplementation.
+
+## ECDSA-SHA256 Implementation
+
+Currently, the Ozone S3 Gateway exclusively supports HMAC for AWS Signature V4. To implement ECDSA (SigV4a), we will extend the authentication flow across the Ozone Manager (OM) and S3 Gateway (S3G) through three key updates:
+
+- Public Key Provisioning (OM): Introduce a new OM API to set/get an ECDSA AccessID and Public Key. This endpoint will integrate with the existing Kerberos security model to ensure all access is verified.
+- HTTP Request Authentication (S3G): Update the verification flow for HTTP requests from S3. OM will verify the HTTP request signature when requests are coming from S3G.
+- Chunked Upload Validation (S3G): Same as HMAC-SHA256, the SignedChunksInputStream will be enhanced to support ECDSA signature verification for chunked uploads.
+
+## SignedChunksInputStream flow
+
+```mermaid
+graph TD
+ classDef state fill:#f9f9fa,stroke:#333,stroke-width:2px;
+ classDef action fill:#e1f5fe,stroke:#0288d1,stroke-width:1px;
+ classDef check fill:#fff3e0,stroke:#f57c00,stroke-width:2px;
+ classDef error fill:#ffebee,stroke:#c62828,stroke-width:2px;
+ classDef endpoint fill:#e8f5e9,stroke:#388e3c,stroke-width:2px;
+
+ Start((Start InputStream)):::endpoint --> GetDerivedKey[Get Derived Key from OM]:::action
+ GetDerivedKey --> Init[Initiate Validator with Derived Key & SignatureInfo]:::action
+ Init --> ReadHeader[Read Chunk Header until CRLF]:::action
+
+ ReadHeader --> ParseHeader{Extract Payload Size <br>& Expected Signature}:::check
+ ParseHeader -- Invalid Format --> ThrowError((Throw IOException)):::error
+
+ ParseHeader -- Valid --> CheckSize{Is Size == 0?}:::check
+
+ %% Final Chunk Flow
+ CheckSize -- "Yes (Final Chunk)" --> ValidateFinal[Validate Final Signature]:::action
+ ValidateFinal -- Valid --> CheckTailer{Is Trailer}:::check
+
+
+ CheckTailer -- "Yes (Trailer?)" --> ValidateTrailer[Validate Trailer]:::action
+ ValidateTrailer --> ReturnEOF((End of the read)):::endpoint
+
+ CheckTailer -- "No (Trailer?)" --> ReturnEOF((End of the read)):::endpoint
+
+ %% Data Chunk Flow
+ CheckSize -- "No (Data Chunk)" --> ReadPayload[Read Payload Bytes]:::action
+ ReadPayload --> UpdateHash[Incrementally update <br> SHA-256 Hash]:::action
+
+ UpdateHash --> CheckChunkEnd{Is Chunk fully read?}:::check
+ CheckChunkEnd -- "No" --> ReadPayload
+
+ CheckChunkEnd -- "Yes" --> ValidateChunk[Verify Chunk Signature via ECDSA]:::action
+
+ ValidateChunk -- "Signature Mismatch" --> ThrowError
+ ValidateChunk -- "Signature Matches" --> ConsumeCRLF[Consume trailing CRLF]:::action
+ ConsumeCRLF --> ReadHeader
+```
+# Trade-offs
+
+## Verification in S3 Gateway (Fail-Fast vs. Backend Processing)
+
+Rather than offloading the verification to the backend Ozone cluster or introducing complex asynchronous pre-fetching, we decided to maintain the current stream-based architecture and execute the verification process entirely within the S3 Gateway (S3G). The primary reasons for this architectural choice are:
+
+- Fail-Fast: This allows us to immediately reject requests with invalid signatures at the edge, preventing malformed data from consuming DataNode I/O, network bandwidth, and storage capacity.
+- Stateless Scalability: Signature verification is a CPU-intensive task. Since the S3 Gateway is stateless and horizontally scalable, offloading the verification computations to S3G prevents the backend Ozone Managers (OM) or DataNodes (DN) from becoming CPU bottlenecks. S3G instances can be scaled out independently as compute demands increase.
+
+## Accumulative Buffering
+
+The chunk size defined by AWS S3 Signature V4 (e.g., 64KB) is fundamentally different from—and typically much smaller than—Ozone's optimal internal transmission chunk size (e.g., 4MB).
+To bridge this impedance mismatch, we introduce an internal buffering mechanism within the SignedChunksInputStream. The stream will aggressively read, verify, and accumulate multiple small S3 chunks in its buffer until the data reaches Ozone's preferred transmission length. This significantly reduces the number of micro-writes and system calls made to the downstream OzoneOutputStream, maximizing Ratis pipeline throughput.
Review Comment:
To clarify, the internal buffer in S3G does not acknowledge individual AWS SigV4 chunks. In the S3 protocol, the client does not receive an ACK for each chunk, it only receives a single HTTP 200 OK at the very end of the entire PUT request (or the specific multipart upload part).
The durability is strictly guaranteed by how we handle the downstream stream lifecycle. The S3 Gateway will only return the final HTTP 200 OK to the S3 client after OzoneOutputStream.close() is successfully called. This close() operation ensures that all buffered data is fully flushed, safely committed across DataNodes via the Ratis pipeline, and the metadata is finalized in the Ozone Manager.
If the S3G crashes or restarts while data is still in this 4MB memory buffer, no ACK is ever sent to the client. The connection will be dropped, and the S3 client (like the AWS SDK) will simply treat it as a network failure and automatically retry the upload. Therefore, there is no risk of data loss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3193864309
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
Review Comment:
Thanks for the review. However, under the AWS Signature V4 protocol, the client never transmits the raw Secret Key over the network. They only send the AWSAccessKeyId and the calculated Signature.
Since S3G never actually receives the user's Secret Key in the request, it is impossible for S3G to calculate the derived key locally. It must query Ozone Manager (OM), which securely holds the master keys, to perform the derivation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3195130997
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
Review Comment:
> Also querying OM for each write will affect performance so it should not be done. Transmitting the secret key securely from OM to S3G is also another security concern. So is there a way to do some chunk signature validation without knowing the user secret key?
Retrieving the signing key from OM is secure due to the following reasons:
1. One-way Hashing: The key S3G receives is a derived key computed through multiple iterations of HMAC-SHA256. Since HMAC is a one-way cryptographic function, even if this derived key is compromised, it is impossible to get the original secret key
2. Time-bound Scoping: The derived key is strictly bound to a specific date. Because the date is a required input for the HMAC calculation, the key is only valid for that particular day and cannot be reused for requests on any other date.
As mentioned in the design doc:
> From a security perspective, this new API **will not expose the raw AWS Secret Key** to the S3G. Instead, S3G will provide the request context (Date, Region, Service), and OM will compute and return the **Derived Key**.
Regarding the performance concern of fetching the signing key from OM.
For a multi-chunk streaming upload, this only adds one additional RPC per request. I believe the impact on overall performance will be small. Or maybe we could piggyback the derived key onto the metadata returned by the `createKey` call. This would eliminate the extra RPC entirely by providing the key upfront for all subsequent chunk uploads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3196161798
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
+
+From a security perspective, this new API **will not expose the raw AWS Secret Key** to the S3G. Instead, S3G will provide the request context (Date, Region, Service), and OM will compute and return the **Derived Key**. This architectural choice provides significant security benefits:
+- **Defense in Depth:** Even in the unlikely event that an S3G instance is compromised or the internal network is intercepted, the Secret Key remains safely isolated within OM.
+- **Limited Blast Radius:** The exposed Derived Key is strictly scoped to a specific date, region, and service. An attacker cannot use it to forge arbitrary requests for other days or different services.
+
+## HMAC-SHA256 Implementation
+
+We add `ChunksValidator` to handle related verification tasks, such as updating hashing, building `strToSign`, and verifying signatures. To achieve this with minimal overhead, we will extract the reusable SigV4 HMAC logic currently embedded in `AWSV4AuthValidator` into a shared utility module that both `ozone-manager` and `s3gateway` can depend on. The current `AWSV4AuthValidator` class is package-private and lives in the `ozone-manager` module, so `SignedChunksInputStream` must not depend on it directly. Instead, the refactoring will move common operations such as signing-key derivation and signature calculation into a buildable shared component, while `AWSV4AuthValidator` and `SignedChunksInputStream` each call that shared code from their respective modules. This still allows `SignedChunksInputStream` to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and preserving reuse of the existing highly-optimized `ThreadLocal Mac`-based i
mplementation.
+
+## ECDSA-SHA256 Implementation
+
+Currently, the Ozone S3 Gateway exclusively supports HMAC for AWS Signature V4. To implement ECDSA (SigV4a), we will extend the authentication flow across the Ozone Manager (OM) and S3 Gateway (S3G) through three key updates:
+
+- Public Key Provisioning (OM): Introduce a new OM API to set/get an ECDSA AccessID and Public Key. This endpoint will integrate with the existing Kerberos security model to ensure all access is verified.
+- HTTP Request Authentication (S3G): Update the verification flow for HTTP requests from S3. OM will verify the HTTP request signature when requests are coming from S3G.
+- Chunked Upload Validation (S3G): Same as HMAC-SHA256, the SignedChunksInputStream will be enhanced to support ECDSA signature verification for chunked uploads.
+
+## SignedChunksInputStream flow
+
+```mermaid
+graph TD
+ classDef state fill:#f9f9fa,stroke:#333,stroke-width:2px;
+ classDef action fill:#e1f5fe,stroke:#0288d1,stroke-width:1px;
+ classDef check fill:#fff3e0,stroke:#f57c00,stroke-width:2px;
+ classDef error fill:#ffebee,stroke:#c62828,stroke-width:2px;
+ classDef endpoint fill:#e8f5e9,stroke:#388e3c,stroke-width:2px;
+
+ Start((Start InputStream)):::endpoint --> GetDerivedKey[Get Derived Key from OM]:::action
+ GetDerivedKey --> Init[Initiate Validator with Derived Key & SignatureInfo]:::action
+ Init --> ReadHeader[Read Chunk Header until CRLF]:::action
+
+ ReadHeader --> ParseHeader{Extract Payload Size <br>& Expected Signature}:::check
+ ParseHeader -- Invalid Format --> ThrowError((Throw IOException)):::error
+
+ ParseHeader -- Valid --> CheckSize{Is Size == 0?}:::check
+
+ %% Final Chunk Flow
+ CheckSize -- "Yes (Final Chunk)" --> ValidateFinal[Validate Final Signature]:::action
+ ValidateFinal -- Valid --> CheckTailer{Is Trailer}:::check
+
+
+ CheckTailer -- "Yes (Trailer?)" --> ValidateTrailer[Validate Trailer]:::action
+ ValidateTrailer --> ReturnEOF((End of the read)):::endpoint
+
+ CheckTailer -- "No (Trailer?)" --> ReturnEOF((End of the read)):::endpoint
+
+ %% Data Chunk Flow
+ CheckSize -- "No (Data Chunk)" --> ReadPayload[Read Payload Bytes]:::action
+ ReadPayload --> UpdateHash[Incrementally update <br> SHA-256 Hash]:::action
+
+ UpdateHash --> CheckChunkEnd{Is Chunk fully read?}:::check
+ CheckChunkEnd -- "No" --> ReadPayload
+
+ CheckChunkEnd -- "Yes" --> ValidateChunk[Verify Chunk Signature via ECDSA]:::action
+
+ ValidateChunk -- "Signature Mismatch" --> ThrowError
+ ValidateChunk -- "Signature Matches" --> ConsumeCRLF[Consume trailing CRLF]:::action
+ ConsumeCRLF --> ReadHeader
+```
+# Trade-offs
+
+## Verification in S3 Gateway (Fail-Fast vs. Backend Processing)
+
+Rather than offloading the verification to the backend Ozone cluster or introducing complex asynchronous pre-fetching, we decided to maintain the current stream-based architecture and execute the verification process entirely within the S3 Gateway (S3G). The primary reasons for this architectural choice are:
+
+- Fail-Fast: This allows us to immediately reject requests with invalid signatures at the edge, preventing malformed data from consuming DataNode I/O, network bandwidth, and storage capacity.
+- Stateless Scalability: Signature verification is a CPU-intensive task. Since the S3 Gateway is stateless and horizontally scalable, offloading the verification computations to S3G prevents the backend Ozone Managers (OM) or DataNodes (DN) from becoming CPU bottlenecks. S3G instances can be scaled out independently as compute demands increase.
+
+## Incremental Hashing
+
+To maintain a low memory footprint during the continuous buffering process, the system utilizes incremental hashing (e.g., MessageDigest.update()) on the incoming byte streams to calculate the payload digest on the fly. This prevents allocating massive temporary byte arrays and avoids Garbage Collection (GC) spikes during large multi-gigabyte uploads. The computed digest is then used to construct the required StringToSign, which dictates the final signature calculation.
Review Comment:
The current `MessageDigest` is simplistic since it does not rely on the previous data chunks so might not be appropriate.
The implementation can be left up to @chungen0126. The main point (as mentioned in the design doc) is that we should not buffer anything than necessary in S3G (the data needs to be streamed seamlessly).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on PR #10006:
URL: https://github.com/apache/ozone/pull/10006#issuecomment-4785359401
Updated and will start to create some pr for this. @ChenSammi @ivandika3 @peterxcli @henrybear327 PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3186429863
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,146 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
+
+From a security perspective, this new API **will not expose the raw AWS Secret Key** to the S3G. Instead, S3G will provide the request context (Date, Region, Service), and OM will compute and return the **Derived Key**. This architectural choice provides significant security benefits:
+- **Defense in Depth:** Even in the unlikely event that an S3G instance is compromised or the internal network is intercepted, the Secret Key remains safely isolated within OM.
+- **Limited Blast Radius:** The exposed Derived Key is strictly scoped to a specific date, region, and service. An attacker cannot use it to forge arbitrary requests for other days or different services.
+
+## HMAC-SHA256 Implementation
+
+We add `ChunksValidator` to handle related verification tasks, such as updating hashing, building `strToSign`, and verifying signatures. To achieve this with minimal overhead, we will extract the reusable SigV4 HMAC logic currently embedded in `AWSV4AuthValidator` into a shared utility module that both `ozone-manager` and `s3gateway` can depend on. The current `AWSV4AuthValidator` class is package-private and lives in the `ozone-manager` module, so `SignedChunksInputStream` must not depend on it directly. Instead, the refactoring will move common operations such as signing-key derivation and signature calculation into a buildable shared component, while `AWSV4AuthValidator` and `SignedChunksInputStream` each call that shared code from their respective modules. This still allows `SignedChunksInputStream` to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and preserving reuse of the existing highly-optimized `ThreadLocal Mac`-based i
mplementation.
+
+## ECDSA-SHA256 Implementation
+
+Currently, the Ozone S3 Gateway exclusively supports HMAC for AWS Signature V4. To implement ECDSA (SigV4a), we will extend the authentication flow across the Ozone Manager (OM) and S3 Gateway (S3G) through three key updates:
+
+- Public Key Provisioning (OM): Introduce a new OM API to set/get an ECDSA AccessID and Public Key. This endpoint will integrate with the existing Kerberos security model to ensure all access is verified.
+- HTTP Request Authentication (S3G): Update the verification flow for HTTP requests from S3. OM will verify the HTTP request signature when requests are coming from S3G.
+- Chunked Upload Validation (S3G): Same as HMAC-SHA256, the SignedChunksInputStream will be enhanced to support ECDSA signature verification for chunked uploads.
+
+## SignedChunksInputStream flow
+
+```mermaid
+graph TD
+ classDef state fill:#f9f9fa,stroke:#333,stroke-width:2px;
+ classDef action fill:#e1f5fe,stroke:#0288d1,stroke-width:1px;
+ classDef check fill:#fff3e0,stroke:#f57c00,stroke-width:2px;
+ classDef error fill:#ffebee,stroke:#c62828,stroke-width:2px;
+ classDef endpoint fill:#e8f5e9,stroke:#388e3c,stroke-width:2px;
+
+ Start((Start InputStream)):::endpoint --> GetDerivedKey[Get Derived Key from OM]:::action
+ GetDerivedKey --> Init[Initiate Validator with Derived Key & SignatureInfo]:::action
+ Init --> ReadHeader[Read Chunk Header until CRLF]:::action
+
+ ReadHeader --> ParseHeader{Extract Payload Size <br>& Expected Signature}:::check
+ ParseHeader -- Invalid Format --> ThrowError((Throw IOException)):::error
+
+ ParseHeader -- Valid --> CheckSize{Is Size == 0?}:::check
+
+ %% Final Chunk Flow
+ CheckSize -- "Yes (Final Chunk)" --> ValidateFinal[Validate Final Signature]:::action
+ ValidateFinal -- Valid --> CheckTailer{Is Trailer}:::check
+
+
+ CheckTailer -- "Yes (Trailer?)" --> ValidateTrailer[Validate Trailer]:::action
+ ValidateTrailer --> ReturnEOF((End of the read)):::endpoint
+
+ CheckTailer -- "No (Trailer?)" --> ReturnEOF((End of the read)):::endpoint
+
+ %% Data Chunk Flow
+ CheckSize -- "No (Data Chunk)" --> ReadPayload[Read Payload Bytes]:::action
+ ReadPayload --> UpdateHash[Incrementally update <br> SHA-256 Hash]:::action
+
+ UpdateHash --> CheckChunkEnd{Is Chunk fully read?}:::check
+ CheckChunkEnd -- "No" --> ReadPayload
+
+ CheckChunkEnd -- "Yes" --> ValidateChunk[Verify Chunk Signature via ECDSA]:::action
+
+ ValidateChunk -- "Signature Mismatch" --> ThrowError
+ ValidateChunk -- "Signature Matches" --> ConsumeCRLF[Consume trailing CRLF]:::action
+ ConsumeCRLF --> ReadHeader
+```
+# Trade-offs
+
+## Verification in S3 Gateway (Fail-Fast vs. Backend Processing)
+
+Rather than offloading the verification to the backend Ozone cluster or introducing complex asynchronous pre-fetching, we decided to maintain the current stream-based architecture and execute the verification process entirely within the S3 Gateway (S3G). The primary reasons for this architectural choice are:
+
+- Fail-Fast: This allows us to immediately reject requests with invalid signatures at the edge, preventing malformed data from consuming DataNode I/O, network bandwidth, and storage capacity.
+- Stateless Scalability: Signature verification is a CPU-intensive task. Since the S3 Gateway is stateless and horizontally scalable, offloading the verification computations to S3G prevents the backend Ozone Managers (OM) or DataNodes (DN) from becoming CPU bottlenecks. S3G instances can be scaled out independently as compute demands increase.
+
+## Accumulative Buffering
+
+The chunk size defined by AWS S3 Signature V4 (e.g., 64KB) is fundamentally different from—and typically much smaller than—Ozone's optimal internal transmission chunk size (e.g., 4MB).
+To bridge this impedance mismatch, we introduce an internal buffering mechanism within the SignedChunksInputStream. The stream will aggressively read, verify, and accumulate multiple small S3 chunks in its buffer until the data reaches Ozone's preferred transmission length. This significantly reduces the number of micro-writes and system calls made to the downstream OzoneOutputStream, maximizing Ratis pipeline throughput.
Review Comment:
The v1, v2 pipeline are already support buffering.
v1:
https://github.com/apache/ozone/blob/576fcce9e6994a06481b364f164c6cba33e692a0/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java#L287-L294
v2:
https://github.com/apache/ozone/blob/576fcce9e6994a06481b364f164c6cba33e692a0/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java#L276-L283
https://github.com/apache/ozone/blob/576fcce9e6994a06481b364f164c6cba33e692a0/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpointStreaming.java#L205-L220
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3194961435
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
Review Comment:
Thanks @ivandika3 for the review.
> I don't think we need to get AWS Secret key at all, AFAIK the information sent in the S3 request is enough for S3G to calculate the chunk signature (I might be wrong).
Just to clarify, the verification process require the secret key. The official AWS documentation explains how the signing key is generated. See https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html.
At step 2.
> `signing key = HMAC-SHA256(HMAC-SHA256(HMAC-SHA256(HMAC-SHA256("AWS4" + "<YourSecretAccessKey>","20130524"),"us-east-1"),"s3"),"aws4_request")`
As shown here, the signing key is derived from the secret key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3196028251
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
Review Comment:
> Just to clarify, the verification process require the secret key. The official AWS documentation explains how the signing key is generated. See https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html.
At step 2.
Yes you're right we do need the secret key. Thanks for the clarification.
> One-way Hashing: The key S3G receives is a derived key computed through multiple iterations of HMAC-SHA256. Since HMAC is a one-way cryptographic function, even if this derived key is compromised, it is impossible to get the original secret key
Yes, I think we can use the signing key to mask the secret key (since it's hashed). The main security point is that the response returned by the OM should not include a plain text secret key.
> For a multi-chunk streaming upload, this only adds one additional RPC per request. I believe the impact on overall performance will be small. Or maybe we could piggyback the derived key onto the metadata returned by the createKey call. This would eliminate the extra RPC entirely by providing the key upfront for all subsequent chunk uploads.
Hm, to be consistent (handle secret revocation change), S3G need to send an additional RPC per PutObject. However, I think this might have some performance implication since each PutObject will generate around 5+ RPC request (S3 secret fetch, get volume, get bucket, open key, commit key). Additionally, the S3GetSecretRequest is implemented as a Ratis transaction so the latency should be higher than normal read request (without Ratis). One idea is to cache the signing key and refresh periodically, but this has consistency and security implication since the secret key might already be revoked but we might still allow the requests, etc. Unless we have a way to notify S3G about the revocation (looks like Minio implemented this way through when using etcd or it will try to broadcast the changes)
The piggyback logic might work to amortize this secret key. We can think further on this.
Our cluster uses a different strategy so I'm not that well-versed about this. We might need to get people more familiar with this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3193928413
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
+
+From a security perspective, this new API **will not expose the raw AWS Secret Key** to the S3G. Instead, S3G will provide the request context (Date, Region, Service), and OM will compute and return the **Derived Key**. This architectural choice provides significant security benefits:
+- **Defense in Depth:** Even in the unlikely event that an S3G instance is compromised or the internal network is intercepted, the Secret Key remains safely isolated within OM.
+- **Limited Blast Radius:** The exposed Derived Key is strictly scoped to a specific date, region, and service. An attacker cannot use it to forge arbitrary requests for other days or different services.
+
+## HMAC-SHA256 Implementation
+
+We add `ChunksValidator` to handle related verification tasks, such as updating hashing, building `strToSign`, and verifying signatures. To achieve this with minimal overhead, we will extract the reusable SigV4 HMAC logic currently embedded in `AWSV4AuthValidator` into a shared utility module that both `ozone-manager` and `s3gateway` can depend on. The current `AWSV4AuthValidator` class is package-private and lives in the `ozone-manager` module, so `SignedChunksInputStream` must not depend on it directly. Instead, the refactoring will move common operations such as signing-key derivation and signature calculation into a buildable shared component, while `AWSV4AuthValidator` and `SignedChunksInputStream` each call that shared code from their respective modules. This still allows `SignedChunksInputStream` to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and preserving reuse of the existing highly-optimized `ThreadLocal Mac`-based i
mplementation.
+
+## ECDSA-SHA256 Implementation
+
+Currently, the Ozone S3 Gateway exclusively supports HMAC for AWS Signature V4. To implement ECDSA (SigV4a), we will extend the authentication flow across the Ozone Manager (OM) and S3 Gateway (S3G) through three key updates:
+
+- Public Key Provisioning (OM): Introduce a new OM API to set/get an ECDSA AccessID and Public Key. This endpoint will integrate with the existing Kerberos security model to ensure all access is verified.
+- HTTP Request Authentication (S3G): Update the verification flow for HTTP requests from S3. OM will verify the HTTP request signature when requests are coming from S3G.
+- Chunked Upload Validation (S3G): Same as HMAC-SHA256, the SignedChunksInputStream will be enhanced to support ECDSA signature verification for chunked uploads.
+
+## SignedChunksInputStream flow
+
+```mermaid
+graph TD
+ classDef state fill:#f9f9fa,stroke:#333,stroke-width:2px;
+ classDef action fill:#e1f5fe,stroke:#0288d1,stroke-width:1px;
+ classDef check fill:#fff3e0,stroke:#f57c00,stroke-width:2px;
+ classDef error fill:#ffebee,stroke:#c62828,stroke-width:2px;
+ classDef endpoint fill:#e8f5e9,stroke:#388e3c,stroke-width:2px;
+
+ Start((Start InputStream)):::endpoint --> GetDerivedKey[Get Derived Key from OM]:::action
+ GetDerivedKey --> Init[Initiate Validator with Derived Key & SignatureInfo]:::action
+ Init --> ReadHeader[Read Chunk Header until CRLF]:::action
+
+ ReadHeader --> ParseHeader{Extract Payload Size <br>& Expected Signature}:::check
+ ParseHeader -- Invalid Format --> ThrowError((Throw IOException)):::error
+
+ ParseHeader -- Valid --> CheckSize{Is Size == 0?}:::check
+
+ %% Final Chunk Flow
+ CheckSize -- "Yes (Final Chunk)" --> ValidateFinal[Validate Final Signature]:::action
+ ValidateFinal -- Valid --> CheckTailer{Is Trailer}:::check
+
+
+ CheckTailer -- "Yes (Trailer?)" --> ValidateTrailer[Validate Trailer]:::action
+ ValidateTrailer --> ReturnEOF((End of the read)):::endpoint
+
+ CheckTailer -- "No (Trailer?)" --> ReturnEOF((End of the read)):::endpoint
+
+ %% Data Chunk Flow
+ CheckSize -- "No (Data Chunk)" --> ReadPayload[Read Payload Bytes]:::action
+ ReadPayload --> UpdateHash[Incrementally update <br> SHA-256 Hash]:::action
+
+ UpdateHash --> CheckChunkEnd{Is Chunk fully read?}:::check
+ CheckChunkEnd -- "No" --> ReadPayload
+
+ CheckChunkEnd -- "Yes" --> ValidateChunk[Verify Chunk Signature via ECDSA]:::action
+
+ ValidateChunk -- "Signature Mismatch" --> ThrowError
+ ValidateChunk -- "Signature Matches" --> ConsumeCRLF[Consume trailing CRLF]:::action
+ ConsumeCRLF --> ReadHeader
+```
+# Trade-offs
+
+## Verification in S3 Gateway (Fail-Fast vs. Backend Processing)
+
+Rather than offloading the verification to the backend Ozone cluster or introducing complex asynchronous pre-fetching, we decided to maintain the current stream-based architecture and execute the verification process entirely within the S3 Gateway (S3G). The primary reasons for this architectural choice are:
+
+- Fail-Fast: This allows us to immediately reject requests with invalid signatures at the edge, preventing malformed data from consuming DataNode I/O, network bandwidth, and storage capacity.
+- Stateless Scalability: Signature verification is a CPU-intensive task. Since the S3 Gateway is stateless and horizontally scalable, offloading the verification computations to S3G prevents the backend Ozone Managers (OM) or DataNodes (DN) from becoming CPU bottlenecks. S3G instances can be scaled out independently as compute demands increase.
+
+## Incremental Hashing
+
+To maintain a low memory footprint during the continuous buffering process, the system utilizes incremental hashing (e.g., MessageDigest.update()) on the incoming byte streams to calculate the payload digest on the fly. This prevents allocating massive temporary byte arrays and avoids Garbage Collection (GC) spikes during large multi-gigabyte uploads. The computed digest is then used to construct the required StringToSign, which dictates the final signature calculation.
Review Comment:
Although we use `MessageDigest` here as well, reusing the one from `MultiDigestInputStream` isn't a good fit. MultiDigestInputStream validates the entire object at once, whereas this implementation requires chunk-by-chunk validation. Trying to combine these two different mechanism would just make the code clunky and less elegant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "henrybear327 (via GitHub)" <gi...@apache.org>.
henrybear327 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3011546888
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,91 @@
+---
+title: S3 Multi Chunks Verification
+date: 2026-03-30
+jira: HDDS-12542
+status: design
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
Review Comment:
```suggestion
Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
```
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,91 @@
+---
+title: S3 Multi Chunks Verification
+date: 2026-03-30
+jira: HDDS-12542
+status: design
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual cryptographic verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## HMAC-SHA256 Implementation
+
+To achieve this with minimal overhead, we will reuse and refactor AWSV4AuthValidator to expose getSigningKey and calculateSignature separately. This allows the SignedChunksInputStream to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and reusing the existing highly-optimized ThreadLocal Mac instances.
+
+## ECDSA-SHA256 Implementation
+
+While this proposal strictly focuses on HMAC-SHA256 verification (Goals), supporting AWS Signature V4a (ECDSA) for multi-region access is a recognized future enhancement.
+AWS SDK relies on the AWS Common Runtime (CRT), a C-based native library, to perform the complex Key Derivation Function (NIST SP 800-108) required for SigV4a. To implement this in Ozone while adhering to Apache's strict supply chain security and avoiding the risks of pre-compiled third-party JNI binaries, we propose the following roadmap for a future Jira ticket:
+
+- Native Module Compilation: Similar to Ozone's existing rocks-native module, we will introduce a dedicated module (e.g., hdds-aws-crt-native). This module will compile the AWS CRT C/C++ source code using CMake directly during the Ozone Maven build process.
+- Dynamic Loading: We will leverage Ozone's existing NativeLibraryLoader infrastructure to safely extract and load the compiled dynamic libraries (.so, .dylib) at runtime.
+- JNI Integration: A Java wrapper will be implemented to pass the secret key to the native KDF function to derive the ECDSA public key, which will then be used by a new ECDSAChunkSignatureVerifier.
+
+# Trade-offs
+
+## Verification in S3 Gateway (Fail-Fast vs. Backend Processing)
+
+Rather than offloading the cryptographic verification to the backend Ozone cluster or introducing complex asynchronous pre-fetching, we decided to maintain the current stream-based architecture and execute the verification process entirely within the S3 Gateway (S3G). The primary reasons for this architectural choice are:
+
+- Fail-Fast: This allows us to immediately reject requests with invalid signatures at the edge, preventing malformed data from consuming DataNode I/O, network bandwidth, and storage capacity.
+- Stateless Scalability: Signature verification is a CPU-intensive task. Since the S3 Gateway is stateless and horizontally scalable, offloading the cryptographic computations to S3G prevents the backend Ozone Managers (OM) or DataNodes (DN) from becoming CPU bottlenecks. S3G instances can be scaled out independently as compute demands increase.
+
+## Accumulative Buffering
+
+The chunk size defined by AWS S3 Signature V4 (e.g., 64KB) is fundamentally different from—and typically much smaller than—Ozone's optimal internal transmission chunk size (e.g., 4MB).
+To bridge this impedance mismatch, we introduce an internal buffering mechanism within the SignedChunksInputStream. The stream will aggressively read, verify, and accumulate multiple small S3 chunks in its buffer until the data reaches Ozone's preferred transmission length. This significantly reduces the number of micro-writes and system calls made to the downstream OzoneOutputStream, maximizing Ratis pipeline throughput.
+
+## Incremental Hashing
+
+To maintain a low memory footprint during the continuous buffering process, the signature calculation utilizes Mac.update() (Incremental Hashing) directly on the incoming byte streams. This ensures that we validate the payload on the fly without allocating massive temporary byte arrays, avoiding Garbage Collection (GC) spikes during large multi-gigabyte uploads.
Review Comment:
I think `the signature calculation utilizes Mac.update() (Incremental Hashing) directly on the incoming byte streams` won't work.
I was reading [Task 3: Calculate Signature](https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html), and I saw that the signing is done through `StringToSign`, which needs to be generated by following the instructions in [Create a string to sign](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv-create-signed-request.html#create-string-to-sign)
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,91 @@
+---
+title: S3 Multi Chunks Verification
+date: 2026-03-30
+jira: HDDS-12542
+status: design
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual cryptographic verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## HMAC-SHA256 Implementation
+
+To achieve this with minimal overhead, we will reuse and refactor AWSV4AuthValidator to expose getSigningKey and calculateSignature separately. This allows the SignedChunksInputStream to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and reusing the existing highly-optimized ThreadLocal Mac instances.
+
+## ECDSA-SHA256 Implementation
+
+While this proposal strictly focuses on HMAC-SHA256 verification (Goals), supporting AWS Signature V4a (ECDSA) for multi-region access is a recognized future enhancement.
+AWS SDK relies on the AWS Common Runtime (CRT), a C-based native library, to perform the complex Key Derivation Function (NIST SP 800-108) required for SigV4a. To implement this in Ozone while adhering to Apache's strict supply chain security and avoiding the risks of pre-compiled third-party JNI binaries, we propose the following roadmap for a future Jira ticket:
+
+- Native Module Compilation: Similar to Ozone's existing rocks-native module, we will introduce a dedicated module (e.g., hdds-aws-crt-native). This module will compile the AWS CRT C/C++ source code using CMake directly during the Ozone Maven build process.
+- Dynamic Loading: We will leverage Ozone's existing NativeLibraryLoader infrastructure to safely extract and load the compiled dynamic libraries (.so, .dylib) at runtime.
+- JNI Integration: A Java wrapper will be implemented to pass the secret key to the native KDF function to derive the ECDSA public key, which will then be used by a new ECDSAChunkSignatureVerifier.
+
+# Trade-offs
+
+## Verification in S3 Gateway (Fail-Fast vs. Backend Processing)
+
+Rather than offloading the cryptographic verification to the backend Ozone cluster or introducing complex asynchronous pre-fetching, we decided to maintain the current stream-based architecture and execute the verification process entirely within the S3 Gateway (S3G). The primary reasons for this architectural choice are:
+
+- Fail-Fast: This allows us to immediately reject requests with invalid signatures at the edge, preventing malformed data from consuming DataNode I/O, network bandwidth, and storage capacity.
+- Stateless Scalability: Signature verification is a CPU-intensive task. Since the S3 Gateway is stateless and horizontally scalable, offloading the cryptographic computations to S3G prevents the backend Ozone Managers (OM) or DataNodes (DN) from becoming CPU bottlenecks. S3G instances can be scaled out independently as compute demands increase.
+
+## Accumulative Buffering
+
+The chunk size defined by AWS S3 Signature V4 (e.g., 64KB) is fundamentally different from—and typically much smaller than—Ozone's optimal internal transmission chunk size (e.g., 4MB).
+To bridge this impedance mismatch, we introduce an internal buffering mechanism within the SignedChunksInputStream. The stream will aggressively read, verify, and accumulate multiple small S3 chunks in its buffer until the data reaches Ozone's preferred transmission length. This significantly reduces the number of micro-writes and system calls made to the downstream OzoneOutputStream, maximizing Ratis pipeline throughput.
+
+## Incremental Hashing
+
+To maintain a low memory footprint during the continuous buffering process, the signature calculation utilizes Mac.update() (Incremental Hashing) directly on the incoming byte streams. This ensures that we validate the payload on the fly without allocating massive temporary byte arrays, avoiding Garbage Collection (GC) spikes during large multi-gigabyte uploads.
+
+# Performance
+
+To ensure that introducing the real-time signature verification process does not significantly degrade the overall upload throughput, the architecture is designed with the following optimizations in mind. Furthermore, we plan to conduct simple benchmarks in the future to validate these performance expectations:
+
+- Constant Memory: Incremental hashing processes byte streams on the fly. This prevents large memory allocations and avoids GC spikes during massive uploads.
+- CPU Offloading & Scalability: Cryptographic computation is isolated in the stateless, horizontally scalable S3G instances. This allows verification throughput to scale easily by adding more S3G nodes, protecting backend OM and DataNodes from CPU bottlenecks.
+
+# Compatibility
+
+A core principle of this design is to introduce robust security enhancements without breaking existing workflows or requiring modifications from end-users. The proposed architecture ensures seamless integration with current S3 clients and minimal impact on Ozone's internal backend components.
+
+- Client Compatibility: Standard S3 clients (e.g., AWS SDK) require no changes to use the signature verification.
+- Backend Compatibility: No changes to existing Ozone data layouts or core RPC protocols. Only a lightweight OM API is added for S3G to retrieve the key.
+
+# Test
+
+To guarantee the correctness, stability, and security of the newly introduced chunk verification logic, a comprehensive testing strategy will be executed. This plan covers both granular unit testing for the stream parsing logic and end-to-end integration testing using official AWS SDKs.
+
+- Unit Tests (SignedChunksInputStream.java): There is an existing test class for SignedChunksInputStream. To complete it, we should make sure it works well on different signatures.
+- Integration Tests:
Review Comment:
We should likely also add integration tests for `STREAMING-AWS4-HMAC-SHA256-PAYLOAD` and `STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER`.
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,91 @@
+---
+title: S3 Multi Chunks Verification
+date: 2026-03-30
+jira: HDDS-12542
+status: design
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual cryptographic verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## HMAC-SHA256 Implementation
+
+To achieve this with minimal overhead, we will reuse and refactor AWSV4AuthValidator to expose getSigningKey and calculateSignature separately. This allows the SignedChunksInputStream to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and reusing the existing highly-optimized ThreadLocal Mac instances.
+
+## ECDSA-SHA256 Implementation
+
+While this proposal strictly focuses on HMAC-SHA256 verification (Goals), supporting AWS Signature V4a (ECDSA) for multi-region access is a recognized future enhancement.
+AWS SDK relies on the AWS Common Runtime (CRT), a C-based native library, to perform the complex Key Derivation Function (NIST SP 800-108) required for SigV4a. To implement this in Ozone while adhering to Apache's strict supply chain security and avoiding the risks of pre-compiled third-party JNI binaries, we propose the following roadmap for a future Jira ticket:
+
+- Native Module Compilation: Similar to Ozone's existing rocks-native module, we will introduce a dedicated module (e.g., hdds-aws-crt-native). This module will compile the AWS CRT C/C++ source code using CMake directly during the Ozone Maven build process.
+- Dynamic Loading: We will leverage Ozone's existing NativeLibraryLoader infrastructure to safely extract and load the compiled dynamic libraries (.so, .dylib) at runtime.
+- JNI Integration: A Java wrapper will be implemented to pass the secret key to the native KDF function to derive the ECDSA public key, which will then be used by a new ECDSAChunkSignatureVerifier.
+
Review Comment:
Maybe it's also worth calling out in the design that for `-PAYLOAD-TRAILER` types, e.g., we need to:
1. include `x-amz-trailer` in the header and specify the trailing header names as a string in a comma-separated list
2. all trailing headers are written after the final chunk. In the case of uploading the data in multiple chunks, we must send a final chunk with 0 bytes of data before sending the trailing header.
Ref: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #10006:
URL: https://github.com/apache/ozone/pull/10006#issuecomment-4371600295
> In S3, an MPU UploadPart request behaves exactly like a standard single PUT request. If a client uses chunked encoding for an MPU part, it will go through this exact same `SignedChunksInputStream` logic. The stream will verify and buffer the chunks for that specific part just like it does for a single PUT.
Actually I’m asking whether this multi‑chunk payload can reuse the existing MPU infrastructure. Sorry, I didn’t phrase it clearly earlier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3194961435
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
Review Comment:
> I don't think we need to get AWS Secret key at all, AFAIK the information sent in the S3 request is enough for S3G to calculate the chunk signature (I might be wrong).
Just to clarify, the verification process require the secret key. The official AWS documentation explains how the signing key is generated. See https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html.
At step 2.
> `signing key = HMAC-SHA256(HMAC-SHA256(HMAC-SHA256(HMAC-SHA256("AWS4" + "<YourSecretAccessKey>","20130524"),"us-east-1"),"s3"),"aws4_request")`
As shown here, the signing key is derived from the secret key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3196028251
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
Review Comment:
> Just to clarify, the verification process require the secret key. The official AWS documentation explains how the signing key is generated. See https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html.
At step 2.
Yes you're right we do need the secret key. Thanks for the clarification.
> One-way Hashing: The key S3G receives is a derived key computed through multiple iterations of HMAC-SHA256. Since HMAC is a one-way cryptographic function, even if this derived key is compromised, it is impossible to get the original secret key
Yes, I think we can use the signing key to mask the secret key (since it's hashed). The main security point is that the response returned by the OM should not include a plain text secret key.
> For a multi-chunk streaming upload, this only adds one additional RPC per request. I believe the impact on overall performance will be small. Or maybe we could piggyback the derived key onto the metadata returned by the createKey call. This would eliminate the extra RPC entirely by providing the key upfront for all subsequent chunk uploads.
Hm, to be consistent (handle secret revocation change), S3G need to send an additional RPC per PutObject. However, I think this might have some performance implication since each PutObject will generate around 5+ RPC request (S3 secret fetch, get volume, get bucket, open key, commit key). Additionally, the S3GetSecretRequest is implemented as a Ratis transaction so the latency should be higher than normal read request (without Ratis). One idea is to cache the signing key and refresh periodically, but this has consistency and security implication since the secret key might already be revoked but we might still allow the requests, etc. Unless we have a way to notify S3G about the revocation (looks like Minio implemented this way through when using etcd or it will try to broadcast the changes)
The piggyback logic might work to amortize this secret key (I can see the OMKeyCreateRequest can return the user signingKey if requested). We can think further on this.
Our cluster uses a different strategy so I'm not that well-versed about this. We might need to get people more familiar with this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10006:
URL: https://github.com/apache/ozone/pull/10006#issuecomment-4686275135
Thank you for your contribution. This PR is being closed due to inactivity. Please contact a maintainer if you would like to reopen it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3196028251
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
Review Comment:
> Just to clarify, the verification process require the secret key. The official AWS documentation explains how the signing key is generated. See https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html.
At step 2.
Yes you're right we do need the secret key. Thanks for the clarification.
> One-way Hashing: The key S3G receives is a derived key computed through multiple iterations of HMAC-SHA256. Since HMAC is a one-way cryptographic function, even if this derived key is compromised, it is impossible to get the original secret key
Yes, I think we can use the signing key to mask the secret key (since it's hashed). The main security point is that the response returned by the OM should not include a plain text secret key.
> For a multi-chunk streaming upload, this only adds one additional RPC per request. I believe the impact on overall performance will be small. Or maybe we could piggyback the derived key onto the metadata returned by the createKey call. This would eliminate the extra RPC entirely by providing the key upfront for all subsequent chunk uploads.
Hm, to be consistent (handle secret revocation change), S3G need to send an additional RPC per PutObject. However, I think this might have some performance implication since each PutObject will generate around 5+ RPC request (S3 secret fetch, get volume, get bucket, open key, commit key). Additionally, the S3GetSecretRequest is implemented as a Ratis transaction so the latency should be higher than normal read request (without Ratis). One idea is to cache the signing key and refresh periodically, but this has consistency and security implication since the secret key might already be revoked but we might still allow the requests, etc. Unless we have a way to notify S3G about the revocation (looks like Minio implemented this way through when using etcd or it will try to broadcast the changes)
The piggyback logic might work to amortize this secret key (I can see the OMKeyCreateRequest can return the user signingKey if requested). We can think further on this.
Our cluster uses a different strategy so I'm not that well-versed about this. We might need to get people more familiar with this. You can also research other systems like Minio and SeaweedFS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3192961427
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
Review Comment:
Users will bring their s3 secret when they are sending s3 request, so can we just use it to calculate the derived key?
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,142 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
+
+From a security perspective, this new API **will not expose the raw AWS Secret Key** to the S3G. Instead, S3G will provide the request context (Date, Region, Service), and OM will compute and return the **Derived Key**. This architectural choice provides significant security benefits:
+- **Defense in Depth:** Even in the unlikely event that an S3G instance is compromised or the internal network is intercepted, the Secret Key remains safely isolated within OM.
+- **Limited Blast Radius:** The exposed Derived Key is strictly scoped to a specific date, region, and service. An attacker cannot use it to forge arbitrary requests for other days or different services.
+
+## HMAC-SHA256 Implementation
+
+We add `ChunksValidator` to handle related verification tasks, such as updating hashing, building `strToSign`, and verifying signatures. To achieve this with minimal overhead, we will extract the reusable SigV4 HMAC logic currently embedded in `AWSV4AuthValidator` into a shared utility module that both `ozone-manager` and `s3gateway` can depend on. The current `AWSV4AuthValidator` class is package-private and lives in the `ozone-manager` module, so `SignedChunksInputStream` must not depend on it directly. Instead, the refactoring will move common operations such as signing-key derivation and signature calculation into a buildable shared component, while `AWSV4AuthValidator` and `SignedChunksInputStream` each call that shared code from their respective modules. This still allows `SignedChunksInputStream` to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and preserving reuse of the existing highly-optimized `ThreadLocal Mac`-based i
mplementation.
+
+## ECDSA-SHA256 Implementation
+
+Currently, the Ozone S3 Gateway exclusively supports HMAC for AWS Signature V4. To implement ECDSA (SigV4a), we will extend the authentication flow across the Ozone Manager (OM) and S3 Gateway (S3G) through three key updates:
+
+- Public Key Provisioning (OM): Introduce a new OM API to set/get an ECDSA AccessID and Public Key. This endpoint will integrate with the existing Kerberos security model to ensure all access is verified.
+- HTTP Request Authentication (S3G): Update the verification flow for HTTP requests from S3. OM will verify the HTTP request signature when requests are coming from S3G.
+- Chunked Upload Validation (S3G): Same as HMAC-SHA256, the SignedChunksInputStream will be enhanced to support ECDSA signature verification for chunked uploads.
+
+## SignedChunksInputStream flow
+
+```mermaid
+graph TD
+ classDef state fill:#f9f9fa,stroke:#333,stroke-width:2px;
+ classDef action fill:#e1f5fe,stroke:#0288d1,stroke-width:1px;
+ classDef check fill:#fff3e0,stroke:#f57c00,stroke-width:2px;
+ classDef error fill:#ffebee,stroke:#c62828,stroke-width:2px;
+ classDef endpoint fill:#e8f5e9,stroke:#388e3c,stroke-width:2px;
+
+ Start((Start InputStream)):::endpoint --> GetDerivedKey[Get Derived Key from OM]:::action
+ GetDerivedKey --> Init[Initiate Validator with Derived Key & SignatureInfo]:::action
+ Init --> ReadHeader[Read Chunk Header until CRLF]:::action
+
+ ReadHeader --> ParseHeader{Extract Payload Size <br>& Expected Signature}:::check
+ ParseHeader -- Invalid Format --> ThrowError((Throw IOException)):::error
+
+ ParseHeader -- Valid --> CheckSize{Is Size == 0?}:::check
+
+ %% Final Chunk Flow
+ CheckSize -- "Yes (Final Chunk)" --> ValidateFinal[Validate Final Signature]:::action
+ ValidateFinal -- Valid --> CheckTailer{Is Trailer}:::check
+
+
+ CheckTailer -- "Yes (Trailer?)" --> ValidateTrailer[Validate Trailer]:::action
+ ValidateTrailer --> ReturnEOF((End of the read)):::endpoint
+
+ CheckTailer -- "No (Trailer?)" --> ReturnEOF((End of the read)):::endpoint
+
+ %% Data Chunk Flow
+ CheckSize -- "No (Data Chunk)" --> ReadPayload[Read Payload Bytes]:::action
+ ReadPayload --> UpdateHash[Incrementally update <br> SHA-256 Hash]:::action
+
+ UpdateHash --> CheckChunkEnd{Is Chunk fully read?}:::check
+ CheckChunkEnd -- "No" --> ReadPayload
+
+ CheckChunkEnd -- "Yes" --> ValidateChunk[Verify Chunk Signature via ECDSA]:::action
+
+ ValidateChunk -- "Signature Mismatch" --> ThrowError
+ ValidateChunk -- "Signature Matches" --> ConsumeCRLF[Consume trailing CRLF]:::action
+ ConsumeCRLF --> ReadHeader
+```
+# Trade-offs
+
+## Verification in S3 Gateway (Fail-Fast vs. Backend Processing)
+
+Rather than offloading the verification to the backend Ozone cluster or introducing complex asynchronous pre-fetching, we decided to maintain the current stream-based architecture and execute the verification process entirely within the S3 Gateway (S3G). The primary reasons for this architectural choice are:
+
+- Fail-Fast: This allows us to immediately reject requests with invalid signatures at the edge, preventing malformed data from consuming DataNode I/O, network bandwidth, and storage capacity.
+- Stateless Scalability: Signature verification is a CPU-intensive task. Since the S3 Gateway is stateless and horizontally scalable, offloading the verification computations to S3G prevents the backend Ozone Managers (OM) or DataNodes (DN) from becoming CPU bottlenecks. S3G instances can be scaled out independently as compute demands increase.
+
+## Incremental Hashing
+
+To maintain a low memory footprint during the continuous buffering process, the system utilizes incremental hashing (e.g., MessageDigest.update()) on the incoming byte streams to calculate the payload digest on the fly. This prevents allocating massive temporary byte arrays and avoids Garbage Collection (GC) spikes during large multi-gigabyte uploads. The computed digest is then used to construct the required StringToSign, which dictates the final signature calculation.
Review Comment:
Could you please take a look if the multi digest already had this incremental hashing capability?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #10006:
URL: https://github.com/apache/ozone/pull/10006#discussion_r3186424132
##########
hadoop-hdds/docs/content/design/s3-multi-chunks-verification.md:
##########
@@ -0,0 +1,146 @@
+---
+title: S3 Multi Chunks Verification
+summary: Add signature verification support for AWS Signature V4 streaming chunked uploads in S3G.
+date: 2026-04-29
+jira: HDDS-12542
+status: proposed
+author: Chung-En Lee
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Context & Motivation
+
+Ozone S3 Gateway (S3G) currently utilizes SignedChunksInputStream to handle aws-chunked content-encoding for AWS Signature V4. However, it doesn’t do any signature verification now. This proposal aims to complete the existing SignedChunksInputStream to make sure signature verification is correct and minimize performance overhead.
+
+# Goal
+
+Support signature verification for AWS Signature Version 4 streaming chunked uploads with the following algorithms:
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD
+- STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD
+- STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
+
+# Proposed Solution
+
+Currently, the SignedChunksInputStream successfully parses the S3 chunked upload payload but lacks the actual signature verification. This proposal enhances the existing stream to perform real-time signature verification, while ensuring the output remains fully compatible with Ozone's native, high-throughput write APIs.
+
+## Secret Key
+
+Currently, the AWS Secret Keys are securely stored and managed exclusively within the Ozone Manager (OM). To enable the S3 Gateway (S3G) to independently verify chunked payloads, it requires access to verification materials. We propose adding a new internal OM API specifically for S3G to retrieve this data.
+
+From a security perspective, this new API **will not expose the raw AWS Secret Key** to the S3G. Instead, S3G will provide the request context (Date, Region, Service), and OM will compute and return the **Derived Key**. This architectural choice provides significant security benefits:
+- **Defense in Depth:** Even in the unlikely event that an S3G instance is compromised or the internal network is intercepted, the Secret Key remains safely isolated within OM.
+- **Limited Blast Radius:** The exposed Derived Key is strictly scoped to a specific date, region, and service. An attacker cannot use it to forge arbitrary requests for other days or different services.
+
+## HMAC-SHA256 Implementation
+
+We add `ChunksValidator` to handle related verification tasks, such as updating hashing, building `strToSign`, and verifying signatures. To achieve this with minimal overhead, we will extract the reusable SigV4 HMAC logic currently embedded in `AWSV4AuthValidator` into a shared utility module that both `ozone-manager` and `s3gateway` can depend on. The current `AWSV4AuthValidator` class is package-private and lives in the `ozone-manager` module, so `SignedChunksInputStream` must not depend on it directly. Instead, the refactoring will move common operations such as signing-key derivation and signature calculation into a buildable shared component, while `AWSV4AuthValidator` and `SignedChunksInputStream` each call that shared code from their respective modules. This still allows `SignedChunksInputStream` to compute the derived key strictly once per request, avoiding expensive HMAC recalculations per chunk and preserving reuse of the existing highly-optimized `ThreadLocal Mac`-based i
mplementation.
+
+## ECDSA-SHA256 Implementation
+
+Currently, the Ozone S3 Gateway exclusively supports HMAC for AWS Signature V4. To implement ECDSA (SigV4a), we will extend the authentication flow across the Ozone Manager (OM) and S3 Gateway (S3G) through three key updates:
+
+- Public Key Provisioning (OM): Introduce a new OM API to set/get an ECDSA AccessID and Public Key. This endpoint will integrate with the existing Kerberos security model to ensure all access is verified.
+- HTTP Request Authentication (S3G): Update the verification flow for HTTP requests from S3. OM will verify the HTTP request signature when requests are coming from S3G.
+- Chunked Upload Validation (S3G): Same as HMAC-SHA256, the SignedChunksInputStream will be enhanced to support ECDSA signature verification for chunked uploads.
+
+## SignedChunksInputStream flow
+
+```mermaid
+graph TD
+ classDef state fill:#f9f9fa,stroke:#333,stroke-width:2px;
+ classDef action fill:#e1f5fe,stroke:#0288d1,stroke-width:1px;
+ classDef check fill:#fff3e0,stroke:#f57c00,stroke-width:2px;
+ classDef error fill:#ffebee,stroke:#c62828,stroke-width:2px;
+ classDef endpoint fill:#e8f5e9,stroke:#388e3c,stroke-width:2px;
+
+ Start((Start InputStream)):::endpoint --> GetDerivedKey[Get Derived Key from OM]:::action
+ GetDerivedKey --> Init[Initiate Validator with Derived Key & SignatureInfo]:::action
+ Init --> ReadHeader[Read Chunk Header until CRLF]:::action
+
+ ReadHeader --> ParseHeader{Extract Payload Size <br>& Expected Signature}:::check
+ ParseHeader -- Invalid Format --> ThrowError((Throw IOException)):::error
+
+ ParseHeader -- Valid --> CheckSize{Is Size == 0?}:::check
+
+ %% Final Chunk Flow
+ CheckSize -- "Yes (Final Chunk)" --> ValidateFinal[Validate Final Signature]:::action
+ ValidateFinal -- Valid --> CheckTailer{Is Trailer}:::check
+
+
+ CheckTailer -- "Yes (Trailer?)" --> ValidateTrailer[Validate Trailer]:::action
+ ValidateTrailer --> ReturnEOF((End of the read)):::endpoint
+
+ CheckTailer -- "No (Trailer?)" --> ReturnEOF((End of the read)):::endpoint
+
+ %% Data Chunk Flow
+ CheckSize -- "No (Data Chunk)" --> ReadPayload[Read Payload Bytes]:::action
+ ReadPayload --> UpdateHash[Incrementally update <br> SHA-256 Hash]:::action
+
+ UpdateHash --> CheckChunkEnd{Is Chunk fully read?}:::check
+ CheckChunkEnd -- "No" --> ReadPayload
+
+ CheckChunkEnd -- "Yes" --> ValidateChunk[Verify Chunk Signature via ECDSA]:::action
+
+ ValidateChunk -- "Signature Mismatch" --> ThrowError
+ ValidateChunk -- "Signature Matches" --> ConsumeCRLF[Consume trailing CRLF]:::action
+ ConsumeCRLF --> ReadHeader
+```
+# Trade-offs
+
+## Verification in S3 Gateway (Fail-Fast vs. Backend Processing)
+
+Rather than offloading the verification to the backend Ozone cluster or introducing complex asynchronous pre-fetching, we decided to maintain the current stream-based architecture and execute the verification process entirely within the S3 Gateway (S3G). The primary reasons for this architectural choice are:
+
+- Fail-Fast: This allows us to immediately reject requests with invalid signatures at the edge, preventing malformed data from consuming DataNode I/O, network bandwidth, and storage capacity.
+- Stateless Scalability: Signature verification is a CPU-intensive task. Since the S3 Gateway is stateless and horizontally scalable, offloading the verification computations to S3G prevents the backend Ozone Managers (OM) or DataNodes (DN) from becoming CPU bottlenecks. S3G instances can be scaled out independently as compute demands increase.
+
+## Accumulative Buffering
+
+The chunk size defined by AWS S3 Signature V4 (e.g., 64KB) is fundamentally different from—and typically much smaller than—Ozone's optimal internal transmission chunk size (e.g., 4MB).
+To bridge this impedance mismatch, we introduce an internal buffering mechanism within the SignedChunksInputStream. The stream will aggressively read, verify, and accumulate multiple small S3 chunks in its buffer until the data reaches Ozone's preferred transmission length. This significantly reduces the number of micro-writes and system calls made to the downstream OzoneOutputStream, maximizing Ratis pipeline throughput.
Review Comment:
I think the v1, v2 pipeline are already support buffering, the `SignedChunksInputStream` only needs to materialised the content if it bytes in the inputBody reach the lenght set in the `x-amz-decoded-content-length` header, then apply the chained SigV4 streaming hash (each chunk signature depending on the previous chunk / prior state)
v1:
https://github.com/apache/ozone/blob/576fcce9e6994a06481b364f164c6cba33e692a0/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java#L287-L294
v2:
https://github.com/apache/ozone/blob/576fcce9e6994a06481b364f164c6cba33e692a0/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java#L276-L283
https://github.com/apache/ozone/blob/576fcce9e6994a06481b364f164c6cba33e692a0/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpointStreaming.java#L205-L220
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on PR #10006:
URL: https://github.com/apache/ozone/pull/10006#issuecomment-4371539891
> I'm thinking if we could reuse the multipart upload? data chunk goes to mpu request, while the final chunk trigger a mpu complete request.
Yes, MPU can reuse this.
In S3, an MPU UploadPart request behaves exactly like a standard single PUT request. If a client uses chunked encoding for an MPU part, it will go through this exact same `SignedChunksInputStream` logic. The stream will verify and buffer the chunks for that specific part just like it does for a single PUT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #10006: HDDS-12542. Support S3 signed multi chunks payload verification
URL: https://github.com/apache/ozone/pull/10006
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12542. Support S3 signed multi chunks payload verification [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10006:
URL: https://github.com/apache/ozone/pull/10006#issuecomment-5028730111
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org