You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@ozone.apache.org by "ss77892 (via GitHub)" <gi...@apache.org> on 2026/10/19 23:00:10 UTC

[PR] HDDS-16559. Datanode ReadBlock fails with RESOURCE_EXHAUSTED when the client response size exceeds the gRPC message limit [ozone]

ss77892 opened a new pull request, #11637:
URL: https://github.com/apache/ozone/pull/11637

## What changes were proposed in this pull request?
HDDS-16559. Datanode ReadBlock fails with RESOURCE_EXHAUSTED when the client response size exceeds the gRPC message limit
For streaming reads (ReadBlock), the client sets the size of each response in ReadBlockRequestProto.responseDataSize, using ozone.client.stream.read.response-data-size. The datanode used this value as is. It only replaced 0 with the 1 MiB default. Two problems follow:
1. Responses too large for gRPC. The client's gRPC inbound message limit is OzoneConsts.OZONE_SCM_CHUNK_MAX_SIZE (32 MiB). If the configured size is near or above that limit, each response (data, checksum list and envelope) is too big, and the read fails with RESOURCE_EXHAUSTED. The client has no way to recover from this.
2. No bound on the field. responseDataSize is a uint32, and the datanode uses it to size its read buffer. A request can ask for any buffer size up to Integer.MAX_VALUE. A value above Integer.MAX_VALUE arrives in Java as a negative int.

This PR makes the datanode limit the response size in KeyValueHandler.boundResponseDataSize:
- A value of zero or less falls back to the 1 MiB default.
- A positive value is capped at MAX_READ_BLOCK_RESPONSE_DATA_SIZE, which is OZONE_SCM_CHUNK_MAX_SIZE - 1 MiB. The checksum list adds at most 32 bytes per 8 KiB of data, so 1 MiB leaves enough room for it and the envelope.

If the client asks for too much, the datanode now sends smaller responses and the read succeeds. There are no protocol or client changes. The client already accepts responses shorter than the requested size (covered by
testReadBlockResponseShorterThanResponseDataSize).

## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16559
## How was this patch tested?
- New unit test TestKeyValueHandler#testReadBlockResponseDataSizeIsBounded. It checks how boundResponseDataSize handles 0, negative values, a normal value, the maximum and Integer.MAX_VALUE. It also runs ReadBlock end to end with response sizes 0, -1 and Integer.MAX_VALUE and checks the data that comes back.
- New integration test TestStreamRead#testResponseDataSizeAboveGrpcMessageLimit. It sets the client's response size to 2 * OZONE_SCM_CHUNK_MAX_SIZE, stream-reads a 40 MB key, and checks its MD5 against the block file on the datanode.
- Moved the block-file lookup in TestStreamRead into a getBlockFile helper so both tests can use it.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

Re: [PR] HDDS-16559. Datanode ReadBlock fails with RESOURCE_EXHAUSTED when the client response size exceeds the gRPC message limit [ozone]

Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.

chungen0126 merged PR #11637:
URL: https://github.com/apache/ozone/pull/11637


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

Re: [PR] HDDS-16559. Datanode ReadBlock fails with RESOURCE_EXHAUSTED when the client response size exceeds the gRPC message limit [ozone]

Posted by "ss77892 (via GitHub)" <gi...@apache.org>.

ss77892 commented on PR #11637:
URL: https://github.com/apache/ozone/pull/11637#issuecomment-6429988933

   > Could you please add some description for this in OzoneClientConfig?
   @chungen0126 Thank you for looking into this. I've added more detailed description.
    
   > Also, just out of curiosity: is there any chance we could still exceed the inbound limit in extreme edge cases—for example, with EC and an intentionally small bytesPerChecksum? We definitely don't have to handle for such extreme cases, but I just wanted to confirm if that's a possibility.
   Well, in theory yes. Like checksum type SHA256, bytes per checksum 8KiB. In this case the overhead would be close to 2MiB.  
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

Re: [PR] HDDS-16559. Datanode ReadBlock fails with RESOURCE_EXHAUSTED when the client response size exceeds the gRPC message limit [ozone]

Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.

chungen0126 commented on PR #11637:
URL: https://github.com/apache/ozone/pull/11637#issuecomment-6439128180

   Merged. Thanks @ss77892 for the patch, @yandrey321 for the review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org