You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Chu Cheng Li (Jira)" <ji...@apache.org> on 2026/10/06 04:40:00 UTC

[jira] [Created] (HDDS-16727) Datanode parses a block's whole metadata for every streaming ReadBlock request

Chu Cheng Li created HDDS-16727:
-----------------------------------

             Summary: Datanode parses a block's whole metadata for every streaming ReadBlock request
                 Key: HDDS-16727
                 URL: https://issues.apache.org/jira/browse/HDDS-16727
             Project: Apache Ozone
          Issue Type: Sub-task
          Components: Ozone Datanode
            Reporter: Chu Cheng Li
            Assignee: Chu Cheng Li


*Background*
A streaming ReadBlock request reads a range of a block. The gRPC streaming read ({{ozone.client.stream.readblock.enable}}) uses it. The Ratis data stream read of HDDS-9904 also uses it.

For each ReadBlock request, the datanode reads the {{BlockData}} of the block from RocksDB. The {{BlockData}} contains the chunk list of the block and the checksums of each chunk.

*Problem*
The datanode parses all of the {{BlockData}} for each request, also when the request reads only a few KB.
* With the default sizes (256 MB block, 4 MB chunk, 16 KB for each checksum), a full block has 64 chunks and 16,384 checksums.
* The serialized {{BlockData}} is about 100 KB. The parse allocates about 1 MB, because each checksum becomes a separate {{ByteString}}.

*Effect*
In a TPC-H SF100 Parquet test from Spark on one datanode, the parse used 47% to 64% of the CPU of the ReadBlock handler.
* The parse took 0.46 ms for each gRPC request and 0.67 ms for each Ratis request. The RocksDB get took about 0.04 ms.
* A test cache of the parsed {{BlockData}} of closed containers decreased the median time of small reads from 1.4–2.2 ms to 0.4–0.8 ms. Warm queries became 7–8% faster on both read paths.
* A cache is not safe. Reconciliation can change the blocks of a closed container and keep the same BCSID. Block deletion and replica replacement also change these blocks.

*Changes*
* For ReadBlock, the datanode reads the serialized {{BlockData}} and walks its chunk list.
* It parses only the chunks in the requested range. For the other chunks, it reads only the offset and the length.
* Nothing is cached. Thus, no data can become stale, and reads from open containers also become faster.
* A block with an incremental chunk list (hsync), or with chunks of different lengths, keeps the full parse. Thus, the checksum lookup does not change for these blocks.

*Results*
A microbenchmark uses one thread and a schema V3 container with one full 256 MB block. The old code and the new code ran in turns in one JVM on a busy laptop. The values are medians of 15 batches.
* Block metadata for each request: 1,072 µs to 91 µs. Allocation: 1,023 KB to 131 KB.
* ReadBlock of 16 KiB: 1,493 µs to 688 µs.
* ReadBlock of 128 KiB: 1,839 µs to 624 µs.
* ReadBlock of 1 MiB: 2,159 µs to 1,308 µs.

*Tests*
* Unit tests examine the chunk walk and the cases that keep the full parse.
* A {{KeyValueHandler}} test examines the checksums of a ReadBlock in the requested range.

Found in the TPC-H benchmark of HDDS-9904 (https://github.com/apache/ozone/pull/10743).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org