You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2026/10/14 05:39:09 UTC
[I] Native HFile reader and writer do not reject block sizes the format cannot represent [hudi]
yihua opened a new issue, #20320:
URL: https://github.com/apache/hudi/issues/20320
The HFile format stores block sizes and key/value lengths as 4-byte integers, so a single block is bounded to less than 2 GiB even though the file itself, addressed by 8-byte offsets, is not (see #20308 for the file-size side). The native reader and writer in `hudi-io` do not state that bound anywhere:
- On read, `HFileBlock` takes the header sizes at face value. A corrupt or truncated header with a negative size, a zero `bytesPerChecksum`, or an on-disk size running past the buffer surfaces as `NegativeArraySizeException`, `ArithmeticException` or `ArrayIndexOutOfBoundsException` from deep inside the block code, with no indication that the block header is the problem.
- On write, `HFileWriterImpl.append` accumulates the record size in an `int` and only flushes when a block passes the configured block size. A single record of about 2 GiB, or a run of records sharing one key (which the writer keeps in a single block), would overflow that accounting and produce a block the header cannot describe, instead of failing with a clear error.
Neither case occurs with metadata table payloads today, so this is hardening rather than a live bug: reject these inputs up front with an error that names the block and the sizes involved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Native HFile reader and writer do not reject block sizes the format cannot represent [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua closed issue #20320: Native HFile reader and writer do not reject block sizes the format cannot represent
URL: https://github.com/apache/hudi/issues/20320
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Native HFile reader and writer do not reject block sizes the format cannot represent [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on issue #20320:
URL: https://github.com/apache/hudi/issues/20320#issuecomment-6161869061
Fix: #20321
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org