You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "leaves12138 (via GitHub)" <gi...@apache.org> on 2026/09/15 07:03:45 UTC
[PR] [core] Expose Avro header and physical block locations [paimon]
leaves12138 opened a new pull request, #9835:
URL: https://github.com/apache/paimon/pull/9835
### Purpose
Expose the original Avro object-container header and physical block offsets/lengths through `RawBlockReader`, `AvroBlockReader`, and `ManifestAvroReader`. These locations allow callers to index existing manifest blocks and reconstruct valid Avro streams from selected blocks. Offsets account for decoder read-ahead, and encoded lengths include the block header and sync marker.
This is the first standalone piece extracted from the manifest sidecar work in #9743. It only adds block metadata APIs and tests; sidecar generation, scan pruning, and caching will follow separately.
### Tests
- Added boundary checks against `DataFileWriter.sync()` for null, deflate, Snappy, and Zstandard codecs, including large headers, one-byte/short reads, repeated look-ahead, borrowed-block reuse, and empty files.
- Reconstructed each block with the original header and verified its records using the standard Avro reader.
- Extended manifest integration coverage to verify header preservation and contiguous physical block coverage.
- Passed 109 tests on Java 8, with the normal Maven validation checks enabled:
```bash
mvn -B -ntp -nsu -pl paimon-core -am \
-DwildcardSuites=none -DfailIfNoTests=false \
-Dtest=AvroBlockReaderTest,AvroFileFormatTest,ManifestFileTest clean test
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Expose Avro header and physical block locations [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #9835:
URL: https://github.com/apache/paimon/pull/9835#issuecomment-5676973549
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Expose Avro header and physical block locations [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9835:
URL: https://github.com/apache/paimon/pull/9835
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org