You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "linliu-code (via GitHub)" <gi...@apache.org> on 2026/08/29 04:22:30 UTC
[PR] feat(core): read Hudi metadata tables through the version two file group reader [hudi-rs]
linliu-code opened a new pull request, #702:
URL: https://github.com/apache/hudi-rs/pull/702
## Description
**An integration branch, not a replacement for review.** This squashes the eighteen open PRs #678–#700 into one commit so the stack gets a single CI run. Every one of those PRs stays open, and the per-PR diff remains the reviewable unit — please review them there, not here.
Why it exists: CI has never run on any of #678–#700, because a fork PR needs a committer to approve the workflow, and eighteen approvals is eighteen chances to stop. One branch is one approval and one signal on the tree that would actually merge.
What it contains, in the order the stack built it:
- **Correctness in the log path** — #678 gate the log-block scan on instant state; #679 fold partial updates in both ordering directions; #680 cover event-time ordering losing in the gold corpus; #682 walk log block headers before the gates decide
- **The reader made asynchronous** — #686 read log files asynchronously in batches; #687 the merge iterator owns the base-file pull; #688 merge base and log as a stream, deleting the synchronous bridge; #689 bound the merge chunk
- **HFile support** — #691 ranged, window-coalesced base file reads; #692 merge HFile log blocks instead of dropping them; #694 seek to the blocks a key predicate can satisfy; #697 copy a record rather than its whole block, making block iteration linear instead of quadratic
- **The metadata table** — #693 merge its records by the payload's own rule; #695 read the files partition through the v2 reader; #696 decode a log block holding more than one batch, and switch the read seam over
- **Two leaves** — #699 resolve the base file format from the path rather than config alone; #700 read the metadata table as Arrow through a plain C ABI, so a JVM caller reaches it without Gluten or Velox
- **Build ergonomics** — #681 let consumers opt out of the merge map's on-disk tier
Two things worth knowing about how it was assembled. **#692 was stranded**: everything above #691 had been built on #693, so `4816cc7` was an ancestor of no other PR and the stack tip did not contain the log-block merge fix. It is rebased in here, and its two tests needed one adaptation — `resolve_reader_context` gained the per-path format argument from #699. **#694 and #697 conflict** in `crates/core/src/hfile/key.rs` over `content_offset` visibility against `from_content`; resolved by keeping both, with `content_offset` private.
The diff is far over this repo's 1000-line guidance. That is the point of the branch rather than an oversight: it is eighteen separately reviewed changes in one commit, not one change of that size.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
Verified on this tree, after the squash:
```
cargo test -p hudi-core --lib 1397 passed, 0 failed
cargo test -p hudi-core --lib --no-default-features 1377 passed, 0 failed
cargo test --workspace --all-targets --all-features 92 table_read_tests, 4 gold parity,
39 + 21 + 12 hudi-datafusion,
20 statistics, 6 hudi-jvm-ffi, all green
cargo clippy -p hudi-core --all-targets -- -D warnings clean
cargo clippy -p hudi-core --lib --no-default-features -- -D warnings clean
cargo fmt --all -- --check clean
```
The two tests ported from #692 were run by name after the rebase, since they were the ones that could have rotted against the newer API: `v2_reads_every_metadata_partition_with_its_log_blocks` and `v2_reads_a_files_slice_matching_the_metadata_table_reader`, both pass.
**Not covered, and stated rather than implied.** The JVM half of #700's ownership contract — that the Arrow stream is freed exactly once including on an exception path, and that a Rust panic does not abort a real JVM — is asserted on the Rust side only. Proving it from Java needs a test module this repo does not have yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): read Hudi metadata tables through the version two file group reader [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #702:
URL: https://github.com/apache/hudi-rs/pull/702#issuecomment-5516633751
Closing this PR in favor of #706
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): read Hudi metadata tables through the version two file group reader [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua closed pull request #702: feat(core): read Hudi metadata tables through the version two file group reader
URL: https://github.com/apache/hudi-rs/pull/702
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org