You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "linliu-code (via GitHub)" <gi...@apache.org> on 2026/08/25 23:29:46 UTC
[PR] fix(core): merge HFile log blocks instead of silently dropping them [hudi-rs]
linliu-code opened a new pull request, #692:
URL: https://github.com/apache/hudi-rs/pull/692
## Description
**Stacked on #691, which is stacked on #689. Review those first.** Until they merge the diff here shows their commits too; the delta is `fd712c9..HEAD`, three files.
A file slice holding an HFile log block reads as though the slice had no log blocks: the read succeeds and returns the base file's rows. Nothing errors, and no counter records a skipped block, so the loss is silent. Version two is the default reader, and the version one fallback does not cover this because version two never reports an inability, it reports success.
The block type was skipped at five points: the prefetch window planner, pass one's classification into the per-instant map, pass three's decode dispatch, pass three's buffer dispatch, and the buffer itself, whose `if let LogBlockContent::Records(..)` has no else and discards anything other shape. Admitting the type at the first four alone would not have helped, because the buffer takes Arrow and nothing else.
An HFile block's records are Avro values stored under HFile keys, so they are decoded against the schema in the block header, which is the same resolution an Avro block already gets, and the buffer receives batches like every other data block type. That resolution now lives in one helper shared by both paths so they cannot drift on promotions. Whether an HFile block is decoded or handed back as key-value pairs is the caller's choice, because the metadata table reader wants the pairs.
A writer may leave the record's key field empty because the HFile entry key already holds it, so an empty field is filled from the entry key. This shares one implementation with the base file reader added in #691, including its guards: a populated value and a null value are both left alone.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
`harness_hfile_log_block_rejected` asserted `Invalid block type: 4`, which held upstream where HFile blocks were unrecognised, and was ignored with a note that the expectation needed re-deriving rather than inverting. It is re-derived and un-ignored as `harness_hfile_log_block_merges`.
Its expectation comes from the inputs, not from this reader: the base file was read with the parquet reader, and the block's two records were decoded with `apache_avro::from_avro_datum` against the schema in the block header. Both sides carry `ts = 100`, so event-time ordering ties and the case's commit-time ordering decides, which makes the log write win on two of the four keys. Without the fix the read returns the base file unchanged, so the case fails on the commit time of the first row.
The `key` column is asserted because it is where the entry key is filled. The base file holds it and the log writer left it empty, so without the fill two rows read as empty next to two holding their own keys.
This is **not** a Hudi parity oracle. The fixture has no gold data, because Spark cannot read it either, so nothing here checks hudi-rs against Hudi; it checks the merge against the decoded inputs.
Run locally: 1337 pass with default features, 1317 with `--no-default-features`, clippy clean on both feature sets with warnings denied, fmt clean, and the metadata table reader's own test still passes, which is what confirms that path still receives raw key-value pairs. No CI run has happened: fork pull requests sit at `action_required` until a committer approves them, so all of the above is local only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(core): cover every metadata partition's log blocks through reader v2 [hudi-rs]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #692:
URL: https://github.com/apache/hudi-rs/pull/692#issuecomment-5517118046
## [Codecov](https://app.codecov.io/gh/apache/hudi-rs/pull/692?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 79.62%. Comparing base ([`04a3853`](https://app.codecov.io/gh/apache/hudi-rs/commit/04a3853ada7adbf18612b7a839553c3f924b9ec0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f7a7a11`](https://app.codecov.io/gh/apache/hudi-rs/commit/f7a7a11baa809ae02dcd5c903ac27fab5cee3abb?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #692 +/- ##
=======================================
Coverage 79.62% 79.62%
=======================================
Files 121 121
Lines 11892 11892
=======================================
Hits 9469 9469
Misses 2423 2423
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/hudi-rs/pull/692?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(core): cover every metadata partition's log blocks through reader v2 [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #692:
URL: https://github.com/apache/hudi-rs/pull/692
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org