You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2026/09/24 06:01:33 UTC
[PR] fix(reader): read Avro enum columns and V2 Avro data log blocks [hudi-rs]
yihua opened a new pull request, #803:
URL: https://github.com/apache/hudi-rs/pull/803
## Description
Closes #802.
Two merge-on-read log block reads fail.
- **V2 Avro data blocks.** Writers before table version 6 (Hudi 0.13 and earlier) stamp log blocks V2, and a table upgraded to version 6 keeps them until compaction. The reader accepted only V3 (`Only support log block version 3 but got: V2`). V2 and V3 Avro data blocks share one layout (version, record count, records), so Avro data blocks now accept V2. A V2 delete block is Kryo-serialized, which this reader cannot decode, so it stays refused with an error naming the block and version.
- **Avro enum columns.** arrow-avro decodes an enum as `Dictionary(Int32, Utf8)`, while the table schema and parquet base files hold it as `Utf8`, so merging a log update or insert on an enum column failed in reconcile. The Avro block decoder now unpacks dictionaries at any depth, and `AvroBlockDecoder::schema()` reports the unpacked schema to match.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
A new integration test builds a version 6 MOR table whose log block updates and inserts rows on an enum column, at block versions V2 and V3, and reads it through `Table::read`; both fail without the fix. A decoder unit test covers an enum at the top level, in a nullable union, a record, an array and a map. Content tests cover a V2 Avro block decoded, a V2 delete block refused and a V1 block refused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(reader): read Avro enum columns and V2 Avro data log blocks [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #803:
URL: https://github.com/apache/hudi-rs/pull/803
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(reader): read Avro enum columns and V2 Avro data log blocks [hudi-rs]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #803:
URL: https://github.com/apache/hudi-rs/pull/803#issuecomment-5808824946
## [Codecov](https://app.codecov.io/gh/apache/hudi-rs/pull/803?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `96.49123%` with `6 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 79.73%. Comparing base ([`47112da`](https://app.codecov.io/gh/apache/hudi-rs/commit/47112da1e6bd41a9501cb3aa0a315054e12c81cd?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f7eb1a3`](https://app.codecov.io/gh/apache/hudi-rs/commit/f7eb1a3c35415d77eabff435e3dfb4dc8d9219db?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi-rs/pull/803?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [crates/core/src/file\_group/log\_file/avro.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/803?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ffile_group%2Flog_file%2Favro.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL2ZpbGVfZ3JvdXAvbG9nX2ZpbGUvYXZyby5ycw==) | 91.30% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/803?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [crates/core/src/file\_group/log\_file/content.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/803?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ffile_group%2Flog_file%2Fcontent.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL2ZpbGVfZ3JvdXAvbG9nX2ZpbGUvY29udGVudC5ycw==) | 83.33% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/803?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [crates/core/tests/log\_block\_compat\_tests.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/803?src=pr&el=tree&filepath=crates%2Fcore%2Ftests%2Flog_block_compat_tests.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvdGVzdHMvbG9nX2Jsb2NrX2NvbXBhdF90ZXN0cy5ycw==) | 99.15% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/803?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #803 +/- ##
==========================================
+ Coverage 79.50% 79.73% +0.23%
==========================================
Files 126 127 +1
Lines 12371 12533 +162
==========================================
+ Hits 9835 9993 +158
- Misses 2536 2540 +4
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/hudi-rs/pull/803?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org