You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "shangxinli (via GitHub)" <gi...@apache.org> on 2026/06/26 22:54:21 UTC

[PR] [MINOR] Add multi-block HBase-reader compat test for native HFile writer [hudi]

shangxinli opened a new pull request, #19080:
URL: https://github.com/apache/hudi/pull/19080

### Change Logs

The existing `TestHFileCompatibility#testHbaseReaderSucceedsWhenKeyValueVersionIsSetTo1` covers the single-data-block happy path with 5 records, which does not stress the load-on-open data structures (data-block index, meta-block index) that `HFileWriterImpl`'s trailer points at.

This PR adds a parameterized cross-version compat test that writes 5,000 records across 5 key/value shapes representative of the Metadata Table (MDT) partitions (`FILES`, `COLUMN_STATS`, `RECORD_INDEX`, `SECONDARY_INDEX`, `BLOOM_FILTERS`), reopens each file with HBase 2.4.x `HFile.createReader`, and asserts:

- the trailer parses (no `CorruptHFileException`),
- `getEntries()` matches the count written,
- a full forward scan returns every key in order.

A future trailer-layout drift in `HFileWriterImpl` (field reorder, missed field, width change) would fail at least one assertion across these shapes, before the change reaches MDT files in production and breaks readers still running an older Hudi release.

### Impact

Test-only. No production code changes. Added test runs in `hudi-io` module, ~2 seconds.

### Risk level

**none** — test-only.

### Documentation Update

N/A.

### Contributor's checklist

- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable *(this PR **is** the test)*
- [x] CI passed

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] Add multi-block HBase-reader compat test for native HFile writer [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #19080:
URL: https://github.com/apache/hudi/pull/19080#issuecomment-4813986422

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "fdc5940bda39ea4693144b01c9e72e1dc00a7558",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14972",
       "triggerID" : "fdc5940bda39ea4693144b01c9e72e1dc00a7558",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1a347766e0f8b28ed0d9be30ad8e47e3b129c826",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1a347766e0f8b28ed0d9be30ad8e47e3b129c826",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * fdc5940bda39ea4693144b01c9e72e1dc00a7558 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14972) 
   * 1a347766e0f8b28ed0d9be30ad8e47e3b129c826 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] Add multi-block HBase-reader compat test for native HFile writer [hudi]

Posted by "shangxinli (via GitHub)" <gi...@apache.org>.

shangxinli commented on PR #19080:
URL: https://github.com/apache/hudi/pull/19080#issuecomment-4814035655

   FYI — cherry-picked internally at Uber to gate our 0.14 → 1.2 cutover ahead of the OSS review cycle: uber-code/data-hoodie_oss#273. Any feedback here will be back-ported.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] Add multi-block HBase-reader compat test for native HFile writer [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #19080:
URL: https://github.com/apache/hudi/pull/19080#issuecomment-4814596549

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "fdc5940bda39ea4693144b01c9e72e1dc00a7558",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14972",
       "triggerID" : "fdc5940bda39ea4693144b01c9e72e1dc00a7558",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1a347766e0f8b28ed0d9be30ad8e47e3b129c826",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14975",
       "triggerID" : "1a347766e0f8b28ed0d9be30ad8e47e3b129c826",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1a347766e0f8b28ed0d9be30ad8e47e3b129c826 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14975) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] Add multi-block HBase-reader compat test for native HFile writer [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #19080:
URL: https://github.com/apache/hudi/pull/19080#issuecomment-4813968372

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "fdc5940bda39ea4693144b01c9e72e1dc00a7558",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fdc5940bda39ea4693144b01c9e72e1dc00a7558",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * fdc5940bda39ea4693144b01c9e72e1dc00a7558 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] Add multi-block HBase-reader compat test for native HFile writer [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #19080:
URL: https://github.com/apache/hudi/pull/19080#issuecomment-4814399265

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "fdc5940bda39ea4693144b01c9e72e1dc00a7558",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14972",
       "triggerID" : "fdc5940bda39ea4693144b01c9e72e1dc00a7558",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1a347766e0f8b28ed0d9be30ad8e47e3b129c826",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14975",
       "triggerID" : "1a347766e0f8b28ed0d9be30ad8e47e3b129c826",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * fdc5940bda39ea4693144b01c9e72e1dc00a7558 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14972) 
   * 1a347766e0f8b28ed0d9be30ad8e47e3b129c826 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14975) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] Add MDT-partition-shape parameterized HBase-read compat test [hudi]

Posted by "shangxinli (via GitHub)" <gi...@apache.org>.

shangxinli commented on PR #19080:
URL: https://github.com/apache/hudi/pull/19080#issuecomment-4833574220

   Thanks @yihua — #19071 and #19083 cover the generic HBase-reader → native multi-block path (point lookup, `seekBefore`, full scan, byte-parity, NONE+GZIP), which is the bulk of what this PR was originally guarding.
   
   The piece still not covered there: those tests exercise a single generic multi-block file. MDT writes files with five distinct key/value sizing profiles (`FILES`, `COLUMN_STATS`, `RECORD_INDEX`, `SECONDARY_INDEX`, `BLOOM_FILTERS`) and the trailer index layout is per-file based on actual key/value sizes — a future writer change that only perturbs one shape's sizing would slip past the generic tests.
   
   I've rescoped this PR to just that parameterized MDT-shape sweep, stacked on #19083. Happy to fold it into #19083 directly if you'd rather, just say the word and I'll close this and send a patch there.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] Add multi-block HBase-reader compat test for native HFile writer [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #19080:
URL: https://github.com/apache/hudi/pull/19080#issuecomment-4814261034

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "fdc5940bda39ea4693144b01c9e72e1dc00a7558",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14972",
       "triggerID" : "fdc5940bda39ea4693144b01c9e72e1dc00a7558",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1a347766e0f8b28ed0d9be30ad8e47e3b129c826",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14975",
       "triggerID" : "1a347766e0f8b28ed0d9be30ad8e47e3b129c826",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * fdc5940bda39ea4693144b01c9e72e1dc00a7558 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14972) 
   * 1a347766e0f8b28ed0d9be30ad8e47e3b129c826 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14975) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [MINOR] Add MDT-partition-shape parameterized HBase-read compat test [hudi]

Posted by "shangxinli (via GitHub)" <gi...@apache.org>.

shangxinli commented on PR #19080:
URL: https://github.com/apache/hudi/pull/19080#issuecomment-6272659204

   @yihua #19071 and #19083 are both merged now, so this is unblocked. It is still just the MDT-shape parameterized sweep that sits on top of them, so happy either way — keep it as its own PR, or I close this and send the sweep as a patch somewhere you prefer. Just say which and I will rebase accordingly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org