You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2026/05/23 04:33:39 UTC
[PR] [HUDI-18825] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
nsivabalan opened a new pull request, #18826:
URL: https://github.com/apache/hudi/pull/18826
### Change Logs
Fixes https://github.com/apache/hudi/issues/18825.
During RLI bootstrap, `HoodieBackedTableMetadataWriter#initializeRecordIndexPartition`
previously persisted the full materialized RDD of RLI records
(`records.persist("MEMORY_AND_DISK_SER")`) and counted it
(`records.count()`) purely to obtain the total record count used to size
the RLI file groups. This is the primary latency and memory bottleneck of
RLI bootstrap on large tables.
This PR:
- Replaces the persist+count of the RLI records with a direct row-count
read from each base file's footer metadata via
`FileFormatUtils.getRowCount(...)`. Footer reads are O(1) per file and
avoid materializing the record dataset.
- Reuses the file slices already collected for record-key reading; for
MOR, base files are extracted from the file slices already in hand
(estimation uses base file row counts only; log file deltas are bounded
and not material for sizing).
- Bypasses estimation entirely when the user pins the RLI file group
count via `min == max`, and uses the configured value directly.
- Adds `TestHoodieBackedMetadata#testRecordIndexFileGroupEstimation` and
`testRecordIndexWithFixedFileGroupCount` covering both COW and MOR.
### Impact
Decouples RLI file group sizing from materializing the record-keys RDD.
Eliminates the persist+count pass during RLI bootstrap, which is the
primary latency bottleneck on large tables.
### Risk level
low
### Documentation Update
No new configs or user-facing behavior changes.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524254281
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `85.71429%` with `5 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 47.45%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 85.71% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (e299b84) and HEAD (dfe8b21). Click for more details.
>
> <details><summary>HEAD has 22 uploads less than BASE</summary>
>
>| Flag | BASE (e299b84) | HEAD (dfe8b21) |
>|------|------|------|
>|spark-java-tests|15|2|
>|spark-scala-tests|10|3|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
=============================================
- Coverage 68.92% 47.45% -21.48%
+ Complexity 29076 16634 -12442
=============================================
Files 2509 2040 -469
Lines 139470 115607 -23863
Branches 17117 14863 -2254
=============================================
- Hits 96130 54860 -41270
- Misses 35584 54915 +19331
+ Partials 7756 5832 -1924
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.51% <85.71%> (-16.84%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.97% <45.71%> (-12.31%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `67.60% <85.71%> (-16.20%)` | :arrow_down: |
... and [1569 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524289708
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `91.42857%` with `3 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 68.71%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 91.42% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
============================================
- Coverage 68.92% 68.71% -0.22%
+ Complexity 29076 28982 -94
============================================
Files 2509 2509
Lines 139470 139495 +25
Branches 17117 17115 -2
============================================
- Hits 96130 95852 -278
- Misses 35584 35917 +333
+ Partials 7756 7726 -30
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.42% <45.71%> (-0.02%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.48% <91.42%> (-0.87%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.19% <85.71%> (-0.08%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.48% <85.71%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `83.88% <91.42%> (+0.09%)` | :arrow_up: |
... and [55 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524267195
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `85.71429%` with `5 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 61.42%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 85.71% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (e299b84) and HEAD (dfe8b21). Click for more details.
>
> <details><summary>HEAD has 15 uploads less than BASE</summary>
>
>| Flag | BASE (e299b84) | HEAD (dfe8b21) |
>|------|------|------|
>|spark-java-tests|15|5|
>|spark-scala-tests|10|5|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
============================================
- Coverage 68.92% 61.42% -7.50%
+ Complexity 29076 25904 -3172
============================================
Files 2509 2509
Lines 139470 139495 +25
Branches 17117 17115 -2
============================================
- Hits 96130 85690 -10440
- Misses 35584 46417 +10833
+ Partials 7756 7388 -368
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.42% <45.71%> (-0.02%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `31.80% <85.71%> (-17.55%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.78% <45.71%> (-12.50%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.48% <85.71%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `73.64% <85.71%> (-10.16%)` | :arrow_down: |
... and [556 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-18825] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524193481
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `20.00000%` with `28 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 44.90%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 20.00% | [27 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (e299b84) and HEAD (dfe8b21). Click for more details.
>
> <details><summary>HEAD has 28 uploads less than BASE</summary>
>
>| Flag | BASE (e299b84) | HEAD (dfe8b21) |
>|------|------|------|
>|spark-java-tests|15|0|
>|spark-client-hadoop-common|1|0|
>|spark-scala-tests|10|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
=============================================
- Coverage 68.92% 44.90% -24.02%
+ Complexity 29076 8575 -20501
=============================================
Files 2509 1202 -1307
Lines 139470 62998 -76472
Branches 17117 6852 -10265
=============================================
- Hits 96130 28290 -67840
+ Misses 35584 31587 -3997
+ Partials 7756 3121 -4635
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `49.47% <20.00%> (-34.33%)` | :arrow_down: |
... and [2104 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #18826:
URL: https://github.com/apache/hudi/pull/18826#discussion_r3292770271
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -824,28 +824,54 @@ private Pair<Integer, HoodieData<HoodieRecord>> initializeRecordIndexPartition(
dataMetaClient,
dataWriteConfig);
- // Initialize the file groups
- final int fileGroupCount = estimateFileGroupCount(records);
+ Pair<Integer, Integer> bounds = getRLIFileGroupCountBounds();
+ int minFileGroupCount = bounds.getLeft();
+ int maxFileGroupCount = bounds.getRight();
+
+ int fileGroupCount;
+ if (minFileGroupCount != maxFileGroupCount) {
+ // Estimate file group count based on record count read from base file footer metadata.
+ // Avoids the expensive records.persist() + records.count() pass over materialized records.
+ List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs = latestMergedPartitionFileSliceList.stream()
Review Comment:
For slices with log files only, should we just log warn and fall back to the old supplier path for just those slices? We can take it up as follow-up to factor in log files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #18826:
URL: https://github.com/apache/hudi/pull/18826#discussion_r3426151437
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -824,28 +824,54 @@ private Pair<Integer, HoodieData<HoodieRecord>> initializeRecordIndexPartition(
dataMetaClient,
dataWriteConfig);
- // Initialize the file groups
- final int fileGroupCount = estimateFileGroupCount(records);
+ Pair<Integer, Integer> bounds = getRLIFileGroupCountBounds();
+ int minFileGroupCount = bounds.getLeft();
+ int maxFileGroupCount = bounds.getRight();
+
+ int fileGroupCount;
+ if (minFileGroupCount != maxFileGroupCount) {
+ // Estimate file group count based on record count read from base file footer metadata.
+ // Avoids the expensive records.persist() + records.count() pass over materialized records.
+ List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs = latestMergedPartitionFileSliceList.stream()
+ .filter(p -> p.getRight().getBaseFile().isPresent())
+ .map(p -> Pair.of(p.getLeft(), p.getRight().getBaseFile().get()))
+ .collect(Collectors.toList());
+ fileGroupCount = estimateFileGroupCountFromBaseFiles(
+ partitionBaseFilePairs, minFileGroupCount, maxFileGroupCount);
+ LOG.info("Estimated {} file groups from base file footer metadata", fileGroupCount);
+ } else {
+ // min == max: skip estimation, use the fixed value directly.
+ fileGroupCount = minFileGroupCount;
+ LOG.info("Using user-configured file group count: {}", fileGroupCount);
+ }
+
LOG.info("Initializing record index with {} file groups.", fileGroupCount);
return Pair.of(fileGroupCount, records);
}
- private int estimateFileGroupCount(HoodieData<HoodieRecord> records) {
- int minFileGroupCount;
- int maxFileGroupCount;
+ /**
+ * Returns the (min, max) file group count bounds for RLI based on which RLI variant is enabled.
+ */
+ private Pair<Integer, Integer> getRLIFileGroupCountBounds() {
if (dataWriteConfig.isRecordLevelIndexEnabled()) {
- minFileGroupCount = dataWriteConfig.getRecordLevelIndexMinFileGroupCount();
- maxFileGroupCount = dataWriteConfig.getRecordLevelIndexMaxFileGroupCount();
- } else {
- minFileGroupCount = dataWriteConfig.getGlobalRecordLevelIndexMinFileGroupCount();
- maxFileGroupCount = dataWriteConfig.getGlobalRecordLevelIndexMaxFileGroupCount();
- }
- Supplier<Long> recordCountSupplier = () -> {
- records.persist("MEMORY_AND_DISK_SER");
- long count = records.count();
- LOG.info("Initializing record index with {} mappings", count);
- return count;
- };
+ return Pair.of(dataWriteConfig.getRecordLevelIndexMinFileGroupCount(),
Review Comment:
in validateRecordIndex, we also got a count on the records, do we need to call ecords.persist("MEMORY_AND_DISK_SER") from there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4807318767
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dfe8b2148953a1079b68bbe0854ecd405657c3c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14242",
"triggerID" : "dfe8b2148953a1079b68bbe0854ecd405657c3c3",
"triggerType" : "PUSH"
}, {
"hash" : "4c398477979d2d99f9b4ed61ed1cfbae8631ea1f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14806",
"triggerID" : "4c398477979d2d99f9b4ed61ed1cfbae8631ea1f",
"triggerType" : "PUSH"
}, {
"hash" : "bec52e4dc96c4a01c7fc752144bc535fd6720941",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14945",
"triggerID" : "bec52e4dc96c4a01c7fc752144bc535fd6720941",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4c398477979d2d99f9b4ed61ed1cfbae8631ea1f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14806)
* bec52e4dc96c4a01c7fc752144bc535fd6720941 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14945)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-18825] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #18826:
URL: https://github.com/apache/hudi/pull/18826#discussion_r3292093962
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -824,28 +824,54 @@ private Pair<Integer, HoodieData<HoodieRecord>> initializeRecordIndexPartition(
dataMetaClient,
dataWriteConfig);
- // Initialize the file groups
- final int fileGroupCount = estimateFileGroupCount(records);
+ Pair<Integer, Integer> bounds = getRLIFileGroupCountBounds();
Review Comment:
🤖 The old code persisted `records` as a side effect inside `recordCountSupplier`, which meant `validateRecordIndex` (when `hoodie.metadata.record.index.enable.validation.on.initialization=true`) could call `recordIndexRecords.count()` cheaply from cache. After this change `records` is never persisted, so the validation path will recompute the entire RLI records pipeline (re-reading base + log files) to satisfy `count()`. Is this regression for validation users intentional? If so, a comment noting the trade-off would help future readers.
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -857,6 +883,32 @@ private int estimateFileGroupCount(HoodieData<HoodieRecord> records) {
);
}
+ /**
+ * Sums row counts read from each base file's footer metadata, in parallel via the engine context.
+ * Used in place of materializing and counting an RDD of records during RLI bootstrap.
+ */
+ private long estimateRecordCountFromBaseFiles(List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs) {
+ if (partitionBaseFilePairs.isEmpty()) {
+ return 0L;
+ }
+ int parallelism = Math.min(partitionBaseFilePairs.size(),
+ dataWriteConfig.getMetadataConfig().getRecordIndexMaxParallelism());
+ StorageConfiguration<?> storageConfBroadcast = storageConf;
+ return engineContext.parallelize(partitionBaseFilePairs, parallelism)
+ .map(partitionAndBaseFile -> {
+ HoodieBaseFile baseFile = partitionAndBaseFile.getValue();
+ StoragePath path = baseFile.getStoragePath();
+ try {
+ HoodieStorage storage = HoodieStorageUtils.getStorage(path, storageConfBroadcast);
+ return HoodieIOFactory.getIOFactory(storage).getFileFormatUtils(path).getRowCount(storage, path);
+ } catch (Exception e) {
+ LOG.warn("Failed to read row count from base file footer: {}", path, e);
+ return 0L;
Review Comment:
🤖 Swallowing the exception and returning 0L per failed file means many failing files would silently under-estimate `estimatedRecordCount` (and therefore under-provision file groups) with only a per-file WARN as evidence. The analogous `countRecordsInHFiles` above throws `HoodieIOException` on read failure. Should this at least surface the failure count somehow, or fail fast on persistent errors, since under-provisioned RLI file groups can be painful to fix later?
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -857,6 +883,32 @@ private int estimateFileGroupCount(HoodieData<HoodieRecord> records) {
);
}
+ /**
+ * Sums row counts read from each base file's footer metadata, in parallel via the engine context.
+ * Used in place of materializing and counting an RDD of records during RLI bootstrap.
+ */
+ private long estimateRecordCountFromBaseFiles(List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs) {
+ if (partitionBaseFilePairs.isEmpty()) {
+ return 0L;
+ }
+ int parallelism = Math.min(partitionBaseFilePairs.size(),
+ dataWriteConfig.getMetadataConfig().getRecordIndexMaxParallelism());
+ StorageConfiguration<?> storageConfBroadcast = storageConf;
+ return engineContext.parallelize(partitionBaseFilePairs, parallelism)
+ .map(partitionAndBaseFile -> {
+ HoodieBaseFile baseFile = partitionAndBaseFile.getValue();
Review Comment:
🤖 nit: the rest of this diff uses `getRight()` on `Pair` (e.g. line 836), so `partitionAndBaseFile.getValue()` is a bit inconsistent — could you change it to `getRight()` for uniformity?
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -824,28 +824,54 @@ private Pair<Integer, HoodieData<HoodieRecord>> initializeRecordIndexPartition(
dataMetaClient,
dataWriteConfig);
- // Initialize the file groups
- final int fileGroupCount = estimateFileGroupCount(records);
+ Pair<Integer, Integer> bounds = getRLIFileGroupCountBounds();
+ int minFileGroupCount = bounds.getLeft();
+ int maxFileGroupCount = bounds.getRight();
+
+ int fileGroupCount;
+ if (minFileGroupCount != maxFileGroupCount) {
+ // Estimate file group count based on record count read from base file footer metadata.
+ // Avoids the expensive records.persist() + records.count() pass over materialized records.
+ List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs = latestMergedPartitionFileSliceList.stream()
Review Comment:
🤖 For MOR, this drops any file slice that has no base file (log-only slice) and ignores log-file record contributions for slices that do have base files. Tables with heavy log-only inserts, or RLI bootstrap on a long-uncompacted MOR table, could see significant undercount that the growth factor may not absorb. The PR description acknowledges this; could you confirm whether realistic large-MOR scenarios were considered, or add a follow-up to factor in log files (e.g. via `HoodieLogFileReader` block header record counts)?
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -857,6 +883,32 @@ private int estimateFileGroupCount(HoodieData<HoodieRecord> records) {
);
}
+ /**
+ * Sums row counts read from each base file's footer metadata, in parallel via the engine context.
+ * Used in place of materializing and counting an RDD of records during RLI bootstrap.
+ */
+ private long estimateRecordCountFromBaseFiles(List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs) {
+ if (partitionBaseFilePairs.isEmpty()) {
+ return 0L;
+ }
+ int parallelism = Math.min(partitionBaseFilePairs.size(),
+ dataWriteConfig.getMetadataConfig().getRecordIndexMaxParallelism());
+ StorageConfiguration<?> storageConfBroadcast = storageConf;
Review Comment:
🤖 nit: `storageConfBroadcast` reads like a Spark `Broadcast<T>` variable, but it's just a local copy of the instance field for lambda capture. Could you rename it to something like `localStorageConf` or `capturedStorageConf` to avoid that confusion?
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #18826:
URL: https://github.com/apache/hudi/pull/18826#discussion_r3292775688
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestHoodieBackedMetadata.java:
##########
@@ -4379,6 +4379,104 @@ private void changeTableVersion(HoodieTableVersion version) throws IOException {
}
}
+ /**
+ * Validates that RLI initialization estimates file group count from base file footer metadata
+ * (instead of materializing and counting records) when min != max file group count.
+ */
+ @ParameterizedTest
+ @EnumSource(HoodieTableType.class)
+ public void testRecordIndexFileGroupEstimation(HoodieTableType tableType) throws Exception {
Review Comment:
Both this and `testRecordIndexWithFixedFileGroupCount` would pass even if `estimateRecordCountFromBaseFiles` returned 0 right? With 200 inserts, default maxFileGroupSizeBytes` and min=1/max=10, the estimator would return minFileGroupCount=1 regardless. The assertion is satisfied. But, i think we should configure a small `record.index.max.file.group.size.bytes` e.g. a few KB and/or a low average record size so the count actually drives the result. Wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4735850630
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dfe8b2148953a1079b68bbe0854ecd405657c3c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14242",
"triggerID" : "dfe8b2148953a1079b68bbe0854ecd405657c3c3",
"triggerType" : "PUSH"
}, {
"hash" : "4c398477979d2d99f9b4ed61ed1cfbae8631ea1f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14806",
"triggerID" : "4c398477979d2d99f9b4ed61ed1cfbae8631ea1f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4c398477979d2d99f9b4ed61ed1cfbae8631ea1f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14806)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524272694
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `85.71429%` with `5 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 65.97%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 85.71% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
============================================
- Coverage 68.92% 65.97% -2.96%
+ Complexity 29076 27825 -1251
============================================
Files 2509 2509
Lines 139470 139495 +25
Branches 17117 17115 -2
============================================
- Hits 96130 92031 -4099
- Misses 35584 39626 +4042
- Partials 7756 7838 +82
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.42% <45.71%> (-0.02%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `31.80% <85.71%> (-17.55%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.92% <85.71%> (-0.36%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.48% <85.71%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `75.83% <85.71%> (-7.97%)` | :arrow_down: |
... and [388 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524333329
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `91.42857%` with `3 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 68.92%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 91.42% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
============================================
- Coverage 68.92% 68.92% -0.01%
- Complexity 29076 29100 +24
============================================
Files 2509 2509
Lines 139470 139495 +25
Branches 17117 17115 -2
============================================
+ Hits 96130 96145 +15
- Misses 35584 35599 +15
+ Partials 7756 7751 -5
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.42% <45.71%> (-0.02%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.37% <91.42%> (+0.02%)` | :arrow_up: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.28% <85.71%> (+<0.01%)` | :arrow_up: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.48% <85.71%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `83.88% <91.42%> (+0.09%)` | :arrow_up: |
... and [13 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524297552
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `91.42857%` with `3 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 68.86%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 91.42% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
============================================
- Coverage 68.92% 68.86% -0.07%
+ Complexity 29076 29060 -16
============================================
Files 2509 2509
Lines 139470 139495 +25
Branches 17117 17115 -2
============================================
- Hits 96130 96060 -70
- Misses 35584 35699 +115
+ Partials 7756 7736 -20
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.42% <45.71%> (-0.02%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.11% <91.42%> (-0.24%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.19% <85.71%> (-0.08%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.48% <85.71%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `83.88% <91.42%> (+0.09%)` | :arrow_up: |
... and [28 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524259031
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `85.71429%` with `5 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 53.58%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 85.71% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (e299b84) and HEAD (dfe8b21). Click for more details.
>
> <details><summary>HEAD has 20 uploads less than BASE</summary>
>
>| Flag | BASE (e299b84) | HEAD (dfe8b21) |
>|------|------|------|
>|spark-java-tests|15|3|
>|spark-scala-tests|10|3|
>|common-and-other-modules|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
=============================================
- Coverage 68.92% 53.58% -15.34%
+ Complexity 29076 18598 -10478
=============================================
Files 2509 2040 -469
Lines 139470 115607 -23863
Branches 17117 14863 -2254
=============================================
- Hits 96130 61951 -34179
- Misses 35584 47212 +11628
+ Partials 7756 6444 -1312
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.16% <85.71%> (-17.19%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.97% <45.71%> (-12.31%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.48% <85.71%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `69.78% <85.71%> (-14.01%)` | :arrow_down: |
... and [1436 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524246696
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `51.42857%` with `17 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 46.11%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 51.42% | [15 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (e299b84) and HEAD (dfe8b21). Click for more details.
>
> <details><summary>HEAD has 25 uploads less than BASE</summary>
>
>| Flag | BASE (e299b84) | HEAD (dfe8b21) |
>|------|------|------|
>|spark-java-tests|15|0|
>|spark-scala-tests|10|2|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
=============================================
- Coverage 68.92% 46.11% -22.82%
+ Complexity 29076 15549 -13527
=============================================
Files 2509 1988 -521
Lines 139470 110324 -29146
Branches 17117 13654 -3463
=============================================
- Hits 96130 50875 -45255
- Misses 35584 53984 +18400
+ Partials 7756 5465 -2291
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.88% <45.71%> (-12.40%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `57.26% <51.42%> (-26.53%)` | :arrow_down: |
... and [1649 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524218983
@codope : closed the prev one and opened a new one.
pretty much what you suggested for earlier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-18825] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524181060
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dfe8b2148953a1079b68bbe0854ecd405657c3c3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14242",
"triggerID" : "dfe8b2148953a1079b68bbe0854ecd405657c3c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dfe8b2148953a1079b68bbe0854ecd405657c3c3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14242)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #18826:
URL: https://github.com/apache/hudi/pull/18826#discussion_r3426151437
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -824,28 +824,54 @@ private Pair<Integer, HoodieData<HoodieRecord>> initializeRecordIndexPartition(
dataMetaClient,
dataWriteConfig);
- // Initialize the file groups
- final int fileGroupCount = estimateFileGroupCount(records);
+ Pair<Integer, Integer> bounds = getRLIFileGroupCountBounds();
+ int minFileGroupCount = bounds.getLeft();
+ int maxFileGroupCount = bounds.getRight();
+
+ int fileGroupCount;
+ if (minFileGroupCount != maxFileGroupCount) {
+ // Estimate file group count based on record count read from base file footer metadata.
+ // Avoids the expensive records.persist() + records.count() pass over materialized records.
+ List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs = latestMergedPartitionFileSliceList.stream()
+ .filter(p -> p.getRight().getBaseFile().isPresent())
+ .map(p -> Pair.of(p.getLeft(), p.getRight().getBaseFile().get()))
+ .collect(Collectors.toList());
+ fileGroupCount = estimateFileGroupCountFromBaseFiles(
+ partitionBaseFilePairs, minFileGroupCount, maxFileGroupCount);
+ LOG.info("Estimated {} file groups from base file footer metadata", fileGroupCount);
+ } else {
+ // min == max: skip estimation, use the fixed value directly.
+ fileGroupCount = minFileGroupCount;
+ LOG.info("Using user-configured file group count: {}", fileGroupCount);
+ }
+
LOG.info("Initializing record index with {} file groups.", fileGroupCount);
return Pair.of(fileGroupCount, records);
}
- private int estimateFileGroupCount(HoodieData<HoodieRecord> records) {
- int minFileGroupCount;
- int maxFileGroupCount;
+ /**
+ * Returns the (min, max) file group count bounds for RLI based on which RLI variant is enabled.
+ */
+ private Pair<Integer, Integer> getRLIFileGroupCountBounds() {
if (dataWriteConfig.isRecordLevelIndexEnabled()) {
- minFileGroupCount = dataWriteConfig.getRecordLevelIndexMinFileGroupCount();
- maxFileGroupCount = dataWriteConfig.getRecordLevelIndexMaxFileGroupCount();
- } else {
- minFileGroupCount = dataWriteConfig.getGlobalRecordLevelIndexMinFileGroupCount();
- maxFileGroupCount = dataWriteConfig.getGlobalRecordLevelIndexMaxFileGroupCount();
- }
- Supplier<Long> recordCountSupplier = () -> {
- records.persist("MEMORY_AND_DISK_SER");
- long count = records.count();
- LOG.info("Initializing record index with {} mappings", count);
- return count;
- };
+ return Pair.of(dataWriteConfig.getRecordLevelIndexMinFileGroupCount(),
Review Comment:
in validateRecordIndex, we also got a count on the records, do we need to call `records.persist("MEMORY_AND_DISK_SER")` from there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524299650
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `91.42857%` with `3 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 68.88%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 91.42% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
============================================
- Coverage 68.92% 68.88% -0.05%
- Complexity 29076 29077 +1
============================================
Files 2509 2509
Lines 139470 139495 +25
Branches 17117 17115 -2
============================================
- Hits 96130 96090 -40
- Misses 35584 35660 +76
+ Partials 7756 7745 -11
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.42% <45.71%> (-0.02%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.29% <91.42%> (-0.06%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.19% <85.71%> (-0.08%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.48% <85.71%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `83.88% <91.42%> (+0.09%)` | :arrow_up: |
... and [21 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on code in PR #18826:
URL: https://github.com/apache/hudi/pull/18826#discussion_r3411494734
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestHoodieBackedMetadata.java:
##########
@@ -4379,6 +4379,104 @@ private void changeTableVersion(HoodieTableVersion version) throws IOException {
}
}
+ /**
+ * Validates that RLI initialization estimates file group count from base file footer metadata
+ * (instead of materializing and counting records) when min != max file group count.
+ */
+ @ParameterizedTest
+ @EnumSource(HoodieTableType.class)
+ public void testRecordIndexFileGroupEstimation(HoodieTableType tableType) throws Exception {
Review Comment:
testRecordIndexWithFixedFileGroupCount cannot exercise estimateRecordCountFromBaseFiles at all: min==max routes through the fixed-count branch that bypasses the estimator, so it only proves the bypass. For testRecordIndexFileGroupEstimation, the record-level min/max default to (1,10) and 200 small records under the default record.index.max.filegroup.size land at fileGroupCount=1, so the [1,10] assertion holds regardless of the footer count. To make the count drive the result, set a small withRecordIndexMaxFileGroupSizeBytes (a few KB) and assert fileGroupCount > 1 and < max, so a regression to 0 or a wrong count fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #18826:
URL: https://github.com/apache/hudi/pull/18826#discussion_r3426103744
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -824,28 +824,54 @@ private Pair<Integer, HoodieData<HoodieRecord>> initializeRecordIndexPartition(
dataMetaClient,
dataWriteConfig);
- // Initialize the file groups
- final int fileGroupCount = estimateFileGroupCount(records);
+ Pair<Integer, Integer> bounds = getRLIFileGroupCountBounds();
+ int minFileGroupCount = bounds.getLeft();
+ int maxFileGroupCount = bounds.getRight();
+
+ int fileGroupCount;
+ if (minFileGroupCount != maxFileGroupCount) {
+ // Estimate file group count based on record count read from base file footer metadata.
+ // Avoids the expensive records.persist() + records.count() pass over materialized records.
+ List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs = latestMergedPartitionFileSliceList.stream()
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4734273309
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dfe8b2148953a1079b68bbe0854ecd405657c3c3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14242",
"triggerID" : "dfe8b2148953a1079b68bbe0854ecd405657c3c3",
"triggerType" : "PUSH"
}, {
"hash" : "4c398477979d2d99f9b4ed61ed1cfbae8631ea1f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14806",
"triggerID" : "4c398477979d2d99f9b4ed61ed1cfbae8631ea1f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dfe8b2148953a1079b68bbe0854ecd405657c3c3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14242)
* 4c398477979d2d99f9b4ed61ed1cfbae8631ea1f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14806)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4808658078
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dfe8b2148953a1079b68bbe0854ecd405657c3c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14242",
"triggerID" : "dfe8b2148953a1079b68bbe0854ecd405657c3c3",
"triggerType" : "PUSH"
}, {
"hash" : "4c398477979d2d99f9b4ed61ed1cfbae8631ea1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14806",
"triggerID" : "4c398477979d2d99f9b4ed61ed1cfbae8631ea1f",
"triggerType" : "PUSH"
}, {
"hash" : "bec52e4dc96c4a01c7fc752144bc535fd6720941",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14945",
"triggerID" : "bec52e4dc96c4a01c7fc752144bc535fd6720941",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bec52e4dc96c4a01c7fc752144bc535fd6720941 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14945)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #18826:
URL: https://github.com/apache/hudi/pull/18826#discussion_r3292757484
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -857,6 +883,32 @@ private int estimateFileGroupCount(HoodieData<HoodieRecord> records) {
);
}
+ /**
+ * Sums row counts read from each base file's footer metadata, in parallel via the engine context.
+ * Used in place of materializing and counting an RDD of records during RLI bootstrap.
+ */
+ private long estimateRecordCountFromBaseFiles(List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs) {
+ if (partitionBaseFilePairs.isEmpty()) {
+ return 0L;
+ }
+ int parallelism = Math.min(partitionBaseFilePairs.size(),
+ dataWriteConfig.getMetadataConfig().getRecordIndexMaxParallelism());
+ StorageConfiguration<?> storageConfBroadcast = storageConf;
+ return engineContext.parallelize(partitionBaseFilePairs, parallelism)
+ .map(partitionAndBaseFile -> {
+ HoodieBaseFile baseFile = partitionAndBaseFile.getValue();
+ StoragePath path = baseFile.getStoragePath();
+ try {
+ HoodieStorage storage = HoodieStorageUtils.getStorage(path, storageConfBroadcast);
+ return HoodieIOFactory.getIOFactory(storage).getFileFormatUtils(path).getRowCount(storage, path);
+ } catch (Exception e) {
+ LOG.warn("Failed to read row count from base file footer: {}", path, e);
+ return 0L;
Review Comment:
hmm.. the bot raises a good point. Shall we match `countRecordsInHFiles` behaviour?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524251770
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `85.71429%` with `5 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 48.14%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 85.71% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (e299b84) and HEAD (dfe8b21). Click for more details.
>
> <details><summary>HEAD has 24 uploads less than BASE</summary>
>
>| Flag | BASE (e299b84) | HEAD (dfe8b21) |
>|------|------|------|
>|spark-java-tests|15|1|
>|spark-scala-tests|10|2|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
=============================================
- Coverage 68.92% 48.14% -20.78%
+ Complexity 29076 16547 -12529
=============================================
Files 2509 2018 -491
Lines 139470 113063 -26407
Branches 17117 14305 -2812
=============================================
- Hits 96130 54439 -41691
- Misses 35584 52862 +17278
+ Partials 7756 5762 -1994
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.94% <85.71%> (-16.41%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.88% <45.71%> (-12.40%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `67.60% <85.71%> (-16.20%)` | :arrow_down: |
... and [1583 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on code in PR #18826:
URL: https://github.com/apache/hudi/pull/18826#discussion_r3411492237
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -857,6 +883,32 @@ private int estimateFileGroupCount(HoodieData<HoodieRecord> records) {
);
}
+ /**
+ * Sums row counts read from each base file's footer metadata, in parallel via the engine context.
+ * Used in place of materializing and counting an RDD of records during RLI bootstrap.
+ */
+ private long estimateRecordCountFromBaseFiles(List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs) {
+ if (partitionBaseFilePairs.isEmpty()) {
+ return 0L;
+ }
+ int parallelism = Math.min(partitionBaseFilePairs.size(),
+ dataWriteConfig.getMetadataConfig().getRecordIndexMaxParallelism());
+ StorageConfiguration<?> storageConfBroadcast = storageConf;
+ return engineContext.parallelize(partitionBaseFilePairs, parallelism)
+ .map(partitionAndBaseFile -> {
+ HoodieBaseFile baseFile = partitionAndBaseFile.getValue();
+ StoragePath path = baseFile.getStoragePath();
+ try {
+ HoodieStorage storage = HoodieStorageUtils.getStorage(path, storageConfBroadcast);
+ return HoodieIOFactory.getIOFactory(storage).getFileFormatUtils(path).getRowCount(storage, path);
+ } catch (Exception e) {
+ LOG.warn("Failed to read row count from base file footer: {}", path, e);
+ return 0L;
Review Comment:
Concrete failure mode for the swallow-and-return-0L path: HFileUtils.getRowCount throws UnsupportedOperationException (hudi-common/src/main/java/org/apache/hudi/common/util/HFileUtils.java:109), so a data table with hoodie.base.file.format=HFILE hits the catch for every base file, sums to 0, and sizes RLI at minFileGroupCount with only per-file WARNs. Parquet/ORC are fine (footer reads). The second option above is the safer one: let UnsupportedOperationException propagate instead of catching Exception, and route that table to the legacy persist+count supplier, so only unsupported formats pay the old cost while Parquet/ORC keep the fast path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on code in PR #18826:
URL: https://github.com/apache/hudi/pull/18826#discussion_r3411486587
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -824,28 +824,54 @@ private Pair<Integer, HoodieData<HoodieRecord>> initializeRecordIndexPartition(
dataMetaClient,
dataWriteConfig);
- // Initialize the file groups
- final int fileGroupCount = estimateFileGroupCount(records);
+ Pair<Integer, Integer> bounds = getRLIFileGroupCountBounds();
Review Comment:
Verified: line 846 was the only persist() in this path, so after the change recordIndexRecords is never cached and recordIndexRecords.unpersist() at line 774 is now dead. With validation enabled, validateRecordIndex (line 885, recordIndexRecords.count()) recomputes the full key-reading pipeline that the bulkCommit at line 768 already materialized once. The guard option above is the targeted fix: persist right after readRecordKeysFromFileSliceSnapshot only when isRecordIndexInitializationValidationEnabled() is set, keep the unpersist, and the common no-validation path keeps the no-persist win.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #18826:
URL: https://github.com/apache/hudi/pull/18826#discussion_r3430817005
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -857,6 +883,32 @@ private int estimateFileGroupCount(HoodieData<HoodieRecord> records) {
);
}
+ /**
+ * Sums row counts read from each base file's footer metadata, in parallel via the engine context.
+ * Used in place of materializing and counting an RDD of records during RLI bootstrap.
+ */
+ private long estimateRecordCountFromBaseFiles(List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs) {
+ if (partitionBaseFilePairs.isEmpty()) {
+ return 0L;
+ }
+ int parallelism = Math.min(partitionBaseFilePairs.size(),
+ dataWriteConfig.getMetadataConfig().getRecordIndexMaxParallelism());
+ StorageConfiguration<?> storageConfBroadcast = storageConf;
+ return engineContext.parallelize(partitionBaseFilePairs, parallelism)
+ .map(partitionAndBaseFile -> {
+ HoodieBaseFile baseFile = partitionAndBaseFile.getValue();
+ StoragePath path = baseFile.getStoragePath();
+ try {
+ HoodieStorage storage = HoodieStorageUtils.getStorage(path, storageConfBroadcast);
+ return HoodieIOFactory.getIOFactory(storage).getFileFormatUtils(path).getRowCount(storage, path);
+ } catch (Exception e) {
+ LOG.warn("Failed to read row count from base file footer: {}", path, e);
+ return 0L;
Review Comment:
🤖 Worth noting that `HFileUtils.getRowCount` already throws `UnsupportedOperationException` today, so the current catch-all `Exception` block masks that as a per-file 0 and would silently estimate the whole table at 0 records (→ `minFileGroupCount`) for an HFile-formatted base file. Letting `UnsupportedOperationException` propagate (or probing `baseFileFormat` upfront) and routing to the legacy supplier in that case seems like the safer behavior — a one-time fall-through is much cheaper than under-provisioning RLI shards.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -859,6 +885,32 @@ private int estimateFileGroupCount(HoodieData<HoodieRecord> records) {
);
}
+ /**
+ * Sums row counts read from each base file's footer metadata, in parallel via the engine context.
+ * Used in place of materializing and counting an RDD of records during RLI bootstrap.
+ */
+ private long estimateRecordCountFromBaseFiles(List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs) {
+ if (partitionBaseFilePairs.isEmpty()) {
+ return 0L;
+ }
+ int parallelism = Math.min(partitionBaseFilePairs.size(),
+ dataWriteConfig.getMetadataConfig().getRecordIndexMaxParallelism());
+ StorageConfiguration<?> storageConfBroadcast = storageConf;
Review Comment:
🤖 nit: `storageConfBroadcast` reads like a Spark `Broadcast<T>` variable, which this isn't — it's a local capture to avoid serializing `this` in the lambda. Could you rename it to something like `localStorageConf` (and maybe add a brief inline comment explaining the lambda-capture idiom)?
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestHoodieBackedMetadata.java:
##########
@@ -4379,6 +4379,104 @@ private void changeTableVersion(HoodieTableVersion version) throws IOException {
}
}
+ /**
+ * Validates that RLI initialization estimates file group count from base file footer metadata
+ * (instead of materializing and counting records) when min != max file group count.
+ */
+ @ParameterizedTest
+ @EnumSource(HoodieTableType.class)
+ public void testRecordIndexFileGroupEstimation(HoodieTableType tableType) throws Exception {
Review Comment:
🤖 Confirming the math: 200 RLI records × `RECORD_INDEX_AVERAGE_RECORD_SIZE=48` ≈ 9.6 KB, which is well below the default `record.index.max.filegroup.size=1 GB`, so the estimator always clamps to `minFileGroupCount=1` regardless of what `estimateRecordCountFromBaseFiles` returns. As-is the test would pass even if the new footer-reading path silently returned 0, so it doesn't actually cover the regression it's meant to guard. Setting `withRecordIndexMaxFileGroupSizeBytes` to a small value (e.g. 1–2 KB) would make the row-count drive the result and exercise the new code path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4734228685
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dfe8b2148953a1079b68bbe0854ecd405657c3c3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14242",
"triggerID" : "dfe8b2148953a1079b68bbe0854ecd405657c3c3",
"triggerType" : "PUSH"
}, {
"hash" : "4c398477979d2d99f9b4ed61ed1cfbae8631ea1f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4c398477979d2d99f9b4ed61ed1cfbae8631ea1f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dfe8b2148953a1079b68bbe0854ecd405657c3c3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14242)
* 4c398477979d2d99f9b4ed61ed1cfbae8631ea1f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524236265
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `51.42857%` with `17 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 39.73%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 51.42% | [15 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (e299b84) and HEAD (dfe8b21). Click for more details.
>
> <details><summary>HEAD has 27 uploads less than BASE</summary>
>
>| Flag | BASE (e299b84) | HEAD (dfe8b21) |
>|------|------|------|
>|spark-java-tests|15|0|
>|spark-client-hadoop-common|1|0|
>|spark-scala-tests|10|1|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
=============================================
- Coverage 68.92% 39.73% -29.20%
+ Complexity 29076 12663 -16413
=============================================
Files 2509 1971 -538
Lines 139470 107938 -31532
Branches 17117 13143 -3974
=============================================
- Hits 96130 42885 -53245
- Misses 35584 59863 +24279
+ Partials 7756 5190 -2566
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.07% <45.71%> (-12.21%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `56.91% <51.42%> (-26.88%)` | :arrow_down: |
... and [1864 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524241047
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `51.42857%` with `17 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 46.59%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 51.42% | [15 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (e299b84) and HEAD (dfe8b21). Click for more details.
>
> <details><summary>HEAD has 26 uploads less than BASE</summary>
>
>| Flag | BASE (e299b84) | HEAD (dfe8b21) |
>|------|------|------|
>|spark-java-tests|15|0|
>|spark-scala-tests|10|1|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
=============================================
- Coverage 68.92% 46.59% -22.33%
+ Complexity 29076 15342 -13734
=============================================
Files 2509 1971 -538
Lines 139470 107938 -31532
Branches 17117 13143 -3974
=============================================
- Hits 96130 50297 -45833
- Misses 35584 52365 +16781
+ Partials 7756 5276 -2480
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.07% <45.71%> (-12.21%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `57.26% <51.42%> (-26.53%)` | :arrow_down: |
... and [1659 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524263469
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `85.71429%` with `5 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 61.21%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 85.71% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (e299b84) and HEAD (dfe8b21). Click for more details.
>
> <details><summary>HEAD has 16 uploads less than BASE</summary>
>
>| Flag | BASE (e299b84) | HEAD (dfe8b21) |
>|------|------|------|
>|spark-java-tests|15|5|
>|spark-scala-tests|10|4|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
============================================
- Coverage 68.92% 61.21% -7.72%
+ Complexity 29076 25759 -3317
============================================
Files 2509 2509
Lines 139470 139495 +25
Branches 17117 17115 -2
============================================
- Hits 96130 85389 -10741
- Misses 35584 46824 +11240
+ Partials 7756 7282 -474
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.42% <45.71%> (-0.02%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `31.80% <85.71%> (-17.55%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.96% <45.71%> (-12.32%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.48% <85.71%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `73.64% <85.71%> (-10.16%)` | :arrow_down: |
... and [558 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524288420
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `91.42857%` with `3 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 68.71%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 91.42% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
============================================
- Coverage 68.92% 68.71% -0.22%
+ Complexity 29076 28982 -94
============================================
Files 2509 2509
Lines 139470 139495 +25
Branches 17117 17115 -2
============================================
- Hits 96130 95852 -278
- Misses 35584 35917 +333
+ Partials 7756 7726 -30
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.42% <45.71%> (-0.02%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.48% <91.42%> (-0.87%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.19% <85.71%> (-0.08%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.48% <85.71%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `83.88% <91.42%> (+0.09%)` | :arrow_up: |
... and [55 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524285089
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `85.71429%` with `5 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 67.88%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 85.71% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
============================================
- Coverage 68.92% 67.88% -1.04%
+ Complexity 29076 28641 -435
============================================
Files 2509 2509
Lines 139470 139495 +25
Branches 17117 17115 -2
============================================
- Hits 96130 94697 -1433
- Misses 35584 37038 +1454
- Partials 7756 7760 +4
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.42% <45.71%> (-0.02%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.08% <85.71%> (-4.27%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.07% <85.71%> (-0.21%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.48% <85.71%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `75.83% <85.71%> (-7.97%)` | :arrow_down: |
... and [196 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524323035
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `91.42857%` with `3 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 68.92%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 91.42% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
============================================
- Coverage 68.92% 68.92% -0.01%
- Complexity 29076 29100 +24
============================================
Files 2509 2509
Lines 139470 139495 +25
Branches 17117 17115 -2
============================================
+ Hits 96130 96145 +15
- Misses 35584 35599 +15
+ Partials 7756 7751 -5
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.42% <45.71%> (-0.02%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.35% <91.42%> (+<0.01%)` | :arrow_up: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.28% <85.71%> (+<0.01%)` | :arrow_up: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.48% <85.71%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `83.88% <91.42%> (+0.09%)` | :arrow_up: |
... and [13 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #18826:
URL: https://github.com/apache/hudi/pull/18826#discussion_r3292760744
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -857,6 +883,32 @@ private int estimateFileGroupCount(HoodieData<HoodieRecord> records) {
);
}
+ /**
+ * Sums row counts read from each base file's footer metadata, in parallel via the engine context.
+ * Used in place of materializing and counting an RDD of records during RLI bootstrap.
+ */
+ private long estimateRecordCountFromBaseFiles(List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs) {
+ if (partitionBaseFilePairs.isEmpty()) {
+ return 0L;
+ }
+ int parallelism = Math.min(partitionBaseFilePairs.size(),
+ dataWriteConfig.getMetadataConfig().getRecordIndexMaxParallelism());
+ StorageConfiguration<?> storageConfBroadcast = storageConf;
+ return engineContext.parallelize(partitionBaseFilePairs, parallelism)
+ .map(partitionAndBaseFile -> {
+ HoodieBaseFile baseFile = partitionAndBaseFile.getValue();
+ StoragePath path = baseFile.getStoragePath();
+ try {
+ HoodieStorage storage = HoodieStorageUtils.getStorage(path, storageConfBroadcast);
+ return HoodieIOFactory.getIOFactory(storage).getFileFormatUtils(path).getRowCount(storage, path);
+ } catch (Exception e) {
+ LOG.warn("Failed to read row count from base file footer: {}", path, e);
+ return 0L;
Review Comment:
another option is to either probe the format upfront and fall through to the old path when `getRowCount` is unsupported, or let `UnsupportedOperationException` propagate (not be caught as Exception) and route the caller to the legacy supplier
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-18825] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524173761
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dfe8b2148953a1079b68bbe0854ecd405657c3c3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dfe8b2148953a1079b68bbe0854ecd405657c3c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dfe8b2148953a1079b68bbe0854ecd405657c3c3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524263957
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `85.71429%` with `5 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 61.21%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 85.71% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (e299b84) and HEAD (dfe8b21). Click for more details.
>
> <details><summary>HEAD has 16 uploads less than BASE</summary>
>
>| Flag | BASE (e299b84) | HEAD (dfe8b21) |
>|------|------|------|
>|spark-java-tests|15|5|
>|spark-scala-tests|10|4|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
============================================
- Coverage 68.92% 61.21% -7.72%
+ Complexity 29076 25759 -3317
============================================
Files 2509 2509
Lines 139470 139495 +25
Branches 17117 17115 -2
============================================
- Hits 96130 85389 -10741
- Misses 35584 46824 +11240
+ Partials 7756 7282 -474
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.42% <45.71%> (-0.02%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `31.80% <85.71%> (-17.55%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.96% <45.71%> (-12.32%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.48% <85.71%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `73.64% <85.71%> (-10.16%)` | :arrow_down: |
... and [558 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524253713
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `85.71429%` with `5 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 47.45%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 85.71% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (e299b84) and HEAD (dfe8b21). Click for more details.
>
> <details><summary>HEAD has 22 uploads less than BASE</summary>
>
>| Flag | BASE (e299b84) | HEAD (dfe8b21) |
>|------|------|------|
>|spark-java-tests|15|2|
>|spark-scala-tests|10|3|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
=============================================
- Coverage 68.92% 47.45% -21.48%
+ Complexity 29076 16634 -12442
=============================================
Files 2509 2040 -469
Lines 139470 115607 -23863
Branches 17117 14863 -2254
=============================================
- Hits 96130 54860 -41270
- Misses 35584 54915 +19331
+ Partials 7756 5832 -1924
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.51% <85.71%> (-16.84%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.97% <45.71%> (-12.31%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `67.60% <85.71%> (-16.20%)` | :arrow_down: |
... and [1569 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524298652
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `91.42857%` with `3 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 68.88%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 91.42% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
============================================
- Coverage 68.92% 68.88% -0.05%
- Complexity 29076 29077 +1
============================================
Files 2509 2509
Lines 139470 139495 +25
Branches 17117 17115 -2
============================================
- Hits 96130 96090 -40
- Misses 35584 35660 +76
+ Partials 7756 7745 -11
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.42% <45.71%> (-0.02%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.29% <91.42%> (-0.06%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.19% <85.71%> (-0.08%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.48% <85.71%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `83.88% <91.42%> (+0.09%)` | :arrow_up: |
... and [21 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524302966
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `91.42857%` with `3 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 68.92%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 91.42% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
============================================
- Coverage 68.92% 68.92% -0.01%
- Complexity 29076 29100 +24
============================================
Files 2509 2509
Lines 139470 139495 +25
Branches 17117 17115 -2
============================================
+ Hits 96130 96140 +10
- Misses 35584 35603 +19
+ Partials 7756 7752 -4
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.42% <45.71%> (-0.02%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.29% <91.42%> (-0.06%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.28% <85.71%> (+<0.01%)` | :arrow_up: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.48% <85.71%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `83.88% <91.42%> (+0.09%)` | :arrow_up: |
... and [16 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524361276
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dfe8b2148953a1079b68bbe0854ecd405657c3c3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14242",
"triggerID" : "dfe8b2148953a1079b68bbe0854ecd405657c3c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dfe8b2148953a1079b68bbe0854ecd405657c3c3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14242)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on code in PR #18826:
URL: https://github.com/apache/hudi/pull/18826#discussion_r3411489163
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -857,6 +883,32 @@ private int estimateFileGroupCount(HoodieData<HoodieRecord> records) {
);
}
+ /**
+ * Sums row counts read from each base file's footer metadata, in parallel via the engine context.
+ * Used in place of materializing and counting an RDD of records during RLI bootstrap.
+ */
+ private long estimateRecordCountFromBaseFiles(List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs) {
+ if (partitionBaseFilePairs.isEmpty()) {
+ return 0L;
+ }
+ int parallelism = Math.min(partitionBaseFilePairs.size(),
+ dataWriteConfig.getMetadataConfig().getRecordIndexMaxParallelism());
+ StorageConfiguration<?> storageConfBroadcast = storageConf;
Review Comment:
storageConfBroadcast is the existing name for this exact lambda-capture pattern at countRecordsInHFiles (HoodieBackedTableMetadataWriter.java:908), which this method mirrors. Renaming only the new copy diverges from that; if the Broadcast-implies-Spark concern is worth acting on, rename both occurrences together, otherwise keeping the existing name is the consistent choice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4977184809
@nsivabalan Can you please help to address the conflicts? Trying to cleanup our approved queue and get PRs landed in master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4814163746
hey @codope @danny0405 : can you review again. addressed all feedback
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #18826:
URL: https://github.com/apache/hudi/pull/18826#discussion_r3484705446
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -859,6 +905,32 @@ private int estimateFileGroupCount(HoodieData<HoodieRecord> records) {
);
}
+ /**
+ * Sums row counts read from each base file's footer metadata, in parallel via the engine context.
+ * Used in place of materializing and counting an RDD of records during RLI bootstrap.
+ *
+ * <p>Errors (e.g. {@link UnsupportedOperationException} from {@code HFileUtils.getRowCount},
+ * IO failures) are rethrown rather than swallowed — silently returning {@code 0L} per failed
+ * file would systematically undercount the estimate and provision RLI at
+ * {@code minFileGroupCount}. The contract mirrors {@link #countRecordsInHFiles(List)}.
+ */
+ private long estimateRecordCountFromBaseFiles(List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs) {
+ if (partitionBaseFilePairs.isEmpty()) {
+ return 0L;
+ }
Review Comment:
🤖 nit: `storageConfBroadcast` reads like a Spark `Broadcast<T>` variable, but this is just a local alias of `storageConf` to satisfy the lambda's effectively-final requirement. Could you rename it to something like `localStorageConf` or `storageConfRef` to avoid that confusion?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -826,28 +842,58 @@ private Pair<Integer, HoodieData<HoodieRecord>> initializeRecordIndexPartition(
dataMetaClient,
dataWriteConfig);
- // Initialize the file groups
- final int fileGroupCount = estimateFileGroupCount(records);
+ Pair<Integer, Integer> bounds = getRLIFileGroupCountBounds();
+ int minFileGroupCount = bounds.getLeft();
+ int maxFileGroupCount = bounds.getRight();
+
+ int fileGroupCount;
+ if (minFileGroupCount != maxFileGroupCount) {
+ // Estimate file group count based on record count read from base file footer metadata.
+ // Avoids the expensive records.persist() + records.count() pass over materialized records.
+ // Filtering to base-file-only slices is safe for RLI: log files in a slice can only carry
+ // updates or deletes for existing records, never new inserts (RLI is keyed by record key
+ // and updates rewrite the entry rather than add one), so they do not contribute to the
+ // distinct-record-count estimate that sizes the file groups.
+ List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs = latestMergedPartitionFileSliceList.stream()
+ .filter(p -> p.getRight().getBaseFile().isPresent())
+ .map(p -> Pair.of(p.getLeft(), p.getRight().getBaseFile().get()))
+ .collect(Collectors.toList());
+ fileGroupCount = estimateFileGroupCountFromBaseFiles(
+ partitionBaseFilePairs, minFileGroupCount, maxFileGroupCount);
+ LOG.info("Estimated {} file groups from base file footer metadata", fileGroupCount);
+ } else {
+ // min == max: skip estimation, use the fixed value directly.
+ fileGroupCount = minFileGroupCount;
+ LOG.info("Using user-configured file group count: {}", fileGroupCount);
+ }
+
LOG.info("Initializing record index with {} file groups.", fileGroupCount);
return Pair.of(fileGroupCount, records);
}
- private int estimateFileGroupCount(HoodieData<HoodieRecord> records) {
- int minFileGroupCount;
- int maxFileGroupCount;
+ /**
+ * Returns the (min, max) file group count bounds for RLI based on which RLI variant is enabled.
+ */
+ private Pair<Integer, Integer> getRLIFileGroupCountBounds() {
if (dataWriteConfig.isRecordLevelIndexEnabled()) {
- minFileGroupCount = dataWriteConfig.getRecordLevelIndexMinFileGroupCount();
- maxFileGroupCount = dataWriteConfig.getRecordLevelIndexMaxFileGroupCount();
- } else {
- minFileGroupCount = dataWriteConfig.getGlobalRecordLevelIndexMinFileGroupCount();
- maxFileGroupCount = dataWriteConfig.getGlobalRecordLevelIndexMaxFileGroupCount();
+ return Pair.of(dataWriteConfig.getRecordLevelIndexMinFileGroupCount(),
+ dataWriteConfig.getRecordLevelIndexMaxFileGroupCount());
}
- Supplier<Long> recordCountSupplier = () -> {
- records.persist("MEMORY_AND_DISK_SER");
- long count = records.count();
- LOG.info("Initializing record index with {} mappings", count);
- return count;
- };
+ return Pair.of(dataWriteConfig.getGlobalRecordLevelIndexMinFileGroupCount(),
+ dataWriteConfig.getGlobalRecordLevelIndexMaxFileGroupCount());
+ }
+
+ /**
+ * Estimates RLI file group count using a record count derived from base file footer metadata.
+ * Reading row counts from the footer is an O(1)-per-file operation that avoids materializing
+ * records to compute a count.
+ */
+ private int estimateFileGroupCountFromBaseFiles(List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs,
+ int minFileGroupCount,
+ int maxFileGroupCount) {
+ long estimatedRecordCount = estimateRecordCountFromBaseFiles(partitionBaseFilePairs);
+ LOG.info("Estimated total record count from base file footers: {}", estimatedRecordCount);
+ Supplier<Long> recordCountSupplier = () -> estimatedRecordCount;
Review Comment:
🤖 nit: the name `recordCountSupplier` carries a connotation of deferred/lazy computation, but `estimatedRecordCount` is already eagerly computed on the line above. Have you considered inlining it as `() -> estimatedRecordCount` directly in the call, or renaming to something like `precomputedCountSupplier` to make it clear the count is already in hand?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #18826:
URL: https://github.com/apache/hudi/pull/18826#discussion_r3485323310
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -826,28 +842,58 @@ private Pair<Integer, HoodieData<HoodieRecord>> initializeRecordIndexPartition(
dataMetaClient,
dataWriteConfig);
- // Initialize the file groups
- final int fileGroupCount = estimateFileGroupCount(records);
+ Pair<Integer, Integer> bounds = getRLIFileGroupCountBounds();
+ int minFileGroupCount = bounds.getLeft();
+ int maxFileGroupCount = bounds.getRight();
+
+ int fileGroupCount;
+ if (minFileGroupCount != maxFileGroupCount) {
+ // Estimate file group count based on record count read from base file footer metadata.
+ // Avoids the expensive records.persist() + records.count() pass over materialized records.
+ // Filtering to base-file-only slices is safe for RLI: log files in a slice can only carry
+ // updates or deletes for existing records, never new inserts (RLI is keyed by record key
+ // and updates rewrite the entry rather than add one), so they do not contribute to the
+ // distinct-record-count estimate that sizes the file groups.
+ List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs = latestMergedPartitionFileSliceList.stream()
Review Comment:
let's ensure we track this folllowup because Flink MOR streaming writes inserts directly to log files right. I am also thinking to support non-blocking clustering, we can perhaps write inserts to log files (however that's TBD). Flink use case is more relevant. Also, uncompacted file slice with only log file silently contribute 0. So, RLI is inconsistent for some corner cases. We will have to fix it soon.
cc: @danny0405
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -826,28 +842,58 @@ private Pair<Integer, HoodieData<HoodieRecord>> initializeRecordIndexPartition(
dataMetaClient,
dataWriteConfig);
- // Initialize the file groups
- final int fileGroupCount = estimateFileGroupCount(records);
+ Pair<Integer, Integer> bounds = getRLIFileGroupCountBounds();
+ int minFileGroupCount = bounds.getLeft();
+ int maxFileGroupCount = bounds.getRight();
+
+ int fileGroupCount;
+ if (minFileGroupCount != maxFileGroupCount) {
+ // Estimate file group count based on record count read from base file footer metadata.
+ // Avoids the expensive records.persist() + records.count() pass over materialized records.
+ // Filtering to base-file-only slices is safe for RLI: log files in a slice can only carry
+ // updates or deletes for existing records, never new inserts (RLI is keyed by record key
+ // and updates rewrite the entry rather than add one), so they do not contribute to the
+ // distinct-record-count estimate that sizes the file groups.
+ List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs = latestMergedPartitionFileSliceList.stream()
Review Comment:
As a side note, are we going to deprecate avro log files in favor columnar/parquet completely? That might simplify the above problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524255813
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `85.71429%` with `5 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 47.49%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 85.71% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (e299b84) and HEAD (dfe8b21). Click for more details.
>
> <details><summary>HEAD has 21 uploads less than BASE</summary>
>
>| Flag | BASE (e299b84) | HEAD (dfe8b21) |
>|------|------|------|
>|spark-java-tests|15|3|
>|spark-scala-tests|10|3|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
=============================================
- Coverage 68.92% 47.49% -21.44%
+ Complexity 29076 16645 -12431
=============================================
Files 2509 2040 -469
Lines 139470 115607 -23863
Branches 17117 14863 -2254
=============================================
- Hits 96130 54903 -41227
- Misses 35584 54875 +19291
+ Partials 7756 5829 -1927
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.16% <85.71%> (-17.19%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.97% <45.71%> (-12.31%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `67.60% <85.71%> (-16.20%)` | :arrow_down: |
... and [1569 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524302408
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `91.42857%` with `3 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 68.92%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 91.42% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
============================================
- Coverage 68.92% 68.92% -0.01%
- Complexity 29076 29100 +24
============================================
Files 2509 2509
Lines 139470 139495 +25
Branches 17117 17115 -2
============================================
+ Hits 96130 96140 +10
- Misses 35584 35603 +19
+ Partials 7756 7752 -4
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.42% <45.71%> (-0.02%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.29% <91.42%> (-0.06%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.28% <85.71%> (+<0.01%)` | :arrow_up: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.48% <85.71%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `83.88% <91.42%> (+0.09%)` | :arrow_up: |
... and [16 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524288709
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `91.42857%` with `3 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 68.71%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 91.42% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
============================================
- Coverage 68.92% 68.71% -0.22%
+ Complexity 29076 28982 -94
============================================
Files 2509 2509
Lines 139470 139495 +25
Branches 17117 17115 -2
============================================
- Hits 96130 95852 -278
- Misses 35584 35917 +333
+ Partials 7756 7726 -30
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.42% <45.71%> (-0.02%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.48% <91.42%> (-0.87%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.19% <85.71%> (-0.08%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.48% <85.71%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `83.88% <91.42%> (+0.09%)` | :arrow_up: |
... and [55 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524282192
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `85.71429%` with `5 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 67.81%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 85.71% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
============================================
- Coverage 68.92% 67.81% -1.12%
+ Complexity 29076 28601 -475
============================================
Files 2509 2509
Lines 139470 139495 +25
Branches 17117 17115 -2
============================================
- Hits 96130 94592 -1538
- Misses 35584 37119 +1535
- Partials 7756 7784 +28
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.42% <45.71%> (-0.02%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.08% <85.71%> (-4.27%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.92% <85.71%> (-0.36%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.48% <85.71%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `75.83% <85.71%> (-7.97%)` | :arrow_down: |
... and [217 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4524306475
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `91.42857%` with `3 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 68.92%. Comparing base ([`e299b84`](https://app.codecov.io/gh/apache/hudi/commit/e299b84b10b5ccd4ee5c75e541f0109a85549d7a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfe8b21`](https://app.codecov.io/gh/apache/hudi/commit/dfe8b2148953a1079b68bbe0854ecd405657c3c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 91.42% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18826 +/- ##
============================================
- Coverage 68.92% 68.92% -0.01%
- Complexity 29076 29100 +24
============================================
Files 2509 2509
Lines 139470 139495 +25
Branches 17117 17115 -2
============================================
+ Hits 96130 96143 +13
- Misses 35584 35601 +17
+ Partials 7756 7751 -5
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.42% <45.71%> (-0.02%)` | :arrow_down: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.90% <20.00%> (-0.02%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.20% <20.00%> (-0.04%)` | :arrow_down: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.32% <91.42%> (-0.03%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.28% <85.71%> (+<0.01%)` | :arrow_up: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/18826/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.48% <85.71%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/18826?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/18826?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `83.88% <91.42%> (+0.09%)` | :arrow_up: |
... and [15 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18826/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #18826:
URL: https://github.com/apache/hudi/pull/18826#discussion_r3292766076
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -824,28 +824,54 @@ private Pair<Integer, HoodieData<HoodieRecord>> initializeRecordIndexPartition(
dataMetaClient,
dataWriteConfig);
- // Initialize the file groups
- final int fileGroupCount = estimateFileGroupCount(records);
+ Pair<Integer, Integer> bounds = getRLIFileGroupCountBounds();
Review Comment:
How about persist records once in `initializeRecordIndexPartition` before returning the pair, or at minimum guard on `isRecordIndexInitializationValidationEnabled()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #18826:
URL: https://github.com/apache/hudi/pull/18826#discussion_r3479804379
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -857,6 +883,32 @@ private int estimateFileGroupCount(HoodieData<HoodieRecord> records) {
);
}
+ /**
+ * Sums row counts read from each base file's footer metadata, in parallel via the engine context.
+ * Used in place of materializing and counting an RDD of records during RLI bootstrap.
+ */
+ private long estimateRecordCountFromBaseFiles(List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs) {
+ if (partitionBaseFilePairs.isEmpty()) {
+ return 0L;
+ }
+ int parallelism = Math.min(partitionBaseFilePairs.size(),
+ dataWriteConfig.getMetadataConfig().getRecordIndexMaxParallelism());
+ StorageConfiguration<?> storageConfBroadcast = storageConf;
+ return engineContext.parallelize(partitionBaseFilePairs, parallelism)
+ .map(partitionAndBaseFile -> {
+ HoodieBaseFile baseFile = partitionAndBaseFile.getValue();
+ StoragePath path = baseFile.getStoragePath();
+ try {
+ HoodieStorage storage = HoodieStorageUtils.getStorage(path, storageConfBroadcast);
+ return HoodieIOFactory.getIOFactory(storage).getFileFormatUtils(path).getRowCount(storage, path);
+ } catch (Exception e) {
+ LOG.warn("Failed to read row count from base file footer: {}", path, e);
+ return 0L;
Review Comment:
Addressed in bec52e4dc96c: removed the catch-all. `getRowCount` is unchecked so `UnsupportedOperationException` (from `HFileUtils.getRowCount`) and `HoodieIOException` (from format-specific readers) now propagate, matching `countRecordsInHFiles`. An HFile-formatted base file will fail fast rather than silently estimate 0; an HFile-friendly estimator path can be a follow-up if needed.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -824,28 +824,54 @@ private Pair<Integer, HoodieData<HoodieRecord>> initializeRecordIndexPartition(
dataMetaClient,
dataWriteConfig);
- // Initialize the file groups
- final int fileGroupCount = estimateFileGroupCount(records);
+ Pair<Integer, Integer> bounds = getRLIFileGroupCountBounds();
Review Comment:
Addressed in bec52e4dc96c: persist is now declared in the caller (`initializeFilegroupsAndCommitToRecordIndexPartition`) only when `isRecordIndexInitializationValidationEnabled()` is true, and *before* the bulk-insert in both partitioned and non-partitioned branches — so bulkCommit's materialization populates the cache and `validateRecordIndex#count()` reads back from it. The matching `unpersist()` lives in the same validation-only branch. Happy path is deliberately left unpersisted (the RDD is consumed exactly once by bulk-insert), which is the original optimization.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -824,28 +824,54 @@ private Pair<Integer, HoodieData<HoodieRecord>> initializeRecordIndexPartition(
dataMetaClient,
dataWriteConfig);
- // Initialize the file groups
- final int fileGroupCount = estimateFileGroupCount(records);
+ Pair<Integer, Integer> bounds = getRLIFileGroupCountBounds();
+ int minFileGroupCount = bounds.getLeft();
+ int maxFileGroupCount = bounds.getRight();
+
+ int fileGroupCount;
+ if (minFileGroupCount != maxFileGroupCount) {
+ // Estimate file group count based on record count read from base file footer metadata.
+ // Avoids the expensive records.persist() + records.count() pass over materialized records.
+ List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs = latestMergedPartitionFileSliceList.stream()
Review Comment:
Addressed in bec52e4dc96c with a code comment: RLI is keyed by record key, and log files in a slice can only carry updates or deletes for existing records (RLI updates rewrite the entry rather than add a new one). Log files never contribute new distinct record-key cardinality, so filtering to base-file-only slices for the row-count estimate cannot undercount the file-group sizing. No follow-up needed unless we revisit the model that log files only carry updates/deletes.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -857,6 +883,32 @@ private int estimateFileGroupCount(HoodieData<HoodieRecord> records) {
);
}
+ /**
+ * Sums row counts read from each base file's footer metadata, in parallel via the engine context.
+ * Used in place of materializing and counting an RDD of records during RLI bootstrap.
+ */
+ private long estimateRecordCountFromBaseFiles(List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs) {
+ if (partitionBaseFilePairs.isEmpty()) {
+ return 0L;
+ }
+ int parallelism = Math.min(partitionBaseFilePairs.size(),
+ dataWriteConfig.getMetadataConfig().getRecordIndexMaxParallelism());
+ StorageConfiguration<?> storageConfBroadcast = storageConf;
Review Comment:
As wombatu-kun noted, this name is the existing pattern at `countRecordsInHFiles` (HoodieBackedTableMetadataWriter.java:908) which this method mirrors. Renaming only the new copy diverges from the established convention; happy to do a separate rename of both call sites as a minor cleanup PR if you prefer.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -857,6 +883,32 @@ private int estimateFileGroupCount(HoodieData<HoodieRecord> records) {
);
}
+ /**
+ * Sums row counts read from each base file's footer metadata, in parallel via the engine context.
+ * Used in place of materializing and counting an RDD of records during RLI bootstrap.
+ */
+ private long estimateRecordCountFromBaseFiles(List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs) {
+ if (partitionBaseFilePairs.isEmpty()) {
+ return 0L;
+ }
+ int parallelism = Math.min(partitionBaseFilePairs.size(),
+ dataWriteConfig.getMetadataConfig().getRecordIndexMaxParallelism());
+ StorageConfiguration<?> storageConfBroadcast = storageConf;
+ return engineContext.parallelize(partitionBaseFilePairs, parallelism)
+ .map(partitionAndBaseFile -> {
+ HoodieBaseFile baseFile = partitionAndBaseFile.getValue();
Review Comment:
Addressed in 4c398477 (now part of bec52e4dc96c after rebase) — switched to `getRight()` for uniformity.
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestHoodieBackedMetadata.java:
##########
@@ -4379,6 +4379,104 @@ private void changeTableVersion(HoodieTableVersion version) throws IOException {
}
}
+ /**
+ * Validates that RLI initialization estimates file group count from base file footer metadata
+ * (instead of materializing and counting records) when min != max file group count.
+ */
+ @ParameterizedTest
+ @EnumSource(HoodieTableType.class)
+ public void testRecordIndexFileGroupEstimation(HoodieTableType tableType) throws Exception {
Review Comment:
Good catch. Fixed in bec52e4dc96c: pinned `growthFactor=2.0` and `maxFileGroupSizeBytes=2400` in `testRecordIndexFileGroupEstimation`. The math now lands at 8 file groups (200 records × 2.0 / (2400/48 = 50) = 8, between min=1 and max=10) and the assertion is bumped to `fileGroupCount > 1` — if the estimator returns 0 or its output is ignored, the count clamps to minFileGroupCount=1 and the test fails. `testRecordIndexWithFixedFileGroupCount` intentionally exercises the bypass branch (the Javadoc already calls that out) so it's left as-is.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -824,28 +824,54 @@ private Pair<Integer, HoodieData<HoodieRecord>> initializeRecordIndexPartition(
dataMetaClient,
dataWriteConfig);
- // Initialize the file groups
- final int fileGroupCount = estimateFileGroupCount(records);
+ Pair<Integer, Integer> bounds = getRLIFileGroupCountBounds();
+ int minFileGroupCount = bounds.getLeft();
+ int maxFileGroupCount = bounds.getRight();
+
+ int fileGroupCount;
+ if (minFileGroupCount != maxFileGroupCount) {
+ // Estimate file group count based on record count read from base file footer metadata.
+ // Avoids the expensive records.persist() + records.count() pass over materialized records.
+ List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs = latestMergedPartitionFileSliceList.stream()
+ .filter(p -> p.getRight().getBaseFile().isPresent())
+ .map(p -> Pair.of(p.getLeft(), p.getRight().getBaseFile().get()))
+ .collect(Collectors.toList());
+ fileGroupCount = estimateFileGroupCountFromBaseFiles(
+ partitionBaseFilePairs, minFileGroupCount, maxFileGroupCount);
+ LOG.info("Estimated {} file groups from base file footer metadata", fileGroupCount);
+ } else {
+ // min == max: skip estimation, use the fixed value directly.
+ fileGroupCount = minFileGroupCount;
+ LOG.info("Using user-configured file group count: {}", fileGroupCount);
+ }
+
LOG.info("Initializing record index with {} file groups.", fileGroupCount);
return Pair.of(fileGroupCount, records);
}
- private int estimateFileGroupCount(HoodieData<HoodieRecord> records) {
- int minFileGroupCount;
- int maxFileGroupCount;
+ /**
+ * Returns the (min, max) file group count bounds for RLI based on which RLI variant is enabled.
+ */
+ private Pair<Integer, Integer> getRLIFileGroupCountBounds() {
if (dataWriteConfig.isRecordLevelIndexEnabled()) {
- minFileGroupCount = dataWriteConfig.getRecordLevelIndexMinFileGroupCount();
- maxFileGroupCount = dataWriteConfig.getRecordLevelIndexMaxFileGroupCount();
- } else {
- minFileGroupCount = dataWriteConfig.getGlobalRecordLevelIndexMinFileGroupCount();
- maxFileGroupCount = dataWriteConfig.getGlobalRecordLevelIndexMaxFileGroupCount();
- }
- Supplier<Long> recordCountSupplier = () -> {
- records.persist("MEMORY_AND_DISK_SER");
- long count = records.count();
- LOG.info("Initializing record index with {} mappings", count);
- return count;
- };
+ return Pair.of(dataWriteConfig.getRecordLevelIndexMinFileGroupCount(),
Review Comment:
Good question. Yes — addressed in bec52e4dc96c. The persist now happens in the caller before `bulkCommit` (only when validation is enabled), so `validateRecordIndex#count()` reads from the cached RDD instead of re-deriving keys. See the reply on the related thread above for the full plumbing.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -859,6 +885,32 @@ private int estimateFileGroupCount(HoodieData<HoodieRecord> records) {
);
}
+ /**
+ * Sums row counts read from each base file's footer metadata, in parallel via the engine context.
+ * Used in place of materializing and counting an RDD of records during RLI bootstrap.
+ */
+ private long estimateRecordCountFromBaseFiles(List<Pair<String, HoodieBaseFile>> partitionBaseFilePairs) {
+ if (partitionBaseFilePairs.isEmpty()) {
+ return 0L;
+ }
+ int parallelism = Math.min(partitionBaseFilePairs.size(),
+ dataWriteConfig.getMetadataConfig().getRecordIndexMaxParallelism());
+ StorageConfiguration<?> storageConfBroadcast = storageConf;
Review Comment:
Same response as the earlier thread on this name — mirroring `countRecordsInHFiles`'s convention. Open to a rename across both call sites in a separate cleanup PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): optimize RLI bootstrap by sizing file groups from base file footer row counts [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18826:
URL: https://github.com/apache/hudi/pull/18826#issuecomment-4807291583
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dfe8b2148953a1079b68bbe0854ecd405657c3c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14242",
"triggerID" : "dfe8b2148953a1079b68bbe0854ecd405657c3c3",
"triggerType" : "PUSH"
}, {
"hash" : "4c398477979d2d99f9b4ed61ed1cfbae8631ea1f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14806",
"triggerID" : "4c398477979d2d99f9b4ed61ed1cfbae8631ea1f",
"triggerType" : "PUSH"
}, {
"hash" : "bec52e4dc96c4a01c7fc752144bc535fd6720941",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bec52e4dc96c4a01c7fc752144bc535fd6720941",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4c398477979d2d99f9b4ed61ed1cfbae8631ea1f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14806)
* bec52e4dc96c4a01c7fc752144bc535fd6720941 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org