You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "lokeshj1703 (via GitHub)" <gi...@apache.org> on 2025/10/09 11:25:04 UTC
[PR] fix: Persist RLI index bootstrap records only if estimation is required and add unpersist [hudi]
lokeshj1703 opened a new pull request, #14069:
URL: https://github.com/apache/hudi/pull/14069
### Describe the issue this Pull Request addresses
Closes #14068 https://github.com/apache/hudi/issues/14068
### Summary and Changelog
If the min and max file group count for record index is configured and set to the same value, record index records should will not be persisted. We would use the configured value as the file group count instead of estimating it in such a case.
### Impact
PR adds an optimisation where record index records will not be persisted if file group count is configured by the user. Also the record index records were not unpersisted after bootstrap. The PR adds unpersist after bootstrap of record index partition.
### Risk Level
Low
### Documentation Update
NA
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Persist RLI index bootstrap records only if estimation is required and add unpersist [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14069:
URL: https://github.com/apache/hudi/pull/14069#issuecomment-3385413033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba3eb74cce8719c9dd70497d65e14b7ca72819c1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ba3eb74cce8719c9dd70497d65e14b7ca72819c1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ba3eb74cce8719c9dd70497d65e14b7ca72819c1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Persist RLI index bootstrap records only if estimation is required and add unpersist [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #14069:
URL: https://github.com/apache/hudi/pull/14069#discussion_r2438798360
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -823,9 +825,15 @@ private int estimateFileGroupCount(long recordCount) {
minFileGroupCount = dataWriteConfig.getRecordIndexMinFileGroupCount();
maxFileGroupCount = dataWriteConfig.getRecordIndexMaxFileGroupCount();
}
+ Supplier<Long> recordCountSupplier = () -> {
+ records.persist("MEMORY_AND_DISK_SER");
+ long count = records.count();
+ LOG.info("Initializing record index with {} mappings", count);
+ return count;
Review Comment:
Created https://github.com/apache/hudi/issues/14111
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Persist RLI index bootstrap records only if estimation is required and add unpersist [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #14069:
URL: https://github.com/apache/hudi/pull/14069#issuecomment-3416640385
<img width="1177" height="583" alt="image" src="https://github.com/user-attachments/assets/67f2acbd-1eb0-4c90-abf9-c11a980fc322" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Persist RLI index bootstrap records only if estimation is required and add unpersist [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14069:
URL: https://github.com/apache/hudi/pull/14069#issuecomment-3385548033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba3eb74cce8719c9dd70497d65e14b7ca72819c1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9116",
"triggerID" : "ba3eb74cce8719c9dd70497d65e14b7ca72819c1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ba3eb74cce8719c9dd70497d65e14b7ca72819c1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9116)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Persist RLI index bootstrap records only if estimation is required and add unpersist [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #14069:
URL: https://github.com/apache/hudi/pull/14069#discussion_r2437060282
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -823,9 +825,15 @@ private int estimateFileGroupCount(long recordCount) {
minFileGroupCount = dataWriteConfig.getRecordIndexMinFileGroupCount();
maxFileGroupCount = dataWriteConfig.getRecordIndexMaxFileGroupCount();
}
+ Supplier<Long> recordCountSupplier = () -> {
+ records.persist("MEMORY_AND_DISK_SER");
+ long count = records.count();
+ LOG.info("Initializing record index with {} mappings", count);
+ return count;
Review Comment:
can we file a follow up to further optimize this by sampling instead of getting the exact count.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Persist RLI index bootstrap records only if estimation is required and add unpersist [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on PR #14069:
URL: https://github.com/apache/hudi/pull/14069#issuecomment-3396647594
Verified locally that persist and unpersist is working as expected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Persist RLI index bootstrap records only if estimation is required and add unpersist [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #14069:
URL: https://github.com/apache/hudi/pull/14069
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Persist RLI index bootstrap records only if estimation is required and add unpersist [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14069:
URL: https://github.com/apache/hudi/pull/14069#issuecomment-3385417435
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba3eb74cce8719c9dd70497d65e14b7ca72819c1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9116",
"triggerID" : "ba3eb74cce8719c9dd70497d65e14b7ca72819c1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ba3eb74cce8719c9dd70497d65e14b7ca72819c1 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9116)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Persist RLI index bootstrap records only if estimation is required and add unpersist [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on PR #14069:
URL: https://github.com/apache/hudi/pull/14069#issuecomment-3414110794
Azure CI is green
![Uploading image.png…]()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org