You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2023/06/26 23:47:31 UTC
[GitHub] [hudi] nsivabalan opened a new pull request, #9057: [HUDI-6446] Fixing MDT commit time parsing and RLI instantiation with MDT
nsivabalan opened a new pull request, #9057:
URL: https://github.com/apache/hudi/pull/9057
### Change Logs
[HUDI-6446] Fixing MDT commit time parsing and RLI instantiation with MDT.
For a fresh table, when both FILES and RLI is enabled, we use default values for num file groups i.e 10 for RLI. and this also creates a log file and does not create a base file since there are no records to instantiate as such. So, we should defer the instantiation to later. either at the end of first commit or when the data table has atleast 1 completed commit.
For an already existing table, this is not an issue since if there are valid records, we will dynamically determine the number of file groups.
### Impact
Deferring instantiation of RLI for a fresh table to later when we have atleast 1 completed commit in DT.
### Risk level (write none, low medium or high below)
low.
### Documentation Update
N/A
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9057: [HUDI-6446] Fixing MDT commit time parsing and RLI instantiation with MDT
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9057:
URL: https://github.com/apache/hudi/pull/9057#issuecomment-1608518499
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "aea6f0bb6a55c8019f34cf9b328abef34f0a5f01",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18116",
"triggerID" : "aea6f0bb6a55c8019f34cf9b328abef34f0a5f01",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aea6f0bb6a55c8019f34cf9b328abef34f0a5f01 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18116)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed pull request #9057: [HUDI-6446][WIP] Fixing MDT commit time parsing and RLI instantiation with MDT
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed pull request #9057: [HUDI-6446][WIP] Fixing MDT commit time parsing and RLI instantiation with MDT
URL: https://github.com/apache/hudi/pull/9057
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9057: [HUDI-6446] Fixing MDT commit time parsing and RLI instantiation with MDT
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9057:
URL: https://github.com/apache/hudi/pull/9057#discussion_r1243067339
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -344,6 +344,13 @@ private boolean initializeFromFilesystem(String initializationTime, List<Metadat
if (!filesPartitionAvailable) {
partitionsToInit.remove(MetadataPartitionType.FILES);
partitionsToInit.add(0, MetadataPartitionType.FILES);
+ // By default we allocate 10 FG's in RLI. but for very large tables, these needs to be configured properly. For an existing table, we dynamically
+ // deduce the number of file groups for RLI. but for a fresh table since there are no records, we might choose default value of 10.
+ // So, deferring te instantiation of RLI to atleast 1 completed commit in DT.
+ if (partitionsToInit.contains(MetadataPartitionType.RECORD_INDEX) &&
+ dataMetaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().countInstants() == 0) {
+ partitionsToInit.remove(MetadataPartitionType.RECORD_INDEX);
+ }
Review Comment:
I'm wondering if the estimation of file groups cnt work well in production, it only takes the exsiting record count for file group estimation, even though there is a growth factor that can be configured by user explicitly, it is still somehow fixed (not dynamic), in production, most of the table records NDV are increasing consinuously, so the estimation may never work as expected, especially for fresh new table.
I would prefer the initial 10 for fresh new table, because very probabily the estimation result with the 1st commit is smaller than 10, but with time goes by, we need far more file groups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9057: [HUDI-6446] Fixing MDT commit time parsing and RLI instantiation with MDT
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9057:
URL: https://github.com/apache/hudi/pull/9057#issuecomment-1608500681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "aea6f0bb6a55c8019f34cf9b328abef34f0a5f01",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aea6f0bb6a55c8019f34cf9b328abef34f0a5f01",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aea6f0bb6a55c8019f34cf9b328abef34f0a5f01 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9057: [HUDI-6446] Fixing MDT commit time parsing and RLI instantiation with MDT
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9057:
URL: https://github.com/apache/hudi/pull/9057#issuecomment-1608689486
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "aea6f0bb6a55c8019f34cf9b328abef34f0a5f01",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18116",
"triggerID" : "aea6f0bb6a55c8019f34cf9b328abef34f0a5f01",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aea6f0bb6a55c8019f34cf9b328abef34f0a5f01 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18116)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org