You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "prashantwason (via GitHub)" <gi...@apache.org> on 2023/04/19 05:16:37 UTC
[GitHub] [hudi] prashantwason opened a new pull request, #8496: [HUDI-6101] Use UnBoundedCompactionStrategy for MDT compactions.
prashantwason opened a new pull request, #8496:
URL: https://github.com/apache/hudi/pull/8496
[HUDI-6101] Use UnBoundedCompactionStrategy for MDT compactions.
### Change Logs
Changed the MDT compaction strategy to UnBoundedCompactionStrategy.
Compaction on metadata table is used as a barrier for archiving on main dataset and for validating the deltacommits having corresponding completed commits. Therefore, we need to compact all fileslices of all partitions together requiring UnBoundedCompactionStrategy.
### Impact
Corrects compaction on MDT when large amount of log data is to be compacted. Without this change, the current default strategy (LogFileSizeBasedCompactionStrategy) may only compact some file groups.
### Risk level (write none, low medium or high below)
None
### Documentation Update
None
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8496: [HUDI-6101] Use UnBoundedCompactionStrategy for MDT compactions.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8496:
URL: https://github.com/apache/hudi/pull/8496#discussion_r1172356870
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -292,6 +292,10 @@ private HoodieWriteConfig createMetadataWriteConfig(
.withInlineCompaction(false)
.withMaxNumDeltaCommitsBeforeCompaction(writeConfig.getMetadataCompactDeltaCommitMax())
.withEnableOptimizedLogBlocksScan(String.valueOf(writeConfig.enableOptimizedLogBlocksScan()))
+ // Compaction on metadata table is used as a barrier for archiving on main dataset and for validating the
+ // deltacommits having corresponding completed commits. Therefore, we need to compact all fileslices of all
+ // partitions together requiring UnBoundedCompactionStrategy.
+ .withCompactionStrategy(new UnBoundedCompactionStrategy())
Review Comment:
A write config test, just in case these options are mis-changed during code evolving. Never mind, we can address that in separate PR, can you fire a PR to add a test for MDT write config validation for all these changes you raised recently?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8496: [HUDI-6101] Use UnBoundedCompactionStrategy for MDT compactions.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8496:
URL: https://github.com/apache/hudi/pull/8496#discussion_r1172185710
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -292,6 +292,10 @@ private HoodieWriteConfig createMetadataWriteConfig(
.withInlineCompaction(false)
.withMaxNumDeltaCommitsBeforeCompaction(writeConfig.getMetadataCompactDeltaCommitMax())
.withEnableOptimizedLogBlocksScan(String.valueOf(writeConfig.enableOptimizedLogBlocksScan()))
+ // Compaction on metadata table is used as a barrier for archiving on main dataset and for validating the
+ // deltacommits having corresponding completed commits. Therefore, we need to compact all fileslices of all
+ // partitions together requiring UnBoundedCompactionStrategy.
+ .withCompactionStrategy(new UnBoundedCompactionStrategy())
Review Comment:
500GB may be enough for most indexes we have today. But record index is usually in TBs and hence this issue was seen there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8496: [HUDI-6101] Use UnBoundedCompactionStrategy for MDT compactions.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8496:
URL: https://github.com/apache/hudi/pull/8496#issuecomment-1514179507
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5880aea4f85677cb1230ba8c93fb90b60dd1c769",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16453",
"triggerID" : "5880aea4f85677cb1230ba8c93fb90b60dd1c769",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5880aea4f85677cb1230ba8c93fb90b60dd1c769 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16453)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8496: [HUDI-6101] Use UnBoundedCompactionStrategy for MDT compactions.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8496:
URL: https://github.com/apache/hudi/pull/8496#issuecomment-1514790114
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5880aea4f85677cb1230ba8c93fb90b60dd1c769",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16453",
"triggerID" : "5880aea4f85677cb1230ba8c93fb90b60dd1c769",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5880aea4f85677cb1230ba8c93fb90b60dd1c769 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16453)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8496: [HUDI-6101] Use UnBoundedCompactionStrategy for MDT compactions.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8496:
URL: https://github.com/apache/hudi/pull/8496#discussion_r1170851945
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -292,6 +292,10 @@ private HoodieWriteConfig createMetadataWriteConfig(
.withInlineCompaction(false)
.withMaxNumDeltaCommitsBeforeCompaction(writeConfig.getMetadataCompactDeltaCommitMax())
.withEnableOptimizedLogBlocksScan(String.valueOf(writeConfig.enableOptimizedLogBlocksScan()))
+ // Compaction on metadata table is used as a barrier for archiving on main dataset and for validating the
+ // deltacommits having corresponding completed commits. Therefore, we need to compact all fileslices of all
+ // partitions together requiring UnBoundedCompactionStrategy.
+ .withCompactionStrategy(new UnBoundedCompactionStrategy())
Review Comment:
Okay, by default we have a 500GB target IO upper bound. Can we add a UT for the write config test of MDT then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #8496: [HUDI-6101] Use UnBoundedCompactionStrategy for MDT compactions.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #8496:
URL: https://github.com/apache/hudi/pull/8496
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8496: [HUDI-6101] Use UnBoundedCompactionStrategy for MDT compactions.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8496:
URL: https://github.com/apache/hudi/pull/8496#issuecomment-1514172695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5880aea4f85677cb1230ba8c93fb90b60dd1c769",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5880aea4f85677cb1230ba8c93fb90b60dd1c769",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5880aea4f85677cb1230ba8c93fb90b60dd1c769 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8496: [HUDI-6101] Use UnBoundedCompactionStrategy for MDT compactions.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8496:
URL: https://github.com/apache/hudi/pull/8496#discussion_r1172186243
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -292,6 +292,10 @@ private HoodieWriteConfig createMetadataWriteConfig(
.withInlineCompaction(false)
.withMaxNumDeltaCommitsBeforeCompaction(writeConfig.getMetadataCompactDeltaCommitMax())
.withEnableOptimizedLogBlocksScan(String.valueOf(writeConfig.enableOptimizedLogBlocksScan()))
+ // Compaction on metadata table is used as a barrier for archiving on main dataset and for validating the
+ // deltacommits having corresponding completed commits. Therefore, we need to compact all fileslices of all
+ // partitions together requiring UnBoundedCompactionStrategy.
+ .withCompactionStrategy(new UnBoundedCompactionStrategy())
Review Comment:
What kind of UT are you suggesting?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org