You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2025/10/01 19:03:37 UTC
[PR] fix: Fixing condition to calculate enforceCoalesceWithRepartition for table services with streaming writes to mdt [hudi]
nsivabalan opened a new pull request, #14026:
URL: https://github.com/apache/hudi/pull/14026
### Describe the issue this Pull Request addresses
With landed a patch to fix performance with streaming writes to metadata here https://github.com/apache/hudi/pull/13976/. We had a minor bug in calculating right value for `enforceCoalesceWithRepartition` for table services (specifically clustering). Fixing the condition check in this patch.
### Summary and Changelog
Fixing computing right value for `enforceCoalesceWithRepartition` for clustering table service with streaming writes to metadata table.
### Impact
Only set `enforceCoalesceWithRepartition` to true when sort columns is not set and if sort mode is NONE for clustering.
### Risk Level
low
### Documentation Update
<!-- Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
- The config description must be updated if new configs are added or the default value of the configs are changed.
- Any new feature or user-facing change requires updating the Hudi website. Please follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website. -->
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fixing condition to calculate enforceCoalesceWithRepartition for table services with streaming writes to mdt [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14026:
URL: https://github.com/apache/hudi/pull/14026#issuecomment-3357746859
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea75374ae74d108b31da873699767e784b7c00b6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea75374ae74d108b31da873699767e784b7c00b6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea75374ae74d108b31da873699767e784b7c00b6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fixing condition to calculate enforceCoalesceWithRepartition for table services with streaming writes to mdt [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14026:
URL: https://github.com/apache/hudi/pull/14026#issuecomment-3358416605
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea75374ae74d108b31da873699767e784b7c00b6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8999",
"triggerID" : "ea75374ae74d108b31da873699767e784b7c00b6",
"triggerType" : "PUSH"
}, {
"hash" : "89624b1ea853a4542d3611ad80507bf3d03d334b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89624b1ea853a4542d3611ad80507bf3d03d334b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea75374ae74d108b31da873699767e784b7c00b6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8999)
* 89624b1ea853a4542d3611ad80507bf3d03d334b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fixing condition to calculate enforceCoalesceWithRepartition for table services with streaming writes to mdt [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14026:
URL: https://github.com/apache/hudi/pull/14026#issuecomment-3358493231
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea75374ae74d108b31da873699767e784b7c00b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8999",
"triggerID" : "ea75374ae74d108b31da873699767e784b7c00b6",
"triggerType" : "PUSH"
}, {
"hash" : "89624b1ea853a4542d3611ad80507bf3d03d334b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9002",
"triggerID" : "89624b1ea853a4542d3611ad80507bf3d03d334b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89624b1ea853a4542d3611ad80507bf3d03d334b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9002)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fixing condition to calculate enforceCoalesceWithRepartition for table services with streaming writes to mdt [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #14026:
URL: https://github.com/apache/hudi/pull/14026#discussion_r2395650986
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDTableServiceClient.java:
##########
@@ -101,14 +101,15 @@ protected HoodieWriteMetadata<HoodieData<WriteStatus>> partialUpdateTableMetadat
String instantTime,
WriteOperationType writeOperationType) {
if (isStreamingWriteToMetadataEnabled(table)) {
- boolean enforceCoalesceWithRepartition = writeOperationType == WriteOperationType.CLUSTER && config.getBulkInsertSortMode() == BulkInsertSortMode.NONE;
+ boolean enforceCoalesceWithRepartition = writeOperationType == WriteOperationType.CLUSTER; // for other table services, enforceCoalesceWithRepartition will be false.
if (enforceCoalesceWithRepartition) {
- // check clustering plan for sort columns. only if there are no sort columns, then we might still set enforceCoalesceWithRepartition to true.
+ // check clustering plan for sort columns. only if there are no sort columns, then fallback to sort mode from write config.
HoodieClusteringPlan clusteringPlan = ClusteringUtils.getClusteringPlan(
table.getMetaClient(), ClusteringUtils.getRequestedClusteringInstant(instantTime, table.getActiveTimeline(), table.getInstantGenerator()).get())
.map(Pair::getRight).orElseThrow(() -> new HoodieClusteringException(
"Unable to read clustering plan for instant: " + instantTime));
- enforceCoalesceWithRepartition = !clusteringPlan.getStrategy().getStrategyParams().containsKey(PLAN_STRATEGY_SORT_COLUMNS.key());
+ enforceCoalesceWithRepartition = (!clusteringPlan.getStrategy().getStrategyParams().containsKey(PLAN_STRATEGY_SORT_COLUMNS.key())
Review Comment:
Let's check if the value is non-null or empty instead of just checking the key is not present
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fixing condition to calculate enforceCoalesceWithRepartition for table services with streaming writes to mdt [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14026:
URL: https://github.com/apache/hudi/pull/14026#issuecomment-3357752957
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea75374ae74d108b31da873699767e784b7c00b6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8999",
"triggerID" : "ea75374ae74d108b31da873699767e784b7c00b6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea75374ae74d108b31da873699767e784b7c00b6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8999)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fixing condition to calculate enforceCoalesceWithRepartition for table services with streaming writes to mdt [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14026:
URL: https://github.com/apache/hudi/pull/14026#issuecomment-3358855528
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89624b1ea853a4542d3611ad80507bf3d03d334b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9002",
"triggerID" : "89624b1ea853a4542d3611ad80507bf3d03d334b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89624b1ea853a4542d3611ad80507bf3d03d334b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9002)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fixing condition to calculate enforceCoalesceWithRepartition for table services with streaming writes to mdt [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14026:
URL: https://github.com/apache/hudi/pull/14026#issuecomment-3358694600
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89624b1ea853a4542d3611ad80507bf3d03d334b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9002",
"triggerID" : "89624b1ea853a4542d3611ad80507bf3d03d334b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89624b1ea853a4542d3611ad80507bf3d03d334b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9002)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fixing condition to calculate enforceCoalesceWithRepartition for table services with streaming writes to mdt [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14026:
URL: https://github.com/apache/hudi/pull/14026#issuecomment-3358420218
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea75374ae74d108b31da873699767e784b7c00b6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8999",
"triggerID" : "ea75374ae74d108b31da873699767e784b7c00b6",
"triggerType" : "PUSH"
}, {
"hash" : "89624b1ea853a4542d3611ad80507bf3d03d334b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9002",
"triggerID" : "89624b1ea853a4542d3611ad80507bf3d03d334b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea75374ae74d108b31da873699767e784b7c00b6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8999)
* 89624b1ea853a4542d3611ad80507bf3d03d334b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9002)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fixing condition to calculate enforceCoalesceWithRepartition for table services with streaming writes to mdt [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14026:
URL: https://github.com/apache/hudi/pull/14026#issuecomment-3358692452
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "89624b1ea853a4542d3611ad80507bf3d03d334b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89624b1ea853a4542d3611ad80507bf3d03d334b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89624b1ea853a4542d3611ad80507bf3d03d334b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fixing condition to calculate enforceCoalesceWithRepartition for table services with streaming writes to mdt [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14026:
URL: https://github.com/apache/hudi/pull/14026#issuecomment-3357999200
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea75374ae74d108b31da873699767e784b7c00b6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8999",
"triggerID" : "ea75374ae74d108b31da873699767e784b7c00b6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea75374ae74d108b31da873699767e784b7c00b6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8999)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fixing condition to calculate enforceCoalesceWithRepartition for table services with streaming writes to mdt [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #14026:
URL: https://github.com/apache/hudi/pull/14026#discussion_r2396108331
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDTableServiceClient.java:
##########
@@ -101,14 +101,15 @@ protected HoodieWriteMetadata<HoodieData<WriteStatus>> partialUpdateTableMetadat
String instantTime,
WriteOperationType writeOperationType) {
if (isStreamingWriteToMetadataEnabled(table)) {
- boolean enforceCoalesceWithRepartition = writeOperationType == WriteOperationType.CLUSTER && config.getBulkInsertSortMode() == BulkInsertSortMode.NONE;
+ boolean enforceCoalesceWithRepartition = writeOperationType == WriteOperationType.CLUSTER; // for other table services, enforceCoalesceWithRepartition will be false.
if (enforceCoalesceWithRepartition) {
- // check clustering plan for sort columns. only if there are no sort columns, then we might still set enforceCoalesceWithRepartition to true.
+ // check clustering plan for sort columns. only if there are no sort columns, then fallback to sort mode from write config.
HoodieClusteringPlan clusteringPlan = ClusteringUtils.getClusteringPlan(
table.getMetaClient(), ClusteringUtils.getRequestedClusteringInstant(instantTime, table.getActiveTimeline(), table.getInstantGenerator()).get())
.map(Pair::getRight).orElseThrow(() -> new HoodieClusteringException(
"Unable to read clustering plan for instant: " + instantTime));
- enforceCoalesceWithRepartition = !clusteringPlan.getStrategy().getStrategyParams().containsKey(PLAN_STRATEGY_SORT_COLUMNS.key());
+ enforceCoalesceWithRepartition = (!clusteringPlan.getStrategy().getStrategyParams().containsKey(PLAN_STRATEGY_SORT_COLUMNS.key())
Review Comment:
sure. pushed a commit to address this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fixing condition to calculate enforceCoalesceWithRepartition for table services with streaming writes to mdt [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #14026:
URL: https://github.com/apache/hudi/pull/14026
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org