You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "JoyJoyJo (via GitHub)" <gi...@apache.org> on 2025/02/03 06:54:05 UTC
[PR] [MINOR] Fix skew in clustering operator [hudi]
JoyJoyJo opened a new pull request, #12765:
URL: https://github.com/apache/hudi/pull/12765
### Change Logs
related to #11757
same skew in clustering
### Impact
none
### Risk level (write none, low medium or high below)
none
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fix skew in clustering operator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12765:
URL: https://github.com/apache/hudi/pull/12765#issuecomment-2638981476
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3344",
"triggerID" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"triggerType" : "PUSH"
}, {
"hash" : "249df0d03775ce2b398ee80d51023b95052e3116",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "249df0d03775ce2b398ee80d51023b95052e3116",
"triggerType" : "PUSH"
}, {
"hash" : "e974bc388e3c2aad6e2a9945c49e3cba5ec97aae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3347",
"triggerID" : "e974bc388e3c2aad6e2a9945c49e3cba5ec97aae",
"triggerType" : "PUSH"
}, {
"hash" : "e1596e13d426376aafb8eab07a7b4db6ed90b099",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3349",
"triggerID" : "e1596e13d426376aafb8eab07a7b4db6ed90b099",
"triggerType" : "PUSH"
}, {
"hash" : "8a84f357581feeda414477eec8fcdff16b63389a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8a84f357581feeda414477eec8fcdff16b63389a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 249df0d03775ce2b398ee80d51023b95052e3116 UNKNOWN
* e1596e13d426376aafb8eab07a7b4db6ed90b099 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3349)
* 8a84f357581feeda414477eec8fcdff16b63389a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fix skew in clustering operator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12765:
URL: https://github.com/apache/hudi/pull/12765#issuecomment-2630222401
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3344",
"triggerID" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"triggerType" : "PUSH"
}, {
"hash" : "249df0d03775ce2b398ee80d51023b95052e3116",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "249df0d03775ce2b398ee80d51023b95052e3116",
"triggerType" : "PUSH"
}, {
"hash" : "e974bc388e3c2aad6e2a9945c49e3cba5ec97aae",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3347",
"triggerID" : "e974bc388e3c2aad6e2a9945c49e3cba5ec97aae",
"triggerType" : "PUSH"
}, {
"hash" : "e1596e13d426376aafb8eab07a7b4db6ed90b099",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e1596e13d426376aafb8eab07a7b4db6ed90b099",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 249df0d03775ce2b398ee80d51023b95052e3116 UNKNOWN
* e974bc388e3c2aad6e2a9945c49e3cba5ec97aae Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3347)
* e1596e13d426376aafb8eab07a7b4db6ed90b099 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fix skew in clustering operator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12765:
URL: https://github.com/apache/hudi/pull/12765#issuecomment-2630119530
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3344",
"triggerID" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f2d59eb53a2455c0069f02549d45478a5c7c2d3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3344)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fix skew in clustering operator [hudi]
Posted by "JoyJoyJo (via GitHub)" <gi...@apache.org>.
JoyJoyJo commented on code in PR #12765:
URL: https://github.com/apache/hudi/pull/12765#discussion_r1944156823
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/clustering/ClusteringPlanOperator.java:
##########
@@ -139,11 +139,13 @@ private void scheduleClustering(HoodieFlinkTable<?> table, long checkpointId) {
ClusteringUtils.transitionClusteringOrReplaceRequestedToInflight(clusteringInstant, Option.empty(), table.getActiveTimeline());
table.getMetaClient().reloadActiveTimeline();
+ int operationIndex = 0;
for (HoodieClusteringGroup clusteringGroup : clusteringPlan.getInputGroups()) {
LOG.info("Execute clustering plan for instant {} as {} file slices", clusteringInstantTime, clusteringGroup.getSlices().size());
output.collect(new StreamRecord<>(
- new ClusteringPlanEvent(clusteringInstantTime, ClusteringGroupInfo.create(clusteringGroup), clusteringPlan.getStrategy().getStrategyParams())
+ new ClusteringPlanEvent(clusteringInstantTime, ClusteringGroupInfo.create(clusteringGroup), clusteringPlan.getStrategy().getStrategyParams(), operationIndex)
Review Comment:
```Java
// the first instant takes the highest priority.
Option<HoodieInstant> firstRequested = Option.fromJavaOptional(
pendingClusteringInstantTimes.stream()
.filter(instant -> instant.getState() == HoodieInstant.State.REQUESTED).findFirst());
```
The `scheduleClustering` method in `ClusteringPlanOperator` only handles the first plan each time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fix skew in clustering operator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12765:
URL: https://github.com/apache/hudi/pull/12765#issuecomment-2630332341
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3344",
"triggerID" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"triggerType" : "PUSH"
}, {
"hash" : "249df0d03775ce2b398ee80d51023b95052e3116",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "249df0d03775ce2b398ee80d51023b95052e3116",
"triggerType" : "PUSH"
}, {
"hash" : "e974bc388e3c2aad6e2a9945c49e3cba5ec97aae",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3347",
"triggerID" : "e974bc388e3c2aad6e2a9945c49e3cba5ec97aae",
"triggerType" : "PUSH"
}, {
"hash" : "e1596e13d426376aafb8eab07a7b4db6ed90b099",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3349",
"triggerID" : "e1596e13d426376aafb8eab07a7b4db6ed90b099",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 249df0d03775ce2b398ee80d51023b95052e3116 UNKNOWN
* e974bc388e3c2aad6e2a9945c49e3cba5ec97aae Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3347)
* e1596e13d426376aafb8eab07a7b4db6ed90b099 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3349)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fix skew in clustering operator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12765:
URL: https://github.com/apache/hudi/pull/12765#issuecomment-2630219518
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3344",
"triggerID" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"triggerType" : "PUSH"
}, {
"hash" : "249df0d03775ce2b398ee80d51023b95052e3116",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "249df0d03775ce2b398ee80d51023b95052e3116",
"triggerType" : "PUSH"
}, {
"hash" : "e974bc388e3c2aad6e2a9945c49e3cba5ec97aae",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3347",
"triggerID" : "e974bc388e3c2aad6e2a9945c49e3cba5ec97aae",
"triggerType" : "PUSH"
}, {
"hash" : "e1596e13d426376aafb8eab07a7b4db6ed90b099",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e1596e13d426376aafb8eab07a7b4db6ed90b099",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f2d59eb53a2455c0069f02549d45478a5c7c2d3 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3344)
* 249df0d03775ce2b398ee80d51023b95052e3116 UNKNOWN
* e974bc388e3c2aad6e2a9945c49e3cba5ec97aae Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3347)
* e1596e13d426376aafb8eab07a7b4db6ed90b099 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fix skew in clustering operator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12765:
URL: https://github.com/apache/hudi/pull/12765#issuecomment-2630211204
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3344",
"triggerID" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"triggerType" : "PUSH"
}, {
"hash" : "249df0d03775ce2b398ee80d51023b95052e3116",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "249df0d03775ce2b398ee80d51023b95052e3116",
"triggerType" : "PUSH"
}, {
"hash" : "e974bc388e3c2aad6e2a9945c49e3cba5ec97aae",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e974bc388e3c2aad6e2a9945c49e3cba5ec97aae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f2d59eb53a2455c0069f02549d45478a5c7c2d3 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3344)
* 249df0d03775ce2b398ee80d51023b95052e3116 UNKNOWN
* e974bc388e3c2aad6e2a9945c49e3cba5ec97aae UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fix skew in clustering operator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12765:
URL: https://github.com/apache/hudi/pull/12765#issuecomment-2630117074
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f2d59eb53a2455c0069f02549d45478a5c7c2d3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fix skew in clustering operator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12765:
URL: https://github.com/apache/hudi/pull/12765#issuecomment-2630213923
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3344",
"triggerID" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"triggerType" : "PUSH"
}, {
"hash" : "249df0d03775ce2b398ee80d51023b95052e3116",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "249df0d03775ce2b398ee80d51023b95052e3116",
"triggerType" : "PUSH"
}, {
"hash" : "e974bc388e3c2aad6e2a9945c49e3cba5ec97aae",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3347",
"triggerID" : "e974bc388e3c2aad6e2a9945c49e3cba5ec97aae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f2d59eb53a2455c0069f02549d45478a5c7c2d3 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3344)
* 249df0d03775ce2b398ee80d51023b95052e3116 UNKNOWN
* e974bc388e3c2aad6e2a9945c49e3cba5ec97aae Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3347)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fix skew in clustering operator [hudi]
Posted by "JoyJoyJo (via GitHub)" <gi...@apache.org>.
JoyJoyJo commented on code in PR #12765:
URL: https://github.com/apache/hudi/pull/12765#discussion_r1942267613
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/clustering/ClusteringPlanOperator.java:
##########
@@ -139,11 +139,13 @@ private void scheduleClustering(HoodieFlinkTable<?> table, long checkpointId) {
ClusteringUtils.transitionClusteringOrReplaceRequestedToInflight(clusteringInstant, Option.empty(), table.getActiveTimeline());
table.getMetaClient().reloadActiveTimeline();
+ int operationIndex = 0;
for (HoodieClusteringGroup clusteringGroup : clusteringPlan.getInputGroups()) {
LOG.info("Execute clustering plan for instant {} as {} file slices", clusteringInstantTime, clusteringGroup.getSlices().size());
output.collect(new StreamRecord<>(
- new ClusteringPlanEvent(clusteringInstantTime, ClusteringGroupInfo.create(clusteringGroup), clusteringPlan.getStrategy().getStrategyParams())
+ new ClusteringPlanEvent(clusteringInstantTime, ClusteringGroupInfo.create(clusteringGroup), clusteringPlan.getStrategy().getStrategyParams(), operationIndex)
Review Comment:
Fileids of each operation in clustering plan is unique. So I think a hash map is not necessary. WDYT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fix skew in clustering operator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12765:
URL: https://github.com/apache/hudi/pull/12765#issuecomment-2638995194
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3344",
"triggerID" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"triggerType" : "PUSH"
}, {
"hash" : "249df0d03775ce2b398ee80d51023b95052e3116",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "249df0d03775ce2b398ee80d51023b95052e3116",
"triggerType" : "PUSH"
}, {
"hash" : "e974bc388e3c2aad6e2a9945c49e3cba5ec97aae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3347",
"triggerID" : "e974bc388e3c2aad6e2a9945c49e3cba5ec97aae",
"triggerType" : "PUSH"
}, {
"hash" : "e1596e13d426376aafb8eab07a7b4db6ed90b099",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3349",
"triggerID" : "e1596e13d426376aafb8eab07a7b4db6ed90b099",
"triggerType" : "PUSH"
}, {
"hash" : "8a84f357581feeda414477eec8fcdff16b63389a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3416",
"triggerID" : "8a84f357581feeda414477eec8fcdff16b63389a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 249df0d03775ce2b398ee80d51023b95052e3116 UNKNOWN
* e1596e13d426376aafb8eab07a7b4db6ed90b099 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3349)
* 8a84f357581feeda414477eec8fcdff16b63389a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3416)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fix skew in clustering operator [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12765:
URL: https://github.com/apache/hudi/pull/12765#discussion_r1942272531
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/clustering/ClusteringPlanOperator.java:
##########
@@ -139,11 +139,13 @@ private void scheduleClustering(HoodieFlinkTable<?> table, long checkpointId) {
ClusteringUtils.transitionClusteringOrReplaceRequestedToInflight(clusteringInstant, Option.empty(), table.getActiveTimeline());
table.getMetaClient().reloadActiveTimeline();
+ int operationIndex = 0;
for (HoodieClusteringGroup clusteringGroup : clusteringPlan.getInputGroups()) {
LOG.info("Execute clustering plan for instant {} as {} file slices", clusteringInstantTime, clusteringGroup.getSlices().size());
output.collect(new StreamRecord<>(
- new ClusteringPlanEvent(clusteringInstantTime, ClusteringGroupInfo.create(clusteringGroup), clusteringPlan.getStrategy().getStrategyParams())
+ new ClusteringPlanEvent(clusteringInstantTime, ClusteringGroupInfo.create(clusteringGroup), clusteringPlan.getStrategy().getStrategyParams(), operationIndex)
Review Comment:
Is it possible the operation come from two different plans?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fix skew in clustering operator [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12765:
URL: https://github.com/apache/hudi/pull/12765#discussion_r1940347744
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/clustering/ClusteringPlanOperator.java:
##########
@@ -139,11 +139,13 @@ private void scheduleClustering(HoodieFlinkTable<?> table, long checkpointId) {
ClusteringUtils.transitionClusteringOrReplaceRequestedToInflight(clusteringInstant, Option.empty(), table.getActiveTimeline());
table.getMetaClient().reloadActiveTimeline();
+ int operationIndex = 0;
for (HoodieClusteringGroup clusteringGroup : clusteringPlan.getInputGroups()) {
LOG.info("Execute clustering plan for instant {} as {} file slices", clusteringInstantTime, clusteringGroup.getSlices().size());
output.collect(new StreamRecord<>(
- new ClusteringPlanEvent(clusteringInstantTime, ClusteringGroupInfo.create(clusteringGroup), clusteringPlan.getStrategy().getStrategyParams())
+ new ClusteringPlanEvent(clusteringInstantTime, ClusteringGroupInfo.create(clusteringGroup), clusteringPlan.getStrategy().getStrategyParams(), operationIndex)
Review Comment:
do we need a hash map like in this PR: https://github.com/apache/hudi/pull/11757/files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fix skew in clustering operator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12765:
URL: https://github.com/apache/hudi/pull/12765#issuecomment-2630193317
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3344",
"triggerID" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"triggerType" : "PUSH"
}, {
"hash" : "249df0d03775ce2b398ee80d51023b95052e3116",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "249df0d03775ce2b398ee80d51023b95052e3116",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f2d59eb53a2455c0069f02549d45478a5c7c2d3 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3344)
* 249df0d03775ce2b398ee80d51023b95052e3116 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fix skew in clustering operator [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #12765:
URL: https://github.com/apache/hudi/pull/12765
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fix skew in clustering operator [hudi]
Posted by "JoyJoyJo (via GitHub)" <gi...@apache.org>.
JoyJoyJo commented on code in PR #12765:
URL: https://github.com/apache/hudi/pull/12765#discussion_r1944195648
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/clustering/ClusteringPlanOperator.java:
##########
@@ -139,11 +139,13 @@ private void scheduleClustering(HoodieFlinkTable<?> table, long checkpointId) {
ClusteringUtils.transitionClusteringOrReplaceRequestedToInflight(clusteringInstant, Option.empty(), table.getActiveTimeline());
table.getMetaClient().reloadActiveTimeline();
+ int operationIndex = 0;
for (HoodieClusteringGroup clusteringGroup : clusteringPlan.getInputGroups()) {
LOG.info("Execute clustering plan for instant {} as {} file slices", clusteringInstantTime, clusteringGroup.getSlices().size());
output.collect(new StreamRecord<>(
- new ClusteringPlanEvent(clusteringInstantTime, ClusteringGroupInfo.create(clusteringGroup), clusteringPlan.getStrategy().getStrategyParams())
+ new ClusteringPlanEvent(clusteringInstantTime, ClusteringGroupInfo.create(clusteringGroup), clusteringPlan.getStrategy().getStrategyParams(), operationIndex)
Review Comment:
ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fix skew in clustering operator [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12765:
URL: https://github.com/apache/hudi/pull/12765#discussion_r1944013015
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/clustering/ClusteringPlanOperator.java:
##########
@@ -139,11 +139,13 @@ private void scheduleClustering(HoodieFlinkTable<?> table, long checkpointId) {
ClusteringUtils.transitionClusteringOrReplaceRequestedToInflight(clusteringInstant, Option.empty(), table.getActiveTimeline());
table.getMetaClient().reloadActiveTimeline();
+ int operationIndex = 0;
for (HoodieClusteringGroup clusteringGroup : clusteringPlan.getInputGroups()) {
LOG.info("Execute clustering plan for instant {} as {} file slices", clusteringInstantTime, clusteringGroup.getSlices().size());
output.collect(new StreamRecord<>(
- new ClusteringPlanEvent(clusteringInstantTime, ClusteringGroupInfo.create(clusteringGroup), clusteringPlan.getStrategy().getStrategyParams())
+ new ClusteringPlanEvent(clusteringInstantTime, ClusteringGroupInfo.create(clusteringGroup), clusteringPlan.getStrategy().getStrategyParams(), operationIndex)
Review Comment:
Why, because the plan generator can actually handle multiple plans actually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fix skew in clustering operator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12765:
URL: https://github.com/apache/hudi/pull/12765#issuecomment-2630534976
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3344",
"triggerID" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"triggerType" : "PUSH"
}, {
"hash" : "249df0d03775ce2b398ee80d51023b95052e3116",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "249df0d03775ce2b398ee80d51023b95052e3116",
"triggerType" : "PUSH"
}, {
"hash" : "e974bc388e3c2aad6e2a9945c49e3cba5ec97aae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3347",
"triggerID" : "e974bc388e3c2aad6e2a9945c49e3cba5ec97aae",
"triggerType" : "PUSH"
}, {
"hash" : "e1596e13d426376aafb8eab07a7b4db6ed90b099",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3349",
"triggerID" : "e1596e13d426376aafb8eab07a7b4db6ed90b099",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 249df0d03775ce2b398ee80d51023b95052e3116 UNKNOWN
* e1596e13d426376aafb8eab07a7b4db6ed90b099 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3349)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fix skew in clustering operator [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12765:
URL: https://github.com/apache/hudi/pull/12765#discussion_r1944179837
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/clustering/ClusteringPlanOperator.java:
##########
@@ -139,11 +139,13 @@ private void scheduleClustering(HoodieFlinkTable<?> table, long checkpointId) {
ClusteringUtils.transitionClusteringOrReplaceRequestedToInflight(clusteringInstant, Option.empty(), table.getActiveTimeline());
table.getMetaClient().reloadActiveTimeline();
+ int operationIndex = 0;
for (HoodieClusteringGroup clusteringGroup : clusteringPlan.getInputGroups()) {
LOG.info("Execute clustering plan for instant {} as {} file slices", clusteringInstantTime, clusteringGroup.getSlices().size());
output.collect(new StreamRecord<>(
- new ClusteringPlanEvent(clusteringInstantTime, ClusteringGroupInfo.create(clusteringGroup), clusteringPlan.getStrategy().getStrategyParams())
+ new ClusteringPlanEvent(clusteringInstantTime, ClusteringGroupInfo.create(clusteringGroup), clusteringPlan.getStrategy().getStrategyParams(), operationIndex)
Review Comment:
let's add a hash map just like the compaction does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fix skew in clustering operator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12765:
URL: https://github.com/apache/hudi/pull/12765#issuecomment-2639191202
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3344",
"triggerID" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"triggerType" : "PUSH"
}, {
"hash" : "249df0d03775ce2b398ee80d51023b95052e3116",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "249df0d03775ce2b398ee80d51023b95052e3116",
"triggerType" : "PUSH"
}, {
"hash" : "e974bc388e3c2aad6e2a9945c49e3cba5ec97aae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3347",
"triggerID" : "e974bc388e3c2aad6e2a9945c49e3cba5ec97aae",
"triggerType" : "PUSH"
}, {
"hash" : "e1596e13d426376aafb8eab07a7b4db6ed90b099",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3349",
"triggerID" : "e1596e13d426376aafb8eab07a7b4db6ed90b099",
"triggerType" : "PUSH"
}, {
"hash" : "8a84f357581feeda414477eec8fcdff16b63389a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3416",
"triggerID" : "8a84f357581feeda414477eec8fcdff16b63389a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 249df0d03775ce2b398ee80d51023b95052e3116 UNKNOWN
* 8a84f357581feeda414477eec8fcdff16b63389a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3416)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fix skew in clustering operator [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12765:
URL: https://github.com/apache/hudi/pull/12765#issuecomment-2630190593
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3344",
"triggerID" : "4f2d59eb53a2455c0069f02549d45478a5c7c2d3",
"triggerType" : "PUSH"
}, {
"hash" : "249df0d03775ce2b398ee80d51023b95052e3116",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "249df0d03775ce2b398ee80d51023b95052e3116",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f2d59eb53a2455c0069f02549d45478a5c7c2d3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3344)
* 249df0d03775ce2b398ee80d51023b95052e3116 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Fix skew in clustering operator [hudi]
Posted by "JoyJoyJo (via GitHub)" <gi...@apache.org>.
JoyJoyJo commented on code in PR #12765:
URL: https://github.com/apache/hudi/pull/12765#discussion_r1942346434
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/clustering/ClusteringPlanOperator.java:
##########
@@ -139,11 +139,13 @@ private void scheduleClustering(HoodieFlinkTable<?> table, long checkpointId) {
ClusteringUtils.transitionClusteringOrReplaceRequestedToInflight(clusteringInstant, Option.empty(), table.getActiveTimeline());
table.getMetaClient().reloadActiveTimeline();
+ int operationIndex = 0;
for (HoodieClusteringGroup clusteringGroup : clusteringPlan.getInputGroups()) {
LOG.info("Execute clustering plan for instant {} as {} file slices", clusteringInstantTime, clusteringGroup.getSlices().size());
output.collect(new StreamRecord<>(
- new ClusteringPlanEvent(clusteringInstantTime, ClusteringGroupInfo.create(clusteringGroup), clusteringPlan.getStrategy().getStrategyParams())
+ new ClusteringPlanEvent(clusteringInstantTime, ClusteringGroupInfo.create(clusteringGroup), clusteringPlan.getStrategy().getStrategyParams(), operationIndex)
Review Comment:
No, a hash map here can only store operations of one plan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org