You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/12/03 09:55:55 UTC
[GitHub] [hudi] Zouxxyy opened a new pull request, #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Zouxxyy opened a new pull request, #7372:
URL: https://github.com/apache/hudi/pull/7372
### Change Logs
fix https://issues.apache.org/jira/browse/HUDI-5326
1. Modify the file division strategy in SparkSizeBasedClusteringPlanStrategy
(1) Sort fileSlices before dividing, which makes dividing more compact
(2) Avoid output clustering group size exceeding `hoodie.clustering.plan.strategy.max.bytes.per.group`
2. Set the maximum size of the file written by clustering to `hoodie.clustering.plan.strategy.max.bytes.per.group` to avoid the generation of small files
E.g: Under the new strategy, if we set we set max.bytes.per.group=20, target.file.max.bytes=10, the size of files by clustering will fluctuate around 10, use `target.file.max.bytes` as the threshold will split many small files when size > 10.
### Impact
refer to above
### Risk level (write none, low medium or high below)
low
### Documentation Update
None
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1376862113
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
}, {
"hash" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439",
"triggerID" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"triggerType" : "PUSH"
}, {
"hash" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181",
"triggerID" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"triggerType" : "PUSH"
}, {
"hash" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14188",
"triggerID" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"triggerType" : "PUSH"
}, {
"hash" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14192",
"triggerID" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"triggerType" : "PUSH"
}, {
"hash" : "1b10f0850302f552778588cd287eb26c091efe88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14199",
"triggerID" : "1b10f0850302f552778588cd287eb26c091efe88",
"triggerType" : "PUSH"
}, {
"hash" : "46b644c42fca50efc48bde48a4a4e32c05aa50c9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14208",
"triggerID" : "46b644c42fca50efc48bde48a4a4e32c05aa50c9",
"triggerType" : "PUSH"
}, {
"hash" : "40f9031fcc3215124deffc83d2b0f8eb55440374",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "40f9031fcc3215124deffc83d2b0f8eb55440374",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 46b644c42fca50efc48bde48a4a4e32c05aa50c9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14208)
* 40f9031fcc3215124deffc83d2b0f8eb55440374 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
Zouxxyy commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1336210565
The failure CI bug has been located, I will fix it https://issues.apache.org/jira/browse/HUDI-5328
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron merged pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
YannByron merged PR #7372:
URL: https://github.com/apache/hudi/pull/7372
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1336135400
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3425dbcfcd5ce8d51d8753806deb59b945246ea UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on a diff in pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
Zouxxyy commented on code in PR #7372:
URL: https://github.com/apache/hudi/pull/7372#discussion_r1065933284
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestLayoutOptimization.scala:
##########
@@ -111,8 +111,7 @@ class TestLayoutOptimization extends HoodieClientTestBase {
.option("hoodie.clustering.inline.max.commits", "1")
.option("hoodie.clustering.plan.strategy.target.file.max.bytes", "1073741824")
.option("hoodie.clustering.plan.strategy.small.file.limit", "629145600")
- .option("hoodie.clustering.plan.strategy.max.bytes.per.group", Long.MaxValue.toString)
Review Comment:
Let me explain the modification here: since we set the maximum size of the file written by clustering to `hoodie.clustering.plan.strategy.max.bytes.per.group`, if the value is Long.MaxValue.toString, after add by compressing ratio, it will be become negative, at this point the file will be split infinitely
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1375268795
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
}, {
"hash" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439",
"triggerID" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"triggerType" : "PUSH"
}, {
"hash" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181",
"triggerID" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 871c809bb5608259d192140e2ef4397ef6e58e56 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1336150091
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
}, {
"hash" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3425dbcfcd5ce8d51d8753806deb59b945246ea Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436)
* 85fd74a3c521f1881def7f02bdd3874f331f9686 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
Zouxxyy commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1354296872
https://github.com/apache/hudi/pull/7468 can solve the failure CI, if It is landed, I will trigger CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1375995040
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
}, {
"hash" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439",
"triggerID" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"triggerType" : "PUSH"
}, {
"hash" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181",
"triggerID" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"triggerType" : "PUSH"
}, {
"hash" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14188",
"triggerID" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"triggerType" : "PUSH"
}, {
"hash" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14192",
"triggerID" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"triggerType" : "PUSH"
}, {
"hash" : "1b10f0850302f552778588cd287eb26c091efe88",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14199",
"triggerID" : "1b10f0850302f552778588cd287eb26c091efe88",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 505effb1f7c1941df7c336cccbd12ad8338db4cb Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14192)
* 1b10f0850302f552778588cd287eb26c091efe88 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14199)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1377848736
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
}, {
"hash" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439",
"triggerID" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"triggerType" : "PUSH"
}, {
"hash" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181",
"triggerID" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"triggerType" : "PUSH"
}, {
"hash" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14188",
"triggerID" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"triggerType" : "PUSH"
}, {
"hash" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14192",
"triggerID" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"triggerType" : "PUSH"
}, {
"hash" : "1b10f0850302f552778588cd287eb26c091efe88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14199",
"triggerID" : "1b10f0850302f552778588cd287eb26c091efe88",
"triggerType" : "PUSH"
}, {
"hash" : "46b644c42fca50efc48bde48a4a4e32c05aa50c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14208",
"triggerID" : "46b644c42fca50efc48bde48a4a4e32c05aa50c9",
"triggerType" : "PUSH"
}, {
"hash" : "40f9031fcc3215124deffc83d2b0f8eb55440374",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14217",
"triggerID" : "40f9031fcc3215124deffc83d2b0f8eb55440374",
"triggerType" : "PUSH"
}, {
"hash" : "5e1056d8235b7aadc3a8f66a6a7fce9c931b3cb3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14225",
"triggerID" : "5e1056d8235b7aadc3a8f66a6a7fce9c931b3cb3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5e1056d8235b7aadc3a8f66a6a7fce9c931b3cb3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14225)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1375985946
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
}, {
"hash" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439",
"triggerID" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"triggerType" : "PUSH"
}, {
"hash" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181",
"triggerID" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"triggerType" : "PUSH"
}, {
"hash" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14188",
"triggerID" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"triggerType" : "PUSH"
}, {
"hash" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14192",
"triggerID" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"triggerType" : "PUSH"
}, {
"hash" : "1b10f0850302f552778588cd287eb26c091efe88",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b10f0850302f552778588cd287eb26c091efe88",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4db23eb5826aaa98eb5792965abb606fc0755832 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14188)
* 505effb1f7c1941df7c336cccbd12ad8338db4cb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14192)
* 1b10f0850302f552778588cd287eb26c091efe88 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1376457788
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
}, {
"hash" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439",
"triggerID" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"triggerType" : "PUSH"
}, {
"hash" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181",
"triggerID" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"triggerType" : "PUSH"
}, {
"hash" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14188",
"triggerID" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"triggerType" : "PUSH"
}, {
"hash" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14192",
"triggerID" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"triggerType" : "PUSH"
}, {
"hash" : "1b10f0850302f552778588cd287eb26c091efe88",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14199",
"triggerID" : "1b10f0850302f552778588cd287eb26c091efe88",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1b10f0850302f552778588cd287eb26c091efe88 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14199)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1336136310
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3425dbcfcd5ce8d51d8753806deb59b945246ea Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
Zouxxyy commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1341286652
The failing UT is test `Test Call run_clustering Procedure with specific instants` in `TestClusteringProcedure`
Before adding my patch, although the test passed. However, the calculation result of the orderAllFiles of the linear cluster in this example is not 20, but 201. There are two reasons:
1. `repartitionRecords` in `RowCustomColumnsSortPartitioner` use coalesce, since the number of input partitions is 1, the number of output partitions is still 1 (although the number of cluster groups is 20)
2. Use max.bytes.per.group to limit the size of the written parquet. It will split the partition in the previous step. Since the size in the memory is larger than the file size in the hard disk (maybe because of compression), the output file is 201
**It can be found that the parallelism is only 1.**
After adding my patch, step 2 doesn't do the splitting, it just generates one file, that's why my test fails.
So, we need to solve the reason 1, this is the ticket, https://issues.apache.org/jira/browse/HUDI-5328
My initial idea is to use `repartiton` instead of `coalesce`, or repartitonWithRange (but it will not sort)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1377298485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
}, {
"hash" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439",
"triggerID" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"triggerType" : "PUSH"
}, {
"hash" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181",
"triggerID" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"triggerType" : "PUSH"
}, {
"hash" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14188",
"triggerID" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"triggerType" : "PUSH"
}, {
"hash" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14192",
"triggerID" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"triggerType" : "PUSH"
}, {
"hash" : "1b10f0850302f552778588cd287eb26c091efe88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14199",
"triggerID" : "1b10f0850302f552778588cd287eb26c091efe88",
"triggerType" : "PUSH"
}, {
"hash" : "46b644c42fca50efc48bde48a4a4e32c05aa50c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14208",
"triggerID" : "46b644c42fca50efc48bde48a4a4e32c05aa50c9",
"triggerType" : "PUSH"
}, {
"hash" : "40f9031fcc3215124deffc83d2b0f8eb55440374",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14217",
"triggerID" : "40f9031fcc3215124deffc83d2b0f8eb55440374",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 40f9031fcc3215124deffc83d2b0f8eb55440374 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14217)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1377540837
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
}, {
"hash" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439",
"triggerID" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"triggerType" : "PUSH"
}, {
"hash" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181",
"triggerID" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"triggerType" : "PUSH"
}, {
"hash" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14188",
"triggerID" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"triggerType" : "PUSH"
}, {
"hash" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14192",
"triggerID" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"triggerType" : "PUSH"
}, {
"hash" : "1b10f0850302f552778588cd287eb26c091efe88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14199",
"triggerID" : "1b10f0850302f552778588cd287eb26c091efe88",
"triggerType" : "PUSH"
}, {
"hash" : "46b644c42fca50efc48bde48a4a4e32c05aa50c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14208",
"triggerID" : "46b644c42fca50efc48bde48a4a4e32c05aa50c9",
"triggerType" : "PUSH"
}, {
"hash" : "40f9031fcc3215124deffc83d2b0f8eb55440374",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14217",
"triggerID" : "40f9031fcc3215124deffc83d2b0f8eb55440374",
"triggerType" : "PUSH"
}, {
"hash" : "5e1056d8235b7aadc3a8f66a6a7fce9c931b3cb3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e1056d8235b7aadc3a8f66a6a7fce9c931b3cb3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 40f9031fcc3215124deffc83d2b0f8eb55440374 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14217)
* 5e1056d8235b7aadc3a8f66a6a7fce9c931b3cb3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on a diff in pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
Zouxxyy commented on code in PR #7372:
URL: https://github.com/apache/hudi/pull/7372#discussion_r1065933284
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestLayoutOptimization.scala:
##########
@@ -111,8 +111,7 @@ class TestLayoutOptimization extends HoodieClientTestBase {
.option("hoodie.clustering.inline.max.commits", "1")
.option("hoodie.clustering.plan.strategy.target.file.max.bytes", "1073741824")
.option("hoodie.clustering.plan.strategy.small.file.limit", "629145600")
- .option("hoodie.clustering.plan.strategy.max.bytes.per.group", Long.MaxValue.toString)
Review Comment:
Let me explain the modification here: since we set the maximum size of the file written by clustering to `hoodie.clustering.plan.strategy.max.bytes.per.group`, if the value is Long.MaxValue.toString, after add by compressing ratio, it will be become negative, at this point the file will be split infinitely, which cause the test case to time out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1375379722
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
}, {
"hash" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439",
"triggerID" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"triggerType" : "PUSH"
}, {
"hash" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181",
"triggerID" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"triggerType" : "PUSH"
}, {
"hash" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14188",
"triggerID" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 871c809bb5608259d192140e2ef4397ef6e58e56 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181)
* 4db23eb5826aaa98eb5792965abb606fc0755832 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14188)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1375607723
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
}, {
"hash" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439",
"triggerID" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"triggerType" : "PUSH"
}, {
"hash" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181",
"triggerID" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"triggerType" : "PUSH"
}, {
"hash" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14188",
"triggerID" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"triggerType" : "PUSH"
}, {
"hash" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14192",
"triggerID" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4db23eb5826aaa98eb5792965abb606fc0755832 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14188)
* 505effb1f7c1941df7c336cccbd12ad8338db4cb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14192)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1376868599
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
}, {
"hash" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439",
"triggerID" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"triggerType" : "PUSH"
}, {
"hash" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181",
"triggerID" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"triggerType" : "PUSH"
}, {
"hash" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14188",
"triggerID" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"triggerType" : "PUSH"
}, {
"hash" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14192",
"triggerID" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"triggerType" : "PUSH"
}, {
"hash" : "1b10f0850302f552778588cd287eb26c091efe88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14199",
"triggerID" : "1b10f0850302f552778588cd287eb26c091efe88",
"triggerType" : "PUSH"
}, {
"hash" : "46b644c42fca50efc48bde48a4a4e32c05aa50c9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14208",
"triggerID" : "46b644c42fca50efc48bde48a4a4e32c05aa50c9",
"triggerType" : "PUSH"
}, {
"hash" : "40f9031fcc3215124deffc83d2b0f8eb55440374",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14217",
"triggerID" : "40f9031fcc3215124deffc83d2b0f8eb55440374",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 46b644c42fca50efc48bde48a4a4e32c05aa50c9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14208)
* 40f9031fcc3215124deffc83d2b0f8eb55440374 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14217)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on a diff in pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
Zouxxyy commented on code in PR #7372:
URL: https://github.com/apache/hudi/pull/7372#discussion_r1042467412
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/plan/strategy/SparkSizeBasedClusteringPlanStrategy.java:
##########
@@ -65,11 +65,19 @@ protected Stream<HoodieClusteringGroup> buildClusteringGroupsForPartition(String
List<Pair<List<FileSlice>, Integer>> fileSliceGroups = new ArrayList<>();
List<FileSlice> currentGroup = new ArrayList<>();
+
+ // Sort fileSlices before dividing, which makes dividing more compact
+ List<FileSlice> sortedFileSlices = new ArrayList<>(fileSlices);
+ sortedFileSlices.sort((o1, o2) -> (int)
+ ((o2.getBaseFile().isPresent() ? o2.getBaseFile().get().getFileSize() : writeConfig.getParquetMaxFileSize())
Review Comment:
thankyou
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1376856211
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
}, {
"hash" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439",
"triggerID" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"triggerType" : "PUSH"
}, {
"hash" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181",
"triggerID" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"triggerType" : "PUSH"
}, {
"hash" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14188",
"triggerID" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"triggerType" : "PUSH"
}, {
"hash" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14192",
"triggerID" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"triggerType" : "PUSH"
}, {
"hash" : "1b10f0850302f552778588cd287eb26c091efe88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14199",
"triggerID" : "1b10f0850302f552778588cd287eb26c091efe88",
"triggerType" : "PUSH"
}, {
"hash" : "46b644c42fca50efc48bde48a4a4e32c05aa50c9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14208",
"triggerID" : "46b644c42fca50efc48bde48a4a4e32c05aa50c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 46b644c42fca50efc48bde48a4a4e32c05aa50c9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14208)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1375597450
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
}, {
"hash" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439",
"triggerID" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"triggerType" : "PUSH"
}, {
"hash" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181",
"triggerID" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"triggerType" : "PUSH"
}, {
"hash" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14188",
"triggerID" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"triggerType" : "PUSH"
}, {
"hash" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 871c809bb5608259d192140e2ef4397ef6e58e56 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181)
* 4db23eb5826aaa98eb5792965abb606fc0755832 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14188)
* 505effb1f7c1941df7c336cccbd12ad8338db4cb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1375369579
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
}, {
"hash" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439",
"triggerID" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"triggerType" : "PUSH"
}, {
"hash" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181",
"triggerID" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"triggerType" : "PUSH"
}, {
"hash" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 871c809bb5608259d192140e2ef4397ef6e58e56 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181)
* 4db23eb5826aaa98eb5792965abb606fc0755832 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1376628003
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
}, {
"hash" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439",
"triggerID" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"triggerType" : "PUSH"
}, {
"hash" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181",
"triggerID" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"triggerType" : "PUSH"
}, {
"hash" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14188",
"triggerID" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"triggerType" : "PUSH"
}, {
"hash" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14192",
"triggerID" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"triggerType" : "PUSH"
}, {
"hash" : "1b10f0850302f552778588cd287eb26c091efe88",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14199",
"triggerID" : "1b10f0850302f552778588cd287eb26c091efe88",
"triggerType" : "PUSH"
}, {
"hash" : "46b644c42fca50efc48bde48a4a4e32c05aa50c9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14208",
"triggerID" : "46b644c42fca50efc48bde48a4a4e32c05aa50c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1b10f0850302f552778588cd287eb26c091efe88 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14199)
* 46b644c42fca50efc48bde48a4a4e32c05aa50c9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14208)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1375043870
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
}, {
"hash" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439",
"triggerID" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"triggerType" : "PUSH"
}, {
"hash" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 85fd74a3c521f1881def7f02bdd3874f331f9686 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439)
* 871c809bb5608259d192140e2ef4397ef6e58e56 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1336151241
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
}, {
"hash" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439",
"triggerID" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3425dbcfcd5ce8d51d8753806deb59b945246ea Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436)
* 85fd74a3c521f1881def7f02bdd3874f331f9686 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1336193770
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
}, {
"hash" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439",
"triggerID" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 85fd74a3c521f1881def7f02bdd3874f331f9686 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1376621847
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
}, {
"hash" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439",
"triggerID" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"triggerType" : "PUSH"
}, {
"hash" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181",
"triggerID" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"triggerType" : "PUSH"
}, {
"hash" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14188",
"triggerID" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"triggerType" : "PUSH"
}, {
"hash" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14192",
"triggerID" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"triggerType" : "PUSH"
}, {
"hash" : "1b10f0850302f552778588cd287eb26c091efe88",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14199",
"triggerID" : "1b10f0850302f552778588cd287eb26c091efe88",
"triggerType" : "PUSH"
}, {
"hash" : "46b644c42fca50efc48bde48a4a4e32c05aa50c9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "46b644c42fca50efc48bde48a4a4e32c05aa50c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1b10f0850302f552778588cd287eb26c091efe88 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14199)
* 46b644c42fca50efc48bde48a4a4e32c05aa50c9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1375081376
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
}, {
"hash" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439",
"triggerID" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"triggerType" : "PUSH"
}, {
"hash" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181",
"triggerID" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 85fd74a3c521f1881def7f02bdd3874f331f9686 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439)
* 871c809bb5608259d192140e2ef4397ef6e58e56 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7372:
URL: https://github.com/apache/hudi/pull/7372#issuecomment-1377550805
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13436",
"triggerID" : "d3425dbcfcd5ce8d51d8753806deb59b945246ea",
"triggerType" : "PUSH"
}, {
"hash" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13439",
"triggerID" : "85fd74a3c521f1881def7f02bdd3874f331f9686",
"triggerType" : "PUSH"
}, {
"hash" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14181",
"triggerID" : "871c809bb5608259d192140e2ef4397ef6e58e56",
"triggerType" : "PUSH"
}, {
"hash" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14188",
"triggerID" : "4db23eb5826aaa98eb5792965abb606fc0755832",
"triggerType" : "PUSH"
}, {
"hash" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14192",
"triggerID" : "505effb1f7c1941df7c336cccbd12ad8338db4cb",
"triggerType" : "PUSH"
}, {
"hash" : "1b10f0850302f552778588cd287eb26c091efe88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14199",
"triggerID" : "1b10f0850302f552778588cd287eb26c091efe88",
"triggerType" : "PUSH"
}, {
"hash" : "46b644c42fca50efc48bde48a4a4e32c05aa50c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14208",
"triggerID" : "46b644c42fca50efc48bde48a4a4e32c05aa50c9",
"triggerType" : "PUSH"
}, {
"hash" : "40f9031fcc3215124deffc83d2b0f8eb55440374",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14217",
"triggerID" : "40f9031fcc3215124deffc83d2b0f8eb55440374",
"triggerType" : "PUSH"
}, {
"hash" : "5e1056d8235b7aadc3a8f66a6a7fce9c931b3cb3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14225",
"triggerID" : "5e1056d8235b7aadc3a8f66a6a7fce9c931b3cb3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 40f9031fcc3215124deffc83d2b0f8eb55440374 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14217)
* 5e1056d8235b7aadc3a8f66a6a7fce9c931b3cb3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14225)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] SteNicholas commented on a diff in pull request #7372: [HUDI-5326] Fix clustering group building in SparkSizeBasedClusteringPlanStrategy
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on code in PR #7372:
URL: https://github.com/apache/hudi/pull/7372#discussion_r1041908081
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/plan/strategy/SparkSizeBasedClusteringPlanStrategy.java:
##########
@@ -65,11 +65,19 @@ protected Stream<HoodieClusteringGroup> buildClusteringGroupsForPartition(String
List<Pair<List<FileSlice>, Integer>> fileSliceGroups = new ArrayList<>();
List<FileSlice> currentGroup = new ArrayList<>();
+
+ // Sort fileSlices before dividing, which makes dividing more compact
+ List<FileSlice> sortedFileSlices = new ArrayList<>(fileSlices);
+ sortedFileSlices.sort((o1, o2) -> (int)
+ ((o2.getBaseFile().isPresent() ? o2.getBaseFile().get().getFileSize() : writeConfig.getParquetMaxFileSize())
Review Comment:
```suggestion
((o2.getBaseFile().isPresent() && o1.getBaseFile().isPresent()) ? o2.getBaseFile().get().getFileSize()-o1.getBaseFile().get().getFileSize() : writeConfig.getParquetMaxFileSize()- writeConfig.getParquetMaxFileSize())
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org