You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/11/02 14:46:35 UTC
[GitHub] [hudi] KnightChess opened a new pull request, #7119: [HUDI-5149] fix spark single file sort plan can not work
KnightChess opened a new pull request, #7119:
URL: https://github.com/apache/hudi/pull/7119
### Change Logs
SparkSingleFileSortExecutionStrategy extend `SparkSizeBasedClusteringPlanStrategy` will filter by config `hoodie.clustering.plan.strategy.small.file.limit`, it can not work for big file to sort inline row group
### Impact
none
### Risk level (write none, low medium or high below)
low
### Documentation Update
none
### Contributor's checklist
- [+ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7119: [HUDI-5149] fix spark single file sort plan can not work
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7119:
URL: https://github.com/apache/hudi/pull/7119#issuecomment-1300724049
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ef9219bc046eb82f8136e7e6266143e551a1de22",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ef9219bc046eb82f8136e7e6266143e551a1de22",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ef9219bc046eb82f8136e7e6266143e551a1de22 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7119: [HUDI-5149] fix spark single file sort plan can not work
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7119:
URL: https://github.com/apache/hudi/pull/7119#issuecomment-1300738736
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ef9219bc046eb82f8136e7e6266143e551a1de22",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12733",
"triggerID" : "ef9219bc046eb82f8136e7e6266143e551a1de22",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ef9219bc046eb82f8136e7e6266143e551a1de22 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12733)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7119: [HUDI-5149] fix spark single file sort plan can not work
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7119:
URL: https://github.com/apache/hudi/pull/7119#issuecomment-1301041078
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ef9219bc046eb82f8136e7e6266143e551a1de22",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12733",
"triggerID" : "ef9219bc046eb82f8136e7e6266143e551a1de22",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ef9219bc046eb82f8136e7e6266143e551a1de22 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12733)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] KnightChess commented on pull request #7119: [HUDI-5149] fix spark single file sort plan can not work
Posted by GitBox <gi...@apache.org>.
KnightChess commented on PR #7119:
URL: https://github.com/apache/hudi/pull/7119#issuecomment-1316382216
@xushiyan sorry for the late reply, I have update the impact.
For clustering job, I think there two main functions
- optimize data layout by merge file size to target file size
- optimize data layout by sort
And if I use upsert, I will auto solve small file quesion when tag, the file size will bigger than `hoodie.clustering.plan.strategy.small.file.limit`.
Now, I want to optimize data layout to make `col stat` more efficiency, and do not change the default file size. Only want to sort data with specified field. But the fille will all be filted by `hoodie.clustering.plan.strategy.small.file.limit`, and can not optimize data by sort, unless set the conf more bigger than max file size to make sure all file will be include. I think this conf is only used to achieve the function `merge small file to bigger`.
Considering this strategy : `SparkSingleFileSortExecutionStrategy`, it is only support `optimize data layout by sort` function, I think should not be limited by `hoodie.clustering.plan.strategy.small.file.limit`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] KnightChess closed pull request #7119: [HUDI-5149] fix spark single file sort plan can not work
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess closed pull request #7119: [HUDI-5149] fix spark single file sort plan can not work
URL: https://github.com/apache/hudi/pull/7119
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org