You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uniffle.apache.org by GitBox <gi...@apache.org> on 2022/11/29 07:04:32 UTC
[GitHub] [incubator-uniffle] zuston opened a new issue, #373: [Improvement] Improve stability when using MEMORY_LOCALFILE type
zuston opened a new issue, #373:
URL: https://github.com/apache/incubator-uniffle/issues/373
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### What would you like to be improved?
I want to raise this issue to improve stability when using MEMORY_LOCALFILE storage type. Maybe some issues will be as sub-tasks in this improvement.
The first improvement is to avoid all apps fail when single disk capacity reaches high-watermark. We could do below optimizations.
1. Introduce the metrics of TOP10 apps which use the number of written bytes #333 .
2. Introduce the free space & total space metrics of every local disk
3. Introduce the pluggable disk selection strategy. Currently the disk will be selected based on the hash. Free-capacity based strategy should be supported.
4. Allow app write data to another disk when encountering the corresponding disk reaching high-watermark #306
### How should we improve?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #373: [Improvement] Improve stability when using MEMORY_LOCALFILE type
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #373:
URL: https://github.com/apache/incubator-uniffle/issues/373#issuecomment-1330444439
3. We choose hash selection strategy. Because we want to reduce the size of meta data which we need maintain in the memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] advancedxy commented on issue #373: [Improvement] Improve stability when using MEMORY_LOCALFILE type
Posted by GitBox <gi...@apache.org>.
advancedxy commented on issue #373:
URL: https://github.com/apache/incubator-uniffle/issues/373#issuecomment-1348082765
> Introduce the free space & total space metrics of every local disk
@zuston how do you plan to collect these metrics? By using `df`, or any other fancy ways?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] advancedxy commented on issue #373: [Improvement] Improve stability when using MEMORY_LOCALFILE type
Posted by GitBox <gi...@apache.org>.
advancedxy commented on issue #373:
URL: https://github.com/apache/incubator-uniffle/issues/373#issuecomment-1344020517
> Introduce the pluggable disk selection strategy. Currently the disk will be selected based on the hash. Free-capacity based strategy should be supported.
Agreed. Currently the hash based strategy may cause unbalanced disk I/Os among different disks as app's shuffle patterns may vary dramatically. Capacity and disk-stats based strategy is very nice to have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #373: [Improvement] Improve stability when using MEMORY_LOCALFILE type
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #373:
URL: https://github.com/apache/incubator-uniffle/issues/373#issuecomment-1330175380
PTAL @jerqi @xianjingfeng @leixm @smallzhongfeng @kaijchen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] xianjingfeng commented on issue #373: [Improvement] Improve stability when using MEMORY_LOCALFILE type
Posted by GitBox <gi...@apache.org>.
xianjingfeng commented on issue #373:
URL: https://github.com/apache/incubator-uniffle/issues/373#issuecomment-1331629262
3. Can we use Consistent Hashing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org