You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uniffle.apache.org by GitBox <gi...@apache.org> on 2022/11/07 05:47:50 UTC
[GitHub] [incubator-uniffle] zuston opened a new issue, #306: [FEATURE] Introduce dynamic disk selection for one partition in LOCALFILE
zuston opened a new issue, #306:
URL: https://github.com/apache/incubator-uniffle/issues/306
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### Describe the feature
Introduce the dynamic disk selection for one partition in LOCALFILE, especially when some disks reached the high-watermark.
### Motivation
Currently, when mounting multiple disks in shuffle-server and using the MEMORY_LOCALFILE storage type, one app has a very large partition data and its selected disk reaches the high watermark, it will preserve the data into the memory. This will make some app hard to get the memory.
### Describe the solution
Maybe we should let the its partial big partition data store to the another disk.
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #306: [FEATURE] Introduce dynamic disk selection for one partition in LOCALFILE
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #306:
URL: https://github.com/apache/incubator-uniffle/issues/306#issuecomment-1305147049
> If you use MEMORY_LOCALFILE_HDFS, it won't be a big problem.
Yes. This is an optional solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #306: [FEATURE] Introduce dynamic disk selection for one partition in LOCALFILE
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #306:
URL: https://github.com/apache/incubator-uniffle/issues/306#issuecomment-1305668975
Actually we ever use raid disk to solve this problem when we only have localfile mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #306: [FEATURE] Introduce dynamic disk selection for one partition in LOCALFILE
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #306:
URL: https://github.com/apache/incubator-uniffle/issues/306#issuecomment-1305132348
If you use MEMORY_LOCALFILE_HDFS, it won't be a big problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org