You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uniffle.apache.org by "zuston (via GitHub)" <gi...@apache.org> on 2023/03/08 06:27:12 UTC

[GitHub] [incubator-uniffle] zuston opened a new issue, #695: [Improvement] Huge partition optimization on local disk

zuston opened a new issue, #695:
URL: https://github.com/apache/incubator-uniffle/issues/695

   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   
   
   ### Search before asking
   
   - [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### What would you like to be improved?
   
   In #378, we have discussed the huge partition problems. And the final solution is to make it flush to HDFS directly and limit memory usage. 
   
   But for some users only using `MEMORY_LOCALFILE` storage type, it's useless. So this issue is to track the huge partition optimization on local disk.
   
   
   ### How should we improve?
   
   ## SubTasks
   1. Support flushing single partition to multiple disks
   2. Support pluggable local disk selection strategy
   3. Support dynamic assignment of shuffle server, especially when the disk of server is full
   4. ...
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] jerqi commented on issue #695: [Improvement] Huge partition optimization on local disk

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #695:
URL: https://github.com/apache/incubator-uniffle/issues/695#issuecomment-1459612007

   We could use more MEMORY_LOCALFILE_HDFS, HDFS have more capacity to process the problems.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] xumanbu commented on issue #695: [Improvement] Huge partition optimization on local disk

Posted by "xumanbu (via GitHub)" <gi...@apache.org>.
xumanbu commented on issue #695:
URL: https://github.com/apache/incubator-uniffle/issues/695#issuecomment-1494511043

   > > We could use more MEMORY_LOCALFILE_HDFS, HDFS have more capacity to process the problems.
   > 
   > Yes, I think so. But users like VIP shop hope this optimization could be applied in local disk. Could you help ping? Let's discuss more.
   
   use hdfs can solve capacity issues, but hdfs write rate is slower than ssd, the memory will be full quickly in large data written. 
   in my mind if we use hdfs storage,uniffle client should have backpressure to limit client write rate.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] zuston commented on issue #695: [Improvement] Huge partition optimization on local disk

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on issue #695:
URL: https://github.com/apache/incubator-uniffle/issues/695#issuecomment-1461190966

   > We could use more MEMORY_LOCALFILE_HDFS, HDFS have more capacity to process the problems.
   
   Yes, I think so. But users like VIP shop hope this optimization could be applied in local disk. Could you help ping? Let's discuss more.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] zuston commented on issue #695: [Improvement] Huge partition optimization on local disk

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on issue #695:
URL: https://github.com/apache/incubator-uniffle/issues/695#issuecomment-1495155805

   > > > We could use more MEMORY_LOCALFILE_HDFS, HDFS have more capacity to process the problems.
   > > 
   > > 
   > > Yes, I think so. But users like VIP shop hope this optimization could be applied in local disk. Could you help ping? Let's discuss more.
   > 
   > use hdfs can solve capacity issues, but hdfs write rate is slower than ssd, the memory will be full quickly in large data written. in my mind if we use hdfs storage,uniffle client should have backpressure to limit client write rate.
   
   Yes, it does. Please see https://github.com/apache/incubator-uniffle/blob/master/docs/server_guide.md#huge-partition-optimization


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-uniffle] jerqi commented on issue #695: [Improvement] Huge partition optimization on local disk

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #695:
URL: https://github.com/apache/incubator-uniffle/issues/695#issuecomment-1461192577

   @xumanbu @Gustfh


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org