You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uniffle.apache.org by GitBox <gi...@apache.org> on 2022/07/27 11:59:40 UTC
[GitHub] [incubator-uniffle] zuston opened a new issue, #92: [Performance Optimization] The huge performance drop due to the method of getBlockIdsByPartitionId
zuston opened a new issue, #92:
URL: https://github.com/apache/incubator-uniffle/issues/92
### Background
I found when getting shuffle result, the flame graph show the method of `getBlockIdsByPartitionId` occupy too much time.
But i found the default bitmapNum is 1, and the conf of `spark.rss.client.bitmap.splitNum` looks abandoned in future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #92: [Performance Optimization] The huge performance drop due to the method of getBlockIdsByPartitionId
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #92:
URL: https://github.com/apache/incubator-uniffle/issues/92#issuecomment-1197607466
Got it. If we have the better design on this, i think it will achieve better performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston closed issue #92: [Performance Optimization] The huge performance drop due to the method of getBlockIdsByPartitionId
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston closed issue #92: [Performance Optimization] The huge performance drop due to the method of getBlockIdsByPartitionId
URL: https://github.com/apache/incubator-uniffle/issues/92
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] colinmjj commented on issue #92: [Performance Optimization] The huge performance drop due to the method of getBlockIdsByPartitionId
Posted by GitBox <gi...@apache.org>.
colinmjj commented on issue #92:
URL: https://github.com/apache/incubator-uniffle/issues/92#issuecomment-1197595382
The performance problem of `getBlockIdsByPartitionId` is a known issue.
With current design, blockId should be stored in shuffle server to support features like block filter, data validation, etc.
Here is the challenge about how to store blockId in shuffle server, and we need consider how to reduce memory cost & have better performance.
The current implementation is target to reduce memory cost in Shuffle Server, but have poor performance when get blockIds, otherwise, GC will be another problem.
I think there should be a lot of works to improve this issue, maybe new design about how to do the data validation & how to store blockIds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #92: [Performance Optimization] The huge performance drop due to the method of getBlockIdsByPartitionId
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #92:
URL: https://github.com/apache/incubator-uniffle/issues/92#issuecomment-1197577668
@colinmjj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org