[GitHub] [incubator-uniffle] colinmjj commented on issue #92: [Performance Optimization] The huge performance drop due to the method of getBlockIdsByPartitionId

[GitBox <gi...@apache.org>]

colinmjj commented on issue #92:
URL: https://github.com/apache/incubator-uniffle/issues/92#issuecomment-1197595382

   The performance problem of `getBlockIdsByPartitionId` is a known issue.
   With current design, blockId should be stored in shuffle server to support features like block filter, data validation, etc.
   Here is the challenge about how to store blockId in shuffle server, and we need consider how to reduce memory cost & have better performance.
   The current implementation is target to reduce memory cost in Shuffle Server, but have poor performance when get blockIds, otherwise, GC will be another problem.
   I think there should be a lot of works to improve this issue, maybe new design about how to do the data validation & how to store blockIds. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org