You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/11/05 13:33:30 UTC

[GitHub] [spark] AngersZhuuuu commented on pull request #30139: [SPARK-31069][CORE] high cpu caused by chunksBeingTransferred in external shuffle service

AngersZhuuuu commented on pull request #30139:
URL: https://github.com/apache/spark/pull/30139#issuecomment-722380427


   > The cost for latter is linearly increasing based on the number of entities in streams, hence the problem appears if the number of entities is quite huge.
   
   Each iterate of all streams is small, but each chunk will trigger count `chunksBeingTransferred `, the cost will be  very serious
   and there is also a competition for `streams`'s locks since it is a ConcurrentHashMap.
   
   > So that sounds like a trade-off. Probably it would be pretty much beneficial to leave it as it is if we assume the number of stream entities will retain small enough, but if we assume the case where the number of stream entities are quite large, the cost of synchronizing numChunksBeingTransferred is going to be smaller.
   
   Yea, that is why I said  the cost of `synchronizing numChunksBeingTransferred` is  acceptable. 
   
   > We agreed that the value of numChunksBeingTransferred doesn't need to be strictly accurate, so this might be acceptable.
   
   From this point, I think the initial approach is good, the code is simple and easy to understand.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org