[GitHub] [incubator-uniffle] advancedxy commented on issue #503: [Bug] Frequent and long lasting full GCs when there are too many partitions on a shuffle server

["advancedxy (via GitHub)" <gi...@apache.org>]

advancedxy commented on issue #503:
URL: https://github.com/apache/incubator-uniffle/issues/503#issuecomment-1413620158

   > > The size of RPC queue is unlimited, is it possible relate to this?
   > 
   > Maybe. But from the metric dashboard, the GC is always along with the huge partition appearance. I will share some metrics of one shuffle-server.
   > 
   > **CMS single GC cost 20s** ![image](https://user-images.githubusercontent.com/8609142/216214614-7080b6ea-3c81-438f-8924-ce3959d5728b.png)
   > 
   > ![image](https://user-images.githubusercontent.com/8609142/216214694-77cba0df-aebf-4bcb-b868-0d6680e27dcf.png) ![image](https://user-images.githubusercontent.com/8609142/216214700-4aeb9c7b-6a5d-4d99-96ec-dc56684b5fe8.png) ![image](https://user-images.githubusercontent.com/8609142/216214707-038bfd6e-b808-4dcf-a30a-09fe29fc2c75.png)
   > 
   > From this metric, I think the single buffer flush may solve this problem.
   
   <img width="917" alt="image" src="https://user-images.githubusercontent.com/807537/216317369-a41ea271-2861-4899-9b15-0c670dcd04a8.png">
   Seems like the pre-allocation is also failed when the huge partition is arriving.  I think this should match your speculation, shuffle server just cannot gc data quicker than the shuffle server. 
   
   > I will enable the single buffer flush to avoid this problem
   
   Single buffer flush should have some effect.
    


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org