You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/10/09 13:09:37 UTC

[GitHub] [spark] tenglei commented on pull request #38181: [SPARK-40720][CORE] Fix spark-ui jobs status not updating under high concurrency scenario

tenglei commented on PR #38181:
URL: https://github.com/apache/spark/pull/38181#issuecomment-1272539284

   Set spark.scheduler.listenerbus.eventqueue.capacity = 500, and submit as many jobs as possible in 30 minutes
   
   Before this pr
   
   We can see there are 806(completed)+66(active) of jobs in spark-ui
   ![image](https://user-images.githubusercontent.com/13283099/194758325-e89ee1c8-9676-429f-828a-43460eb6525f.png)
   ![image](https://user-images.githubusercontent.com/13283099/194758331-110a091c-e6e4-4b2c-b49a-dfba63a06c4a.png)
   
   Wait for 3h, there still lot of jobs in spark-ui that status not update
   ![image](https://user-images.githubusercontent.com/13283099/194758336-9d5f22f7-cb5a-4ca4-9c75-8b3472c92989.png)
   
   After this pr, and set 
   
   We can see there are 857(completed)+46(active) of jobs in spark-ui
   ![image](https://user-images.githubusercontent.com/13283099/194758503-81cd5ada-f22c-434e-9dec-774b2b4c6db5.png)
   
   Wait for 7 min, there 935(completed) of jobs and no active job
   ![image](https://user-images.githubusercontent.com/13283099/194758578-235e7c06-7e89-4814-9d2f-688a6f54325b.png)
   
   
   
   
   
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org