You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/04/24 05:43:10 UTC

[GitHub] [spark] attilapiros commented on pull request #31790: [SPARK-34509][K8S] Make dynamic allocation upscaling more progressive on K8S

attilapiros commented on pull request #31790:
URL: https://github.com/apache/spark/pull/31790#issuecomment-826039250


   cc @dongjoon-hyun
   
   With this change pending PODs are not counted as outstanding PODs so their number can be quite high in k8s cluster. But still I would keep the allocation batch size to limit the max number of POD requests made at once. 
   
   I am thinking about introducing a new limit for the max number of pending PODs (if k8s tends to struggle to handle high number of pending PODs). This new limit must significantly higher then the POD allocation size (we could even derive it from the batchsize with using constant multiplier like * 10 or make the factor configurable).
   
   WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org