Re: [PR] [SPARK-35723] set k8s pod container request, limit memory separately. [spark]

["bnetzi (via GitHub)" <gi...@apache.org>]

bnetzi commented on PR #40771:
URL: https://github.com/apache/spark/pull/40771#issuecomment-2619648271

   I know this PR is closed for a while but I think the decision not to merge it was not correct.
   As mentioned in this issue:
    https://issues.apache.org/jira/browse/SPARK-35723
    And in 
    https://issues.apache.org/jira/browse/SPARK-37358
    
   Although the default of using limits same as request make some sense, in some cases it is much wiser to have a larger limit.
   
   Let's say many executors / drivers are sharing the same instance - if the limit and the request are the same - you must allocate to each pod the **max** memory it would use, causing a huge over allocation.
   It is very typical to have a momentarily high memory usage, every user should have the power to decide if it makes sense for him to have a large limit. it has its risks, but in many use cases the cost saving might worth it.
   
   For now in our env we are actually using (for almost a year) webhooks to override this behavior, I think it should be allowed natively.  
    


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org