[jira] [Created] (SPARK-37358) Spark-on-K8S: Allow disabling of resources.limits.memory in executor pod spec

["Andrew de Quincey (Jira)" <ji...@apache.org>]

Andrew de Quincey created SPARK-37358:
-----------------------------------------

             Summary: Spark-on-K8S: Allow disabling of resources.limits.memory in executor pod spec
                 Key: SPARK-37358
                 URL: https://issues.apache.org/jira/browse/SPARK-37358
             Project: Spark
          Issue Type: Improvement
          Components: Kubernetes
    Affects Versions: 3.2.0
            Reporter: Andrew de Quincey


When spark creates an executor pod on my Kubernetes cluster, it adds the following resources definition:

{{      resources:}}
{{        limits:}}
{{          memory: 896Mi}}
{{        requests:}}
{{          cpu: '4'}}
{{          memory: 896Mi}}
Note that resources.limits.cpu is not set. This is controlled by the spark.kubernetes.driver.limit.cores setting (which we intentionally do not set).

We'd like to be able to omit resources.limit.memory as well to let the spark worker expand memory as necessary.

However, this isn't possible. The scala code in BasicExecutorFeatureStep.scala is as follows:
{{{}.editOrNewResources(){}}}{{{}.addToRequests("memory", executorMemoryQuantity){}}}{{{}.addToLimits("memory", executorMemoryQuantity){}}}{{{}.addToRequests("cpu", executorCpuQuantity){}}}{{{}.addToLimits(executorResourceQuantities.asJava){}}}{{{}.endResources(){}}}{{{}{}}}
 
i.e. it always adds the memory limit, and there's no way to stop it.

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org