You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Björn Boschman (Jira)" <ji...@apache.org> on 2023/11/30 13:58:00 UTC

[jira] [Commented] (SPARK-37358) Spark-on-K8S: Allow disabling of resources.limits.memory in executor pod spec

    [ https://issues.apache.org/jira/browse/SPARK-37358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17791638#comment-17791638 ] 

Björn Boschman commented on SPARK-37358:
----------------------------------------

anybody ever looking into this?

we can provide a patch

> Spark-on-K8S: Allow disabling of resources.limits.memory in executor pod spec
> -----------------------------------------------------------------------------
>
>                 Key: SPARK-37358
>                 URL: https://issues.apache.org/jira/browse/SPARK-37358
>             Project: Spark
>          Issue Type: Improvement
>          Components: Kubernetes
>    Affects Versions: 3.2.0
>            Reporter: Andrew de Quincey
>            Priority: Major
>
> When spark creates an executor pod on my Kubernetes cluster, it adds the following resources definition:
> {{      resources:}}
> {{        limits:}}
> {{          memory: 896Mi}}
> {{        requests:}}
> {{          cpu: '4'}}
> {{          memory: 896Mi}}
> Note that resources.limits.cpu is not set. This is controlled by the spark.kubernetes.driver.limit.cores setting (which we intentionally do not set).
> We'd like to be able to omit the resources.limit.memory setting as well to let the spark worker expand its memory as necessary.
> However, this isn't possible. The scala code in BasicExecutorFeatureStep.scala is as follows:
> {{{}.editOrNewResources(){}}}{{{}.addToRequests("memory", executorMemoryQuantity){}}}{{{}.addToLimits("memory", executorMemoryQuantity){}}}{{{}.addToRequests("cpu", executorCpuQuantity){}}}{{{}.addToLimits(executorResourceQuantities.asJava){}}}{{{}.endResources(){}}}{{{{}}{}}}
>  
> i.e. it always adds the memory limit, and there's no way to stop it.
> Oh - most of our code is in python, so it is not bound by the JVM memory settings,



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org