Re: [EXTERNAL] [SPARK Memory management] Does Spark support setting limits/requests for driver/executor memory ?

[Shay Elbaz <sh...@gm.com>]

Had the same issue, it seems that it is simply not possible - https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L195

There's also a Jira ticket - https://issues.apache.org/jira/browse/SPARK-37358?jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20text%20~%20%22kubernetes%20request%20limit%22%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC

[https://opengraph.githubassets.com/0507edbcc6647cf1b2753e345f2e763cefb4dacc64818621fa683d29e0ceee20/apache/spark]<https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L195>
spark/BasicExecutorFeatureStep.scala at master · apache/spark<https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L195>
Apache Spark - A unified analytics engine for large-scale data processing - spark/BasicExecutorFeatureStep.scala at master · apache/spark
github.com


________________________________
From: Yosr Kchaou <yo...@gmail.com>
Sent: Wednesday, December 7, 2022 10:19 AM
To: user@spark.apache.org <us...@spark.apache.org>
Subject: [EXTERNAL] [SPARK Memory management] Does Spark support setting limits/requests for driver/executor memory ?


ATTENTION: This email originated from outside of GM.


Hello,

We are running Spark on Kubernetes and noticed that driver/executors use the same value for memory request and memory limit. We see that limits/requests can be set only for cpu using the following options: spark.kubernetes.{driver/executor}.limit.cores and spark.kubernetes.{driver/executor}.request.cores.

In our case, it is useful to set a memory request lower than the memory limit as it helps us better manage our cluster resources.

Is there a way to define limits/requests for memory resources for driver/executor pods ?

Thanks in advance.

Yosr