[jira] [Created] (SPARK-23825) [K8s] Spark pods should request memory + memoryOverhead as resources

["David Vogelbacher (JIRA)" <ji...@apache.org>]

David Vogelbacher created SPARK-23825:
-----------------------------------------

             Summary: [K8s] Spark pods should request memory + memoryOverhead as resources
                 Key: SPARK-23825
                 URL: https://issues.apache.org/jira/browse/SPARK-23825
             Project: Spark
          Issue Type: Bug
          Components: Kubernetes
    Affects Versions: 2.3.0
            Reporter: David Vogelbacher


We currently request `spark.{driver,executor}.memory` as memory from Kubernetes (e.g., [here|https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/steps/BasicDriverConfigurationStep.scala#L95]).
The limit is set to `spark.{driver,executor}.memory + spark.kubernetes.{driver,executor}.memoryOverhead`.
This seems to be using Kubernetes wrong. 
[How Pods with resource limits are run|https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#how-pods-with-resource-limits-are-run], states"

{noformat}
If a Container exceeds its memory request, it is likely that its Pod will be evicted whenever the node runs out of memory.
{noformat}

Thus, if a the  spark driver/executor uses `memory + memoryOverhead` memory, it can be evicted. While an executor might get restarted (but it would still be very bad performance-wise), the driver would be hard to recover.

I think spark should be able to run with the requested (and, thus, guaranteed) resources from Kubernetes. It shouldn't rely on optional resources above the request and, therefore, be in danger of termination on high cluster utilization.

Thus, we shoud request `memory + memoryOverhead` memory from Kubernetes (and this should also be the limit).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org