[jira] [Created] (SPARK-43657) reuse SPARK_CONF_DIR config maps between driver and executor

["YE (Jira)" <ji...@apache.org>]

YE created SPARK-43657:
--------------------------

             Summary: reuse SPARK_CONF_DIR config maps between driver and executor
                 Key: SPARK-43657
                 URL: https://issues.apache.org/jira/browse/SPARK-43657
             Project: Spark
          Issue Type: Improvement
          Components: Kubernetes
    Affects Versions: 3.4.0, 3.3.2, 3.2.4
            Reporter: YE


Currently, Spark on k8s-cluster creates two config maps per application: one for the driver and another for the executor. However the config map for executor is almost identical for config map for driver, there's no need to create there two duplicate config maps. As ConfigMaps are object on K8S, there would be some limitations for ConfigMaps on K8S:
 # more config maps means more objects on etcd, and adds overhead to API server
 # Spark driver pod might be ran under limited permission, which means, it might not be possible to create resources rather than exec pod. Therefore driver might not be allowed to create config maps.

I would submit a pr to reuse SPARK_CONF_DIR config maps for running spark on k8s-cluster mode.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org