[jira] [Created] (SPARK-46091) [KUBERNETES] Respect the existing kubernetes container SPARK_LOCAL_DIRS env

["Fei Wang (Jira)" <ji...@apache.org>]

Fei Wang created SPARK-46091:
--------------------------------

             Summary: [KUBERNETES] Respect the existing kubernetes container SPARK_LOCAL_DIRS env
                 Key: SPARK-46091
                 URL: https://issues.apache.org/jira/browse/SPARK-46091
             Project: Spark
          Issue Type: Improvement
          Components: Kubernetes
    Affects Versions: 3.5.0
            Reporter: Fei Wang


Respect the user defined SPARK_LOCAL_DIRS container env when setup local dirs.

 

For example, we use hostPath for spark local dir.

But we do not mount the sub disks directly to the pod, we mount a root path for spark driver/executor pod.

 

For example, the root path is `/hadoop`.

 

And there are sub disks under that, likes `hadoop/1, /hadoop/2, /hadoop/3, /hadoop4`.

 

And we want to define the SPARK_LOCAL_DIRS in the driver/executor pod env.

 

But now, the user specified SPARK_LOCAL_DIRS does not work.

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org