[jira] [Commented] (SPARK-27499) Support mapping spark.local.dir to hostPath volume

["Marcelo Vanzin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-27499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16885578#comment-16885578 ] 

Marcelo Vanzin commented on SPARK-27499:
----------------------------------------

I can't see an option to reopen this, so I'll clone it instead. This seems like a simple fix that can at least help people experiment with different storage.

> Support mapping spark.local.dir to hostPath volume
> --------------------------------------------------
>
>                 Key: SPARK-27499
>                 URL: https://issues.apache.org/jira/browse/SPARK-27499
>             Project: Spark
>          Issue Type: Improvement
>          Components: Kubernetes
>    Affects Versions: 3.0.0
>            Reporter: Junjie Chen
>            Priority: Minor
>
> Currently, the k8s executor builder mount spark.local.dir as emptyDir or memory, it should satisfy some small workload, while in some heavily workload like TPCDS, both of them can have some problem, such as pods are evicted due to disk pressure when using emptyDir, and OOM when using tmpfs.
> In particular on cloud environment, users may allocate cluster with minimum configuration and add cloud storage when running workload. In this case, we can specify multiple elastic storage as spark.local.dir to accelerate the spilling. 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org