[jira] [Commented] (SPARK-3684) Can't configure local dirs in Yarn mode

["Thomas Graves (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-3684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644359#comment-14644359 ] 

Thomas Graves commented on SPARK-3684:
--------------------------------------

What is thie use case for this?  YARN specifically doesn't allow users to do this so they don't use a directory they shouldn't be and it doesn't get cleaned up or they run out of space, etc.  As mentioned you should configure YARN to have what directories are available and then Spark use them.



> Can't configure local dirs in Yarn mode
> ---------------------------------------
>
>                 Key: SPARK-3684
>                 URL: https://issues.apache.org/jira/browse/SPARK-3684
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.1.0
>            Reporter: Andrew Or
>
> We can't set SPARK_LOCAL_DIRS or spark.local.dirs because they're not picked up in Yarn mode. However, we can't set YARN_LOCAL_DIRS or LOCAL_DIRS either because these are overridden by Yarn.
> I'm trying to set these through SPARK_YARN_USER_ENV. I'm aware that the default behavior is for Spark to use Yarn's local dirs, but right now there's no way to change it even if the user wants to.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org