You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2019/02/14 08:13:00 UTC
[jira] [Assigned] (SPARK-26877) Support user-level app staging
directory in yarn mode when spark.yarn.stagingDir specified
[ https://issues.apache.org/jira/browse/SPARK-26877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-26877:
------------------------------------
Assignee: Apache Spark
> Support user-level app staging directory in yarn mode when spark.yarn.stagingDir specified
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-26877
> URL: https://issues.apache.org/jira/browse/SPARK-26877
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 2.4.0
> Reporter: liupengcheng
> Assignee: Apache Spark
> Priority: Minor
>
> Currently, when running applications on yarn mode, the app staging directory of is controlled by `spark.yarn.stagingDir` config if specified, and this directory cannot separate different users, somtimes, it's inconvenient for file and quota management for users.
> For example, user may have different quota for their own app staging dir. and they may also not want to use the home directory as the staging dir, because it might used mainly for private user files or data. The quota may also be different between private data and the app staging dir.
> So I propose to add user sub directories under this app staging dir.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org