You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Cristòfol Torrens (JIRA)" <ji...@apache.org> on 2018/04/26 11:44:00 UTC
[jira] [Work started] (AIRFLOW-2380) Add support for environment
variables in Spark submit operator
[ https://issues.apache.org/jira/browse/AIRFLOW-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on AIRFLOW-2380 started by Cristòfol Torrens.
--------------------------------------------------
> Add support for environment variables in Spark submit operator
> --------------------------------------------------------------
>
> Key: AIRFLOW-2380
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2380
> Project: Apache Airflow
> Issue Type: Improvement
> Components: contrib, operators
> Reporter: Cristòfol Torrens
> Assignee: Cristòfol Torrens
> Priority: Minor
>
> Add support for environment variables in Spark submit operator.
> For example, to pass the *HADOOP_CONF_DIR* in case of use same Spark cluster with multiple HDFS.
> The idea is to pass as a dict, and resolve it in the case of using *yarn-*_client/cluster_*,* and *standalone-*_client_ mode.
> In *standalone-*_cluster_ mode is not possible to do this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)