You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Cristòfol Torrens (JIRA)" <ji...@apache.org> on 2018/04/26 11:44:00 UTC

[jira] [Work started] (AIRFLOW-2380) Add support for environment variables in Spark submit operator

     [ https://issues.apache.org/jira/browse/AIRFLOW-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on AIRFLOW-2380 started by Cristòfol Torrens.
--------------------------------------------------
> Add support for environment variables in Spark submit operator
> --------------------------------------------------------------
>
>                 Key: AIRFLOW-2380
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2380
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: contrib, operators
>            Reporter: Cristòfol Torrens
>            Assignee: Cristòfol Torrens
>            Priority: Minor
>
> Add support for environment variables in Spark submit operator.
> For example, to pass the *HADOOP_CONF_DIR* in case of use same Spark cluster with multiple HDFS.
> The idea is to pass as a dict, and resolve it in the case of using *yarn-*_client/cluster_*,* and *standalone-*_client_ mode.
> In *standalone-*_cluster_ mode is not possible to do this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)