[jira] [Commented] (AIRFLOW-6086) SparkSubmitOperator - Unable to override spark_binary

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000877#comment-17000877 ] 

ASF subversion and git services commented on AIRFLOW-6086:
----------------------------------------------------------

Commit 6cf276be67383754c7f815dfd437e563f1e63570 in airflow's branch refs/heads/master from Florian Ferreira
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=6cf276b ]

[AIRFLOW-6086] Correctly pick up spark_binary from Connectin in SparkSubmitOperator (#6680)

Remove default value of spark_binary from the Operator because it's better managed on spark hook.

> SparkSubmitOperator - Unable to override spark_binary 
> ------------------------------------------------------
>
>                 Key: AIRFLOW-6086
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6086
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: contrib, core
>    Affects Versions: 1.10.6
>            Reporter: Florian FERREIRA
>            Priority: Major
>
> Hello,
> I have a connection "spark2_default" : 
> || Conn Id || Conn Type ||  Host || Port || Is Encrypted || Is Extra Encrypted || Extra  
> | 'spark2_default' | 'spark2'  | 'yarn-cluster' | None | False | False | {"master":"yarn-cluster","deploy-mode":"cluster","spark-binary":"spark2-submit"} |
> Extra contains 'spark-binary' key that was use by airflow 1.10.2 to choose spark-submit operator. But in version 1.10.6 this config is ignore.
> I think that , in class SparkSubmitOperator in init function they has a default value "spark-submit" for spark_binary parameter. 
> {code}
>  spark_binary="spark-submit",
> {code}
> Therefore in class SparkSubmitHook when we control if spark_binary is empty it can't be.
> {code}
> conn_data['spark_binary'] = self._spark_binary or  \
>                 extra.get('spark-binary', "spark-submit")
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)