[jira] [Resolved] (AIRFLOW-4717) The spark_binary arg to SparkSubmitOperator has no effect when connection exists.

["Ash Berlin-Taylor (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-4717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ash Berlin-Taylor resolved AIRFLOW-4717.
----------------------------------------
    Resolution: Duplicate

> The spark_binary arg to SparkSubmitOperator has no effect when connection exists.
> ---------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-4717
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4717
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: hooks
>    Affects Versions: 1.10.3
>            Reporter: yunsu lee
>            Assignee: yunsu lee
>            Priority: Major
>
> Apache spark depending on the desto, the spark binary name may be different. (ex. spark2-submit)
> For this reason, the spark_binary option has been added to sparkSubmitOperator
> (Reference : [https://github.com/apache/airflow/pull/4360/files])
>  
> However, this option does not work.
> This is because there is logic to hard-code and override the spark-binary option value in spark_submit_hook.py
> (Full path : airflow/contrib/hooks/spark_submit_hook.py)
>  
> {code:java}
> ...
> conn_data['spark_binary'] = extra.get('spark-binary', "spark-submit")
> ...{code}
> It is necessary to delete the corresponding line.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)