[jira] [Commented] (AIRFLOW-6086) SparkSubmitOperator - Unable to override spark_binary

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983615#comment-16983615 ] 

ASF GitHub Bot commented on AIRFLOW-6086:
-----------------------------------------

flo076 commented on pull request #6680: [AIRFLOW-6086] Remove default value spark_binary
URL: https://github.com/apache/airflow/pull/6680
 
 
   Remove default value of spark_binary because it's better managed on spark hook.
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [AIRFLOW-6086](https://issues.apache.org/jira/browse/AIRFLOW-6086/) issue
   
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> SparkSubmitOperator - Unable to override spark_binary 
> ------------------------------------------------------
>
>                 Key: AIRFLOW-6086
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6086
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: contrib, core
>    Affects Versions: 1.10.6
>            Reporter: Florian FERREIRA
>            Priority: Major
>
> Hello,
> I have a connection "spark2_default" : 
> || Conn Id || Conn Type ||  Host || Port || Is Encrypted || Is Extra Encrypted || Extra  
> | 'spark2_default' | 'spark2'  | 'yarn-cluster' | None | False | False | {"master":"yarn-cluster","deploy-mode":"cluster","spark-binary":"spark2-submit"} |
> Extra contains 'spark-binary' key that was use by airflow 1.10.2 to choose spark-submit operator. But in version 1.10.6 this config is ignore.
> I think that , in class SparkSubmitOperator in init function they has a default value "spark-submit" for spark_binary parameter. 
> {code}
>  spark_binary="spark-submit",
> {code}
> Therefore in class SparkSubmitHook when we control if spark_binary is empty it can't be.
> {code}
> conn_data['spark_binary'] = self._spark_binary or  \
>                 extra.get('spark-binary', "spark-submit")
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)