[jira] [Created] (AIRFLOW-6086) SparkSubmitOperator - Unable to override spark_binary

["Florian FERREIRA (Jira)" <ji...@apache.org>]

Florian FERREIRA created AIRFLOW-6086:
-----------------------------------------

             Summary: SparkSubmitOperator - Unable to override spark_binary 
                 Key: AIRFLOW-6086
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6086
             Project: Apache Airflow
          Issue Type: Bug
          Components: contrib, core
    Affects Versions: 1.10.6
            Reporter: Florian FERREIRA


Hello,

I have a connection "spark2_default" : 

|| Conn Id || Conn Type ||  Host || Port || Is Encrypted || Is Extra Encrypted || Extra  
| 'spark2_default' | 'spark2'  | 'yarn-cluster' | None | False | False | {"master":"yarn-cluster","deploy-mode":"cluster","spark-binary":"spark2-submit"} |

Extra contains 'spark-binary' key that was use by airflow 1.10.2 to choose spark-submit operator. But in version 1.10.6 this config is ignore.

I think that , in class SparkSubmitOperator in init function they has a default value "spark-submit" for spark_binary parameter. 
{code}
 spark_binary="spark-submit",
{code}

Therefore in class SparkSubmitHook when we control if spark_binary is empty it can't be.

{code}
conn_data['spark_binary'] = self._spark_binary or  \
                extra.get('spark-binary', "spark-submit")
{code}





--
This message was sent by Atlassian Jira
(v8.3.4#803005)