[jira] [Created] (AIRFLOW-4717) The spark_binary option does not apply in sparkSubmitOperator.

["yunsu lee (JIRA)" <ji...@apache.org>]

yunsu lee created AIRFLOW-4717:
----------------------------------

             Summary: The spark_binary option does not apply in sparkSubmitOperator.
                 Key: AIRFLOW-4717
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4717
             Project: Apache Airflow
          Issue Type: Bug
          Components: hooks
    Affects Versions: 1.10.3
            Reporter: yunsu lee


Apache spark depending on the desto, the spark binary name may be different. (ex. spark2-submit)
For this reason, the spark_binary option has been added to sparkSubmitOperator

(Reference : [https://github.com/apache/airflow/pull/4360/files])

 

However, this option does not work.

This is because there is logic to hard-code and override the spark-binary option value in spark_submit_hook.py

(Full path : airflow/contrib/hooks/spark_submit_hook.py)

 
{code:java}
...
conn_data['spark_binary'] = extra.get('spark-binary', "spark-submit")
...{code}
It is necessary to delete the corresponding line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)