You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "Luis Angel Vicente Sanchez (JIRA)" <ji...@apache.org> on 2017/07/13 11:40:00 UTC

[jira] [Created] (AIRFLOW-1409) SparkSubmitHook doesn't generate a valid master URL

Luis Angel Vicente Sanchez created AIRFLOW-1409:
---------------------------------------------------

             Summary: SparkSubmitHook doesn't generate a valid master URL
                 Key: AIRFLOW-1409
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1409
             Project: Apache Airflow
          Issue Type: Bug
            Reporter: Luis Angel Vicente Sanchez


If you configure the connection using an ENV variable (and possibly through the UI), SparkSubmitHook is ignoring the scheme and doesn't produce valid master URL. Example, this ENV variable,

AIRFLOW_CONN_SPARK_CLUSTER=spark://spark-master:7077 

will produce the following master URL:

spark-master:7077

This URL is not valid and the job submission fails (the same happens with a mesos master).

I will create a PR on github that fixes this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)