You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Luis Angel Vicente Sanchez (JIRA)" <ji...@apache.org> on 2017/07/13 11:44:00 UTC
[jira] [Updated] (AIRFLOW-1409) SparkSubmitHook doesn't generate a
valid master URL
[ https://issues.apache.org/jira/browse/AIRFLOW-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luis Angel Vicente Sanchez updated AIRFLOW-1409:
------------------------------------------------
Description:
If you configure the connection using an ENV variable (and possibly through the UI), SparkSubmitHook is ignoring the scheme and doesn't produce valid master URL. Example, this ENV variable,
AIRFLOW_CONN_SPARK_CLUSTER=spark://spark-master:7077
will produce the following master URL:
spark-master:7077
This URL is not valid and the job submission fails. The issue is how the url is parsed to create a connection, I have a partial fix working and I will submit a PR to github soon.
was:
If you configure the connection using an ENV variable (and possibly through the UI), SparkSubmitHook is ignoring the scheme and doesn't produce valid master URL. Example, this ENV variable,
AIRFLOW_CONN_SPARK_CLUSTER=spark://spark-master:7077
will produce the following master URL:
spark-master:7077
This URL is not valid and the job submission fails (the same happens with a mesos master).
I will create a PR on github that fixes this.
> SparkSubmitHook doesn't generate a valid master URL
> ---------------------------------------------------
>
> Key: AIRFLOW-1409
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1409
> Project: Apache Airflow
> Issue Type: Bug
> Reporter: Luis Angel Vicente Sanchez
>
> If you configure the connection using an ENV variable (and possibly through the UI), SparkSubmitHook is ignoring the scheme and doesn't produce valid master URL. Example, this ENV variable,
> AIRFLOW_CONN_SPARK_CLUSTER=spark://spark-master:7077
> will produce the following master URL:
> spark-master:7077
> This URL is not valid and the job submission fails. The issue is how the url is parsed to create a connection, I have a partial fix working and I will submit a PR to github soon.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)