You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Jarek Potiuk (Jira)" <ji...@apache.org> on 2020/03/03 22:57:00 UTC
[jira] [Commented] (AIRFLOW-6949) SparkSubitOperator ignores
explicit spark.kubernetes.namespace config option
[ https://issues.apache.org/jira/browse/AIRFLOW-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17050618#comment-17050618 ]
Jarek Potiuk commented on AIRFLOW-6949:
---------------------------------------
I removed the cherry-picked commit as it was failing with missing conn_data:
commit f8a83a7188c83113c69b9711b9a42e69d2fc6ba7 (polidea/v1-10-test, origin/v1-10-test)
Author: Ash Berlin-Taylor <as...@firemirror.com>
Date: Fri Feb 28 11:14:21 2020 +0000
[AIRFLOW-6949] Respect explicit `spark.kubernetes.namespace` conf to SparkSubmitOperator (#7575)
This means the value from the Operator/dag file takes precedence over
the connection
The previous behaviour was to emit one line from the conf arg, but then
a later one from the connection:
```
--conf spark.kubernetes.namespace=airflow \
--conf spark.kubernetes.namespace=default \
```
(cherry picked from commit b59042b5ab083c77ba08ba804df76b7c728815dc)
> SparkSubitOperator ignores explicit spark.kubernetes.namespace config option
> ----------------------------------------------------------------------------
>
> Key: AIRFLOW-6949
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6949
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
> Affects Versions: 1.10.9
> Reporter: Ash Berlin-Taylor
> Assignee: Ash Berlin-Taylor
> Priority: Minor
> Fix For: 1.10.10
>
>
> If a user explicitly passes {{spark.kubernetes.namespace}} in the config attribute, we should respect that over what is given in the connection.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)