[jira] [Comment Edited] (AIRFLOW-2917) Set AIRFLOW__CORE__SQL_ALCHEMY_CONN only when needed for k8s executor

["Dimitris Theodorou (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16877788#comment-16877788 ] 

Dimitris Theodorou edited comment on AIRFLOW-2917 at 7/3/19 12:12 PM:
----------------------------------------------------------------------

This issue is resolved in a bogus way. It assumes that if a configmap is set, then that configmap contains the sql_alchemy_conn does not pick up the connection string from the executor's configuration. So when a configmap is specified without a connection string, the workers end up trying to use the bogus default sqlite connection.

The problem is that it is almost always a bad practice to store secrets in a configmap in the context of Kubernetes secret management. The connection string itself is a secret and should be passed as a secret, external to the configmap. So I consider the use-case of supporting passing the connection string from the configmap of much less importance than supporting the use-case of passing it from the executor.

I think this use case should still be supported, but the configmap should be explicitly checked that it contains a connection string, and if not fall back to using the executor's


was (Author: dtheodor):
This issue is resolved in a bogus way. It assumes that if a configmap is set, then that configmap contains the `AIRFLOW__CORE____SQL_ALCHEMY_CONN` and does not pick up the connection string from the executor's configuration. So when a configmap is specified without a connection string, the workers end up trying to use the bogus default sqlite connection.

The problem is that it is almost always a bad practice to store secrets in a configmap in the context of Kubernetes secret management. The connection string itself is a secret and should be passed as a secret, external to the configmap. So I consider the use-case of supporting passing the connection string from the configmap of much less importance than supporting the use-case of passing it from the executor.

I think this use case should still be supported, but the configmap should be explicitly checked that it contains a connection string, and if not fall back to using the executor's

> Set AIRFLOW__CORE__SQL_ALCHEMY_CONN only when needed for k8s executor
> ---------------------------------------------------------------------
>
>                 Key: AIRFLOW-2917
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2917
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: executors
>    Affects Versions: 1.10.0
>            Reporter: John Cheng
>            Assignee: John Cheng
>            Priority: Minor
>             Fix For: 1.10.2
>
>
> In Kubernetes executor, `AIRFLOW__CORE__SQL_ALCHEMY_CONN` is set as an environment variable even when it is specified in configmap or secrets.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)