You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/04/30 09:01:16 UTC
[GitHub] [airflow] aalemanq opened a new issue #8643: Can't setup enviroments for use secrets in docker-swarm
aalemanq opened a new issue #8643:
URL: https://github.com/apache/airflow/issues/8643
Hello, I spend so many weeks trying to configure airflow using secrets in docker swarm.
I try to use this config in my compose:
environment:
- LOAD_EX=y
- FERNET_KEY_CMD=$$(cat /run/secrets/fernet_key) (If I put '$" swarm tells me that I need to escape with another $)
- EXECUTOR=Celery
- AIRFLOW__CELERY__BROKER_URL_CMD=$$(cat /run/secrets/broker_url)
- AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD=$$(cat /run/secrets/sql_alchemy_conn)
- AIRFLOW__CELERY__RESULT_BACKEND_CMD=$$(cat /run/secrets/result_backend)
I tried with another bash commands to do cat to this secret and all fails, it never gets enviroments and use redis default.
I tried to follow oficial doc but...
https://airflow.readthedocs.io/en/stable/howto/set-config.html
`The _cmd config options can also be set using a corresponding environment variable the same way the usual config options can. For example:
export AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD=bash_command_to_run`
Really is so difficult touse enviroments pointing to file like another software?
enviroments:
- AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD=$(cat /run/blabla) ?¿?
A little debug:
Thanks for reply. I tried to apply env_file but same issue. When I use enviroments like:
```
LOAD_EX=y
FERNET_KEY=XXXX
EXECUTOR=Celery
AIRFLOW__CELERY__BROKER_URL=pyamqp://airflow:airflow@rabbitmq:5672/airflow
AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__RESULT_BACKEND=db+postgresql://airflow:airflow@postgres/airflow
```
It's works
but if I apply secrets using this in env_file, is not work :( :
```
AIRFLOW__CORE__FERNET_KEY_CMD=$(cat /run/secrets/fernet_key)
AIRFLOW__CELERY__BROKER_URL_CMD=$(cat /run/secrets/broker_url)
AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD=$(cat /run/secrets/sql_alchemy_conn)
AIRFLOW__CELERY__RESULT_BACKEND_CMD=$(cat /run/secrets/result_backend)
```
Airflow never get broker_url and rabbitmq conection is replaced by redis...
I check if everything is right and I don't see any error about secrets/chains:
Some debug:
I deploy stack and enter in worker airflow container to check if secrets exists:
```
airflow@03eb98bd469d:~$ cat /run/secrets/sql_alchemy_conn
postgresql+psycopg2://airflow:airflow@postgres/airflow
airflow@03eb98bd469d:~$ cat /run/secrets/broker_url
pyamqp://airflow:airflow@rabbitmq:5672/airflow
airflow@03eb98bd469d:~$ cat /run/secrets/result_backend
db+postgresql://airflow:airflow@postgres/airflow
```
I execute by hand this enviroments:
```
airflow@03eb98bd469d:~$ LOAD_EX=y
airflow@03eb98bd469d:~$ AIRFLOW__CORE__FERNET_KEY_CMD=$(cat /run/secrets/fernet_key)
airflow@03eb98bd469d:~$ EXECUTOR=Celery
airflow@03eb98bd469d:~$ AIRFLOW__CELERY__BROKER_URL_CMD=$(cat /run/secrets/broker_url)
airflow@03eb98bd469d:~$ AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD=$(cat /run/secrets/sql_alchemy_conn)
airflow@03eb98bd469d:~$ AIRFLOW__CELERY__RESULT_BACKEND_CMD=$(cat /run/secrets/result_backend)
```
And echo is right:
```
airflow@03eb98bd469d:~$ echo $AIRFLOW__CORE__FERNET_KEY_CMD
46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
airflow@03eb98bd469d:~$ echo $AIRFLOW__CELERY__BROKER_URL_CMD
pyamqp://airflow:airflow@rabbitmq:5672/airflow
airflow@03eb98bd469d:~$ echo $AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD
postgresql+psycopg2://airflow:airflow@postgres/airflow
airflow@03eb98bd469d:~$ echo $AIRFLOW__CELERY__RESULT_BACKEND_CMD
db+postgresql://airflow:airflow@postgres/airflow
```
I don't know what I can do, secrets testes, enviroments testesd, no aiflow running with enviroments and secrets but without secrets it works as exepted...
Regards
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #8643: Can't setup enviroments for use secrets in docker-swarm
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #8643:
URL: https://github.com/apache/airflow/issues/8643
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #8643: Can't setup enviroments for use secrets in docker-swarm
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #8643:
URL: https://github.com/apache/airflow/issues/8643#issuecomment-621708157
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org