[GitHub] [airflow] dskoda1 opened a new issue #22529: Airflow Worker not picking up "cmd" in config file

[GitBox <gi...@apache.org>]

dskoda1 opened a new issue #22529:
URL: https://github.com/apache/airflow/issues/22529


   ### Apache Airflow version
   
   2.2.3
   
   ### What happened
   
   We are leveraging the ability to fetch secrets in the cfg files at runtime, by adding the `_cmd` suffix to certain configs, such as `sql_alchemy_conn_cmd` and `broker_url_cmd`, as described in [the docs](https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html).
   
   This is working fine for the scheduler and webserver components - additionally, for the worker, all but the `sql_alchemy_conn_cmd` are working. Unless we set `AIRFLOW__CORE__SQL_ALCHEMY_CONN` as an environment variable, the worker will always try and connect to the (default) sqlite connection string for the `sql_alchemy_conn`.
   
   Sadly, if we run `airflow config list`, though we don't have it in our cfg file, `sql_alchemy_conn` is present and set to the default sqlite connection string.
   
   ### What you think should happen instead
   
   Airflow worker should behave the same as webserver and scheduler, and leverage `sql_alchemy_conn_cmd` given it is present in the cfg file, and not fall back to the default value for `sql_alchemy_conn`.
   
   ### How to reproduce
   
   In a config file, set `sql_alchemy_conn_cmd` to a legitimate db backend (i.e. not sqlite). Start webserver, scheduler, verify they are able to connect. Start a celery worker, and validate it starts up ok.
   
   Next, run any dag (we ran any of the example dags as long as they aren't dummy operators), "tutorial" dag is fine here.
   
   Click into a task that is ran and check logs, and see the following error:
   
   ```
   Traceback (most recent call last):
     File "/bb/bin/package/c/csi-airflow2/1.1.0/libexec/csi-airflow2/dependencies/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
       self.dialect.do_execute(
     File "/bb/bin/package/c/csi-airflow2/1.1.0/libexec/csi-airflow2/dependencies/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
       cursor.execute(statement, parameters)
   sqlite3.OperationalError: no such table: csi_airflow2_db.dag_run
   ```
   
   indicating it is trying to connect to sqlite rather than the configured DB backend. 
   
   ### Operating System
   
   RHEL 7.6
   
   ### Versions of Apache Airflow Providers
   
   ```
   apache-airflow-providers-celery==2.1.0
       # via apache-airflow
   apache-airflow-providers-docker==2.4.0
       # via -r requirements.in
   apache-airflow-providers-ftp==2.0.1
       # via apache-airflow
   apache-airflow-providers-http==2.0.2
       # via apache-airflow
   apache-airflow-providers-imap==2.1.0
       # via apache-airflow
   apache-airflow-providers-postgres==2.4.0
       # via apache-airflow
   apache-airflow-providers-sqlite==2.0.1
       # via apache-airflow
   ```
   
   ### Deployment
   
   Virtualenv installation
   
   ### Deployment details
   
   Deployed using internal company tooling, roughly a virtualenv / self contained application
   
   ### Anything else
   
   the celery worker is able to parse and use other configs which use `_cmd`, except for sql_alchemy_conn.
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #22529: Airflow Worker not picking up "cmd" in config file

[GitBox <gi...@apache.org>]

boring-cyborg[bot] commented on issue #22529:
URL: https://github.com/apache/airflow/issues/22529#issuecomment-1079097969


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #22529: Airflow Worker not picking up "cmd" in config file

[GitBox <gi...@apache.org>]

potiuk commented on issue #22529:
URL: https://github.com/apache/airflow/issues/22529#issuecomment-1080007133


   This is very unlikely to happen. There is no distinction vs. commands in Airflow. You must have a different configuration of your deployment tooling.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org