You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/06 17:57:36 UTC
[GitHub] [airflow] raphaelauv opened a new issue, #24259: cli airflow db without dags and plugings
raphaelauv opened a new issue, #24259:
URL: https://github.com/apache/airflow/issues/24259
### Description
Currently If in my conf I have a xcom_backend or/and a secret_backend I can't run the cli `airflow db` without giving access to the dags and plugins
```yaml
version: '3.8'
services:
airflow_init_db:
image: airflow_v2
env_file:
- ./conf/airflow.env
volumes:
# Link for custom_xcom_backend and custom_secret_backend
- $AIRFLOW_DAGS_SRC_PATH/utils/backends/:/usr/local/airflow/dags/utils/backends/
command: airflow db upgrade
```
without the volume mapping it fail with
```log
[2022-06-06 17:50:03,697] {configuration.py:484} ERROR - No module named 'utils'
2022-06-06T17:50:03.705575364Z Traceback (most recent call last):
2022-06-06T17:50:03.705607597Z File "/usr/local/lib/python3.8/site-packages/airflow/configuration.py", line 482, in getimport
2022-06-06T17:50:03.705616559Z return import_string(full_qualified_path)
2022-06-06T17:50:03.705622858Z File "/usr/local/lib/python3.8/site-packages/airflow/utils/module_loading.py", line 32, in import_string
2022-06-06T17:50:03.705628878Z module = import_module(module_path)
2022-06-06T17:50:03.705634245Z File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
2022-06-06T17:50:03.705640056Z return _bootstrap._gcd_import(name[level:], package, level)
2022-06-06T17:50:03.705645481Z File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
2022-06-06T17:50:03.705651321Z File "<frozen importlib._bootstrap>", line 991, in _find_and_load
2022-06-06T17:50:03.705657152Z File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
2022-06-06T17:50:03.705662652Z File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2022-06-06T17:50:03.705668816Z File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
2022-06-06T17:50:03.705674535Z File "<frozen importlib._bootstrap>", line 991, in _find_and_load
2022-06-06T17:50:03.705680730Z File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
2022-06-06T17:50:03.705686719Z File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2022-06-06T17:50:03.705692431Z File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
2022-06-06T17:50:03.705697930Z File "<frozen importlib._bootstrap>", line 991, in _find_and_load
2022-06-06T17:50:03.705703442Z File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
2022-06-06T17:50:03.705708839Z ModuleNotFoundError: No module named 'utils'
2022-06-06T17:50:03.705713891Z
2022-06-06T17:50:03.705718604Z During handling of the above exception, another exception occurred:
2022-06-06T17:50:03.705723909Z
2022-06-06T17:50:03.705728911Z Traceback (most recent call last):
2022-06-06T17:50:03.705734002Z File "/usr/local/bin/airflow", line 8, in <module>
2022-06-06T17:50:03.705739747Z sys.exit(main())
2022-06-06T17:50:03.705744848Z File "/usr/local/lib/python3.8/site-packages/airflow/__main__.py", line 48, in main
2022-06-06T17:50:03.705750643Z args.func(args)
2022-06-06T17:50:03.705755795Z File "/usr/local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 47, in command
2022-06-06T17:50:03.705787110Z func = import_string(import_path)
2022-06-06T17:50:03.705795008Z File "/usr/local/lib/python3.8/site-packages/airflow/utils/module_loading.py", line 32, in import_string
2022-06-06T17:50:03.705801328Z module = import_module(module_path)
2022-06-06T17:50:03.705806591Z File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
2022-06-06T17:50:03.705812365Z return _bootstrap._gcd_import(name[level:], package, level)
2022-06-06T17:50:03.705819554Z File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
2022-06-06T17:50:03.705825591Z File "<frozen importlib._bootstrap>", line 991, in _find_and_load
2022-06-06T17:50:03.705831388Z File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
2022-06-06T17:50:03.705836771Z File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
2022-06-06T17:50:03.705842707Z File "<frozen importlib._bootstrap_external>", line 783, in exec_module
2022-06-06T17:50:03.705848644Z File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2022-06-06T17:50:03.705854751Z File "/usr/local/lib/python3.8/site-packages/airflow/cli/commands/db_command.py", line 24, in <module>
2022-06-06T17:50:03.705861189Z from airflow.utils import cli as cli_utils, db
2022-06-06T17:50:03.705866663Z File "/usr/local/lib/python3.8/site-packages/airflow/utils/db.py", line 28, in <module>
2022-06-06T17:50:03.705872575Z from airflow.jobs.base_job import BaseJob # noqa: F401
2022-06-06T17:50:03.705878078Z File "/usr/local/lib/python3.8/site-packages/airflow/jobs/__init__.py", line 19, in <module>
2022-06-06T17:50:03.705883975Z import airflow.jobs.backfill_job
2022-06-06T17:50:03.705889540Z File "/usr/local/lib/python3.8/site-packages/airflow/jobs/backfill_job.py", line 28, in <module>
2022-06-06T17:50:03.705895379Z from airflow import models
2022-06-06T17:50:03.705900376Z File "/usr/local/lib/python3.8/site-packages/airflow/models/__init__.py", line 20, in <module>
2022-06-06T17:50:03.705906053Z from airflow.models.baseoperator import BaseOperator, BaseOperatorLink
2022-06-06T17:50:03.705911136Z File "/usr/local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 61, in <module>
2022-06-06T17:50:03.705916510Z from airflow.models.taskinstance import Context, TaskInstance, clear_task_instances
2022-06-06T17:50:03.705921856Z File "/usr/local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 81, in <module>
2022-06-06T17:50:03.705927088Z from airflow.models.xcom import XCOM_RETURN_KEY, XCom
2022-06-06T17:50:03.705932440Z File "/usr/local/lib/python3.8/site-packages/airflow/models/xcom.py", line 496, in <module>
2022-06-06T17:50:03.705938338Z XCom = resolve_xcom_backend()
2022-06-06T17:50:03.705959642Z File "/usr/local/lib/python3.8/site-packages/airflow/models/xcom.py", line 483, in resolve_xcom_backend
2022-06-06T17:50:03.705966771Z clazz = conf.getimport("core", "xcom_backend", fallback=f"airflow.models.xcom.{BaseXCom.__name__}")
2022-06-06T17:50:03.705972908Z File "/usr/local/lib/python3.8/site-packages/airflow/configuration.py", line 485, in getimport
2022-06-06T17:50:03.705978934Z raise AirflowConfigException(
2022-06-06T17:50:03.705985412Z airflow.exceptions.AirflowConfigException: The object could not be loaded. Please check "xcom_backend" key in "core" section. Current value: "utils.backends.custom_xcom_backend.XComBackend".
```
### Use case/motivation
It would allow running the cli `airflow db ...` without access to dags or plugins
( my main motivation is that I'm using gcp composer and the db update K8S job do not sync the dags and plugins ( from gcs ) before running the cmd `airflow db update` . So I can't update my composer env )
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24259: cli airflow db without dags and plugings
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24259:
URL: https://github.com/apache/airflow/issues/24259#issuecomment-1147773781
You can override the configuration that currently causes the plugins to be loaded to not do it. AIRFLOW__CORE__..... speciflically to run the CLI.
Would that solve the problem @raphaelauv ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org