You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/15 18:08:01 UTC
[GitHub] [airflow] mjpieters opened a new issue #11556: Anything touching airflow.utils.db triggers an import for kubernetes
mjpieters opened a new issue #11556:
URL: https://github.com/apache/airflow/issues/11556
**Apache Airflow version**: 2.0.0a1
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): No kubernetes used.
**Environment**:
- **OS** (e.g. from /etc/os-release): MacOS
**What happened**:
After updating my local dev environment configuration for Airflow 2.0.0a0 I wanted to see if the database needed updating:
```
$ airflow db check
Traceback (most recent call last):
File "/.../bin/airflow", line 8, in <module>
sys.exit(main())
File "/.../lib/python3.6/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/.../lib/python3.6/site-packages/airflow/cli/cli_parser.py", line 52, in command
func = import_string(import_path)
File "/.../lib/python3.6/site-packages/airflow/utils/module_loading.py", line 32, in import_string
module = import_module(module_path)
File "/.../lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/.../lib/python3.6/site-packages/airflow/cli/commands/db_command.py", line 24, in <module>
from airflow.utils import cli as cli_utils, db
File "/.../lib/python3.6/site-packages/airflow/utils/db.py", line 26, in <module>
from airflow.jobs.base_job import BaseJob # noqa: F401 # pylint: disable=unused-import
File "/.../lib/python3.6/site-packages/airflow/jobs/__init__.py", line 22, in <module>
import airflow.jobs.scheduler_job # noqa
File "/.../lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 50, in <module>
from airflow.models.serialized_dag import SerializedDagModel
File "/.../lib/python3.6/site-packages/airflow/models/serialized_dag.py", line 35, in <module>
from airflow.serialization.serialized_objects import SerializedDAG
File "/.../lib/python3.6/site-packages/airflow/serialization/serialized_objects.py", line 37, in <module>
from airflow.kubernetes.pod_generator import PodGenerator
File "/.../lib/python3.6/site-packages/airflow/kubernetes/pod_generator.py", line 35, in <module>
from kubernetes.client import models as k8s
ModuleNotFoundError: No module named 'kubernetes'
```
**What you expected to happen**:
`kubernetes` is an optional extra dependency, if it is not installed things should not break.
**How to reproduce it**:
This happens in a new setup, the following steps reproduce the issue:
```python
$ virtualenv -p python3.8 testenv
$ source testenv/bin/activate
$ pip install --find-links https://dist.apache.org/repos/dist/dev/airflow/2.0.0a1/ apache-airflow==2.0.0a1
$ airflow db check
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mjpieters commented on issue #11556: Anything touching airflow.utils.db triggers an import for kubernetes
Posted by GitBox <gi...@apache.org>.
mjpieters commented on issue #11556:
URL: https://github.com/apache/airflow/issues/11556#issuecomment-709510685
Looks like #10831 was meant to address this, but it missed out on the other import as well as the way `k8s` is actually used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mjpieters commented on issue #11556: Anything touching airflow.utils.db triggers an import for kubernetes
Posted by GitBox <gi...@apache.org>.
mjpieters commented on issue #11556:
URL: https://github.com/apache/airflow/issues/11556#issuecomment-709505389
I fear there are a few more issues in this area.
The problems lie primarily in two sections of `airflow/serialization/serialized_objects.py`:
https://github.com/apache/airflow/blob/4551a5e6d56ef4c141aa2ef0f7df20bbf2b09800/airflow/serialization/serialized_objects.py#L29-L37
This code guards against the import error when `kubernetes` is not installed, but a few lines down then imports `airflow.kubernetes.pod_generator`, which really can't avoid having to import `kubernetes`.
Further down, there are other issues that will break if the above `airflow.kubernetes.pod_generator` import is moved into the `ImportError` guard:
https://github.com/apache/airflow/blob/4551a5e6d56ef4c141aa2ef0f7df20bbf2b09800/airflow/serialization/serialized_objects.py#L190-L197
If `k8s` can be set to `None`, then referencing `k8s.V1Pod` will raise an `AttributeError`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb closed issue #11556: Anything touching airflow.utils.db triggers an import for kubernetes
Posted by GitBox <gi...@apache.org>.
ashb closed issue #11556:
URL: https://github.com/apache/airflow/issues/11556
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org