You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/12 09:06:29 UTC
[GitHub] [airflow] ecerulm opened a new issue #15325: Custom XCom backends circular import when using cli command like airflow connections list
ecerulm opened a new issue #15325:
URL: https://github.com/apache/airflow/issues/15325
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
<!--
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.
-->
**Apache Airflow version**: 2.0.1
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
**Environment**:
- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:
**What happened**:
While running `airflow connections list` with a `xcom_backend = xcom.MyXComBackend` in `airflow.cfg` I will get a
ImportError: cannot import name 'BaseHook' from partially initialized module 'airflow.hooks.base' (most likely due to a circular import) (/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/hooks/base.py)
(see below for complete traceback).
The issue seems to be that `airflow/cli/commands/connection_command.py` imports `BaseHook` directly and `BaseHook -> Connection -> BaseOperator -> TaskInstance -> XCom -> MyXComBackend -> S3Hook -> BaseHook ` (again see complete traceback below)
```
airflow connections list
[2021-04-12 10:51:34,020] {configuration.py:459} ERROR - cannot import name 'BaseHook' from partially initialized module 'airflow.hooks.base' (most likely due to a circular import) (/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/hooks/base.py)
Traceback (most recent call last):
File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/configuration.py", line 457, in getimport
return import_string(full_qualified_path)
File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/utils/module_loading.py", line 32, in import_string
module = import_module(module_path)
File "/Users/rubelagu/.pyenv/versions/3.8.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/Users/rubelagu/tmp/airflow-xcom-backend/plugins/xcom.py", line 4, in <module>
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/providers/amazon/aws/hooks/s3.py", line 37, in <module>
from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py", line 41, in <module>
from airflow.hooks.base import BaseHook
ImportError: cannot import name 'BaseHook' from partially initialized module 'airflow.hooks.base' (most likely due to a circular import) (/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/hooks/base.py)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/bin/airflow", line 8, in <module>
sys.exit(main())
File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 47, in command
func = import_string(import_path)
File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/utils/module_loading.py", line 32, in import_string
module = import_module(module_path)
File "/Users/rubelagu/.pyenv/versions/3.8.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/cli/commands/connection_command.py", line 30, in <module>
from airflow.hooks.base import BaseHook
File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/hooks/base.py", line 23, in <module>
from airflow.models.connection import Connection
File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/models/__init__.py", line 20, in <module>
from airflow.models.baseoperator import BaseOperator, BaseOperatorLink
File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 55, in <module>
from airflow.models.taskinstance import Context, TaskInstance, clear_task_instances
File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 58, in <module>
from airflow.models.xcom import XCOM_RETURN_KEY, XCom
File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/models/xcom.py", line 289, in <module>
XCom = resolve_xcom_backend()
File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/models/xcom.py", line 279, in resolve_xcom_backend
clazz = conf.getimport("core", "xcom_backend", fallback=f"airflow.models.xcom.{BaseXCom.__name__}")
File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/configuration.py", line 460, in getimport
raise AirflowConfigException(
airflow.exceptions.AirflowConfigException: The object could not be loaded. Please check "xcom_backend" key in "core" section. Current value: "xcom.MyXComBackend".
```
**What you expected to happen**:
I expected to be able to use S3Hook or GCSHook from a cusom XCom backedn following the example in what I though was the canonical example at https://medium.com/apache-airflow/airflow-2-0-dag-authoring-redesigned-651edc397178
**How to reproduce it**:
* In `airflow.cfg` set `xcom_backend = xcom.MyXComBackend`
* Create file `plugins/xcom.py` with the following contents
```
from airflow.models.xcom import BaseXCom
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
class MyXComBackend(BaseXCom):
pass
```
**Anything else we need to know**:
This can be workaround by not importing the hooks on the module level but at the method level, but that is
* really ugly
* not evident for anybody trying to create a custom xcom backend for the first time.
```
from airflow.models.xcom import BaseXCom
class MyXComBackend(BaseXCom):
@staticmethod
def serialize_value(value):
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
hook = S3Hook()
pass
@staticmethod
def deserialize_value(value):
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
hook = S3Hook()
pass
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ecerulm commented on issue #15325: Custom XCom backends circular import when using cli command like airflow connections list
Posted by GitBox <gi...@apache.org>.
ecerulm commented on issue #15325:
URL: https://github.com/apache/airflow/issues/15325#issuecomment-817881307
Well, it turns out it does not work `airflow.models.xcom.XCom` is used in `datamodel = AirflowModelView.CustomSQLAInterface(XCom)` for example that breaks if XCom is a Proxy object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #15325: Custom XCom backends circular import when using cli command like airflow connections list
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15325:
URL: https://github.com/apache/airflow/issues/15325#issuecomment-819169076
Uh, so a straight-up proxy class doesn’t work since `XCom` is also a SQLAlchemy model, and a more significant is needed. The strategy can still be simple (Django’s example still applies; they are also dealing with an ORM class there), it’s the refactoring (and potential backward compatibility issues) needed behind it that’s significant. That’s *probably* still worthwhile in the long run, but will need more discussion.
In the meantime, I’ll submit the “easy” fix to untangle the circular import.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ecerulm commented on issue #15325: Custom XCom backends circular import when using cli command like airflow connections list
Posted by GitBox <gi...@apache.org>.
ecerulm commented on issue #15325:
URL: https://github.com/apache/airflow/issues/15325#issuecomment-817869309
If airflow was Python 3.7+ instead of Python 3.6+ we could change `airflow/models/xcom.py` so that instead of resolving the XCom backend right away it would be delayed until first use.
So instead of:
```
XCom = resolve_xcom_backend()
```
```
def __getattr__(name):
if name == "XCom":
xcom = resolve_xcom_backend()
sys.module[__name__].XCom = xcom
return xcom
raise AttributeError(f"module {__name__} has no attribute {name}")
```
I'm going to try to use
```
XCom = lazy_object_proxy.Proxy(resolve_xcom_backend)
```
if it works I'll create a PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #15325: Custom XCom backends circular import when using cli command like airflow connections list
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15325:
URL: https://github.com/apache/airflow/issues/15325#issuecomment-817793534
The “ultimate” fix IMO would be to make the XCom class a proxy that’s lazily loaded instead (similar to [how Django loads the user class](https://docs.djangoproject.com/en/3.2/topics/auth/customizing/#django.contrib.auth.get_user_model)). But for a simple workaround, `airflow.hooks.base` only reference `Connection` once in `BaseHook.get_connection()`, so that’s a good place to break the cycle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #15325: Custom XCom backends circular import when using cli command like airflow connections list
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #15325:
URL: https://github.com/apache/airflow/issues/15325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #15325: Custom XCom backends circular import when using cli command like airflow connections list
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15325:
URL: https://github.com/apache/airflow/issues/15325#issuecomment-817950237
I was thinking more about proxying the class itself, something like
```python
class _XComMeta(type):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._impl = resolve_xcom_backend()
def __getattr__(self, name):
return getattr(self._impl, name)
class XCom(BaseXCom, metaclass=_XComMeta):
def __new__(cls, *args, **kwargs):
return cls._impl(*args, **kwargs)
```
There are other things that need consideration such as `__issubclass__` and `__isinstance__`, but it should be possible with the right tweaks to the code base. Whether that’s too much work, however…
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org