You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/25 07:11:31 UTC
[GitHub] [airflow] rino0601 opened a new issue, #25271: Version 2.3.3 breaks "Plugins as Python packages" feature
rino0601 opened a new issue, #25271:
URL: https://github.com/apache/airflow/issues/25271
### Apache Airflow version
2.3.3 (latest released)
### What happened
In 2.3.3
If I use https://airflow.apache.org/docs/apache-airflow/stable/plugins.html#plugins-as-python-packages feature, then I see these Error:
short:
`ValueError: The name 'airs' is already registered for this blueprint. Use 'name=' to provide a unique name.`
long:
> i'm trying to reproduce it...
If I don't use it(workarounding by AIRFLOW__CORE__PLUGINS_FOLDER), errors doesn't occur.
It didn't happend in 2.3.2 and earlier
### What you think should happen instead
Looks like plugins are import multiple times if it is plugins-as-python-packages.
Perhaps flask's major version change is the main cause.
Presumably, in flask 1.0, duplicate registration of blueprint was quietly filtered out, but in 2.0 it seems to have been changed to generate an error. (I am trying to find out if this hypothesis is correct)
Anyway, use the latest version of FAB is important. we will have to adapt to this change, so plugins will have to be imported once regardless how it defined.
### How to reproduce
> It was reproduced in the environment used at work, but it is difficult to disclose or explain it.
> I'm working to reproduce it with the breeze command, and I open the issue first with the belief that it's not just me.
### Operating System
CentOS Linux release 7.9.2009 (Core)
### Versions of Apache Airflow Providers
```sh
$ SHIV_INTERPRETER=1 airsflow -m pip freeze | grep apache-
apache-airflow==2.3.3
apache-airflow-providers-apache-hive==3.1.0
apache-airflow-providers-apache-spark==2.1.0
apache-airflow-providers-celery==3.0.0
apache-airflow-providers-common-sql==1.0.0
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-http==3.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-postgres==5.1.0
apache-airflow-providers-redis==3.0.0
apache-airflow-providers-sqlite==3.1.0
```
but I think these are irrelevant.
### Deployment
Other 3rd-party Helm chart
### Deployment details
docker image based on centos7, python 3.9.10 interpreter, self-written helm2 chart ....
... but I think these are irrelevant.
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #25271: Version 2.3.3 breaks "Plugins as Python packages" feature
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #25271:
URL: https://github.com/apache/airflow/issues/25271#issuecomment-1195760270
> as `pip install` seems to copy the package to both directories.
It’s probably not pip installing things to both directories; on some systems these two are symbolically linked (i.e. the same directory).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] VladimirYushkevich commented on issue #25271: Version 2.3.3 breaks "Plugins as Python packages" feature
Posted by "VladimirYushkevich (via GitHub)" <gi...@apache.org>.
VladimirYushkevich commented on issue #25271:
URL: https://github.com/apache/airflow/issues/25271#issuecomment-1542699877
I have very similar issue with duplicate plugins for `2.6.0`:
```
Python 3.9.13 (main, Sep 22 2022, 15:42:13)
[Clang 14.0.0 (clang-1400.0.29.102)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib_metadata
>>> for dist in importlib_metadata.distributions():
... for e in dist.entry_points:
... if e.group != "airflow.plugins":
... continue
... print(dist._path)
...
my_package.egg-info
/Users/my_user/path_to_my_project/my_package.egg-info
```
Any ideas why I have 2 entries in `importlib_metadata.distributions()` for `airflow.plugins` group (looks for me they are both pointing to same `egg-info`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] VladimirYushkevich commented on issue #25271: Version 2.3.3 breaks "Plugins as Python packages" feature
Posted by "VladimirYushkevich (via GitHub)" <gi...@apache.org>.
VladimirYushkevich commented on issue #25271:
URL: https://github.com/apache/airflow/issues/25271#issuecomment-1545942960
> I have very similar issue with duplicate plugins for 2.6.0:
Small update: Seems like above issue happens for me only during running `airflow standalone` (I also have virtual python environment). Maybe it is relates to: https://github.com/python/importlib_metadata/issues/410.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #25271: Version 2.3.3 breaks "Plugins as Python packages" feature
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25271:
URL: https://github.com/apache/airflow/issues/25271#issuecomment-1193761640
Yeah. It would be great to get to the bottom of this. Happy to help with investigation if you have some more findings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #25271: Version 2.3.3 breaks "Plugins as Python packages" feature
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #25271:
URL: https://github.com/apache/airflow/issues/25271#issuecomment-1193667770
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #25271: Version 2.3.3 breaks "Plugins as Python packages" feature
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #25271:
URL: https://github.com/apache/airflow/issues/25271#issuecomment-1194971825
The most likely cause is the plugin appears twice in `sys.path` and thus loaded multiple times by `importlib.metadata`. We can add some sort of registry that keeps track what distributions have been loaded (by distribution name) to avoid this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed issue #25271: Version 2.3.3 breaks "Plugins as Python packages" feature
Posted by GitBox <gi...@apache.org>.
uranusjr closed issue #25271: Version 2.3.3 breaks "Plugins as Python packages" feature
URL: https://github.com/apache/airflow/issues/25271
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bmoon4 commented on issue #25271: Version 2.3.3 breaks "Plugins as Python packages" feature
Posted by GitBox <gi...@apache.org>.
bmoon4 commented on issue #25271:
URL: https://github.com/apache/airflow/issues/25271#issuecomment-1194844825
In FAB 2.X, registering the same blueprint more than once is prohibited.
Problem is `entry_points_with_dist('airflow.plugins')` is returning the plugin twice.
https://github.com/apache/airflow/blob/2.3.3/airflow/plugins_manager.py#L224-L225
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rgmz commented on issue #25271: Version 2.3.3 breaks "Plugins as Python packages" feature
Posted by GitBox <gi...@apache.org>.
rgmz commented on issue #25271:
URL: https://github.com/apache/airflow/issues/25271#issuecomment-1195707123
> The most likely cause is the plugin appears twice in `sys.path` and thus loaded multiple times by `importlib.metadata`.
Indeed. One reason this can happen is if there are separate `lib` and `lib64` directories, as `pip install` seems to copy the package to both directories.
For example:
```
>>> import importlib_metadata
>>> for dist in importlib_metadata.distributions():
... for e in dist.entry_points:
... if e.group != "airflow.plugins":
... continue
... print(dist._path)
...
/opt/app-root/lib64/python3.9/site-packages/custom_plugin-1.3.16.dist-info
/opt/app-root/lib/python3.9/site-packages/custom_plugin-1.3.16.dist-info
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org