You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/14 14:30:52 UTC
[GitHub] [airflow] RikHeijdens opened a new issue #13063: Importing entry_point plugins fails on Airflow v1.10.14 and Python 3.8
RikHeijdens opened a new issue #13063:
URL: https://github.com/apache/airflow/issues/13063
**Apache Airflow version**: 1.10.14
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): N/A
**Environment**:
- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release): Debian GNU/Linux 10 (buster)
- **Kernel** (e.g. `uname -a`): Linux 3f9779a91397 5.4.0-52-generic #57-Ubuntu SMP Thu Oct 15 10:57:00 UTC 2020 x86_64 GNU/Linux
- **Others**: Python 3.8
**What happened**:
On Airflow 1.10.14 and Python 3.8 importing plugins that have been provided through a [setuptools entrypoint](https://airflow.apache.org/docs/apache-airflow/stable/plugins.html#plugins-as-python-packages) fails with the following error:
```
[2020-12-14 13:35:12,200] {{plugins_manager.py:159}} ERROR - Failed to import plugin gantry
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/airflow/plugins_manager.py", line 150, in load_entrypoint_plugins
plugin_obj.__usable_import_name = entry_point.module
AttributeError: 'EntryPoint' object has no attribute 'module'
/usr/local/lib/python3.8/site-packages/gantry_plugin/plugin.py:11: FutureWarning: Registering operators or sensors in plugins is deprecated -- these should be treated like 'plain' python modules, and imported normally in DAGs.
Airflow 2.0 has removed the ability to register these types in plugins. See <http://airflow.apache.org/docs/stable/howto/custom-operator.html>.
class GantryPlugin(AirflowPlugin):
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/airflow/plugins_manager.py", line 150, in load_entrypoint_plugins
plugin_obj.__usable_import_name = entry_point.module
AttributeError: 'EntryPoint' object has no attribute 'module'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 25, in <module>
from airflow.configuration import conf
File "/usr/local/lib/python3.8/site-packages/airflow/__init__.py", line 100, in <module>
operators._integrate_plugins()
File "/usr/local/lib/python3.8/site-packages/airflow/operators/__init__.py", line 107, in _integrate_plugins
from airflow.plugins_manager import operators_modules
File "/usr/local/lib/python3.8/site-packages/airflow/plugins_manager.py", line 229, in <module>
plugins = load_entrypoint_plugins(
File "/usr/local/lib/python3.8/site-packages/airflow/plugins_manager.py", line 160, in load_entrypoint_plugins
import_errors[entry_point.module] = str(e)
AttributeError: 'EntryPoint' object has no attribute 'module'
```
This plugin can be imported without issues on Airflow v1.10.12 and v1.10.13.
**What you expected to happen**:
I had expected Airflow to import the plugin without issues.
**How to reproduce it**:
I believe one should be able to reproduce this by creating a plugin that uses setuptools' `entry_points` mechanism as described [here](https://airflow.apache.org/docs/apache-airflow/stable/plugins.html#plugins-as-python-packages).
**Anything else we need to know**:
I believe this bug may be related to the changes introduced in https://github.com/apache/airflow/commit/b49838fac8b04c32b4aeb211de858e7897d5b3e5.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #13063: Importing entry_point plugins fails on Airflow v1.10.14 and Python 3.8
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13063:
URL: https://github.com/apache/airflow/issues/13063#issuecomment-744519082
Yes. This is a known problem, once that made us think that we might need 1.10.15 but we want to see how big of a problem it is, since there is a known workaround: https://github.com/apache/airflow/issues/13019:
```
pip install importlib-metadata~=2.0
```
Can you please let us know if it fixes your problem @RikHeijdens ? (CC: @kaxil @ashb)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #13063: Importing entry_point plugins fails on Airflow v1.10.14 and Python 3.8
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #13063:
URL: https://github.com/apache/airflow/issues/13063#issuecomment-744611311
This was a pure mistake that we only noticed after the release, It is already fixed in v1-10-test branch - so in case we release 1.10.15. See https://github.com/apache/airflow/pull/12859 for the history of it and
https://github.com/apache/airflow/commit/e75deee11ab8ed626979f1fe3927049a200ab676 for the commit fixing it merged to v1-10-test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RikHeijdens edited a comment on issue #13063: Importing entry_point plugins fails on Airflow v1.10.14 and Python 3.8
Posted by GitBox <gi...@apache.org>.
RikHeijdens edited a comment on issue #13063:
URL: https://github.com/apache/airflow/issues/13063#issuecomment-744598980
I can indeed confirm that this issue does not occur when `importlib-metadata` is installed. One thing that is not quite clear to me though is why this package did not get installed as part of one of Airflow's dependencies when running on Python 3.8. I probably wouldn't have stumbled into this issue if that had been the case.
i.e. this statement here should have been updated: https://github.com/apache/airflow/blob/1.10.14/setup.py#L624, similar to L442: https://github.com/apache/airflow/blob/1.10.14/setup.py#L442
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #13063: Importing entry_point plugins fails on Airflow v1.10.14 and Python 3.8
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #13063:
URL: https://github.com/apache/airflow/issues/13063
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RikHeijdens commented on issue #13063: Importing entry_point plugins fails on Airflow v1.10.14 and Python 3.8
Posted by GitBox <gi...@apache.org>.
RikHeijdens commented on issue #13063:
URL: https://github.com/apache/airflow/issues/13063#issuecomment-744664901
Ah, excellent, I hadn't noticed that the issue was fixed on the `v1-10-test` branch. I had only looked at the `v1-10-stable` branch before opening this issue. I'll make sure to check that branch before opening an issue the next time.
Either way, thanks for the quick response to this bug report :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #13063: Importing entry_point plugins fails on Airflow v1.10.14 and Python 3.8
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #13063:
URL: https://github.com/apache/airflow/issues/13063#issuecomment-744611311
This was a pure mistake that we only noticed after the release, It is already fixed in v1-10-test branch - so in case we release 1.10.15. See https://github.com/apache/airflow/pull/12859 for the history of it and https://github.com/apache/airflow/commit/e75deee11ab8ed626979f1fe3927049a200ab676 for the commit fixing it merged to v1-10-test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #13063: Importing entry_point plugins fails on Airflow v1.10.14 and Python 3.8
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13063:
URL: https://github.com/apache/airflow/issues/13063#issuecomment-744703207
Be sure to watch the devlist. It might well be that 1.10.15 will be released at some point in time :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13063: Importing entry_point plugins fails on Airflow v1.10.14 and Python 3.8
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13063:
URL: https://github.com/apache/airflow/issues/13063#issuecomment-744637470
> I can indeed confirm that this issue does not occur when `importlib-metadata` is installed. One thing that is not quite clear to me though is why this package did not get installed as part of one of Airflow's dependencies by pip when running on Python 3.8. I probably wouldn't have stumbled into this issue if that had been the case.
>
> i.e. this statement here should have been updated: https://github.com/apache/airflow/blob/1.10.14/setup.py#L624, similar to L442: https://github.com/apache/airflow/blob/1.10.14/setup.py#L442
Yeat that fix I included only updated importlib-metadata in `devel` requirements. Hence it updated breeze and CI where I tested that changed. However, I missed the entry in install_requires, hence it is not installed for normal users who do not install `devel`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RikHeijdens edited a comment on issue #13063: Importing entry_point plugins fails on Airflow v1.10.14 and Python 3.8
Posted by GitBox <gi...@apache.org>.
RikHeijdens edited a comment on issue #13063:
URL: https://github.com/apache/airflow/issues/13063#issuecomment-744598980
I can indeed confirm that this issue does not occur when `importlib-metadata` is installed. One thing that is not quite clear to me though is why this package did not get installed as part of one of Airflow's dependencies by pip when running on Python 3.8. I probably wouldn't have stumbled into this issue if that had been the case.
i.e. this statement here should have been updated: https://github.com/apache/airflow/blob/1.10.14/setup.py#L624, similar to L442: https://github.com/apache/airflow/blob/1.10.14/setup.py#L442
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #13063: Importing entry_point plugins fails on Airflow v1.10.14 and Python 3.8
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13063:
URL: https://github.com/apache/airflow/issues/13063#issuecomment-744611311
This was a pure mistake that we only noticed after the release, It is already fixed in v1-10-test branch - so in case we release 1.10.15. See https://github.com/apache/airflow/pull/12859.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RikHeijdens commented on issue #13063: Importing entry_point plugins fails on Airflow v1.10.14 and Python 3.8
Posted by GitBox <gi...@apache.org>.
RikHeijdens commented on issue #13063:
URL: https://github.com/apache/airflow/issues/13063#issuecomment-744598980
I can indeed confirm that this issue does not occur when `importlib-metadata` is installed. One thing that is not quite clear to me though is why this package did not get installed as part of one of Airflow's dependencies when running on Python 3.8. I probably wouldn't have stumbled into this issue if that had been the case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #13063: Importing entry_point plugins fails on Airflow v1.10.14 and Python 3.8
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #13063:
URL: https://github.com/apache/airflow/issues/13063#issuecomment-744557230
I think this is also only a problem when the plugin fails to import - i.e. while annoying it only affects Dev. (That is my understanding anyway)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org