You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "nyoungstudios (via GitHub)" <gi...@apache.org> on 2023/04/04 00:39:56 UTC
[GitHub] [airflow] nyoungstudios opened a new issue, #30449: Nested plugins with the same name causes import error
nyoungstudios opened a new issue, #30449:
URL: https://github.com/apache/airflow/issues/30449
### Apache Airflow version
2.5.3
### What happened
## Overview
When a Python file in a submodule in the plugins folder has the same name as a Python file in the root of the plugins folder and another file in the submodule tries to import from the Python file in the root of the plugins folder, the dag that uses that submodule folder will have an import error.
Additionally, there is some different behavior (more explained in the "How to reproduce" section) when that Python file with the same name is specifically called `utils.py`.
## Screenshots
In the screenshots, I replaced the full path with the identifier: `$AIRFLOW_HOME` since my local filepath isn't important to the issue.
### Airflow 2.5.3
This is with the filename as `helpers.py`

This is with the filename as `utils.py`

### Airflow 2.3.4
In previous versions of Airflow (specifically 2.3.4), the dag still ran as expected.
This is with the filename as `helpers.py`. The dag ran as expected.

This is with the filename as `utils.py`. The dag ran as expected, but there is an error message in the UI still.

## Thoughts
I think the reason why name the filename as `utils.py` might cause a problem is because there is a module in the Airflow Python package called "utils" as well - although, I have not quite looked into this yet.
### What you think should happen instead
The dag should run as expected no matter what the filenames are in the plugins folder. And you should be able to have a filename in a submodule that is the same as a filename in the root of the plugins folder without running into errors.
### How to reproduce
## Setup
Create these files and folder structure.
### Folder structure:
```
dags/
-> hello_world.py
plugins/
-> fruits/
-> __init__.py
-> apple.py
-> helpers.py
-> helpers.py
```
### Contents of Files
```python
# dags/hello_world.py
from airflow.models import DAG
from airflow.operators.python import PythonOperator
from fruits.apple import apple
from datetime import datetime
with DAG(
dag_id="hello_world",
description="Hello world",
start_date=datetime(2023, 4, 1),
schedule_interval=None,
catchup=False
) as dag:
PythonOperator(
task_id="greeting",
python_callable=apple
)
```
```python
# plugins/fruits/apple.py
from helpers import greeting
def apple():
return greeting() + " apple"
```
```python
# plugins/fruits/helpers.py
def stuff():
return "asdf"
```
```python
# plugins/helpers.py
def greeting():
return "hi"
```
### Initialization Commands
Run these commands to initialize airflow
```bash
airflow db init
airflow users create --role Admin --username admin --email admin@local.com --firstname admin --lastname admin --password admin
```
I also set `load_examples = False` in the `airflow.cfg` file.
## Testing
### Airflow 2.5.3
Upon opening the webserver (after running `airflow webserver` and `airflow scheduler`), you are greeted with this error:
```python
Broken DAG: [$AIRFLOW_HOME/dags/hello_world.py] Traceback (most recent call last):
File "$AIRFLOW_HOME/dags/hello_world.py", line 4, in <module>
from fruits.apple import apple
File "$AIRFLOW_HOME/plugins/fruits/apple.py", line 1, in <module>
from helpers import greeting
ImportError: cannot import name 'greeting' from 'helpers' ($AIRFLOW_HOME/plugins/fruits/helpers.py)
```
And there will be no dags listed in the UI. However, if you run the `airflow dags list` command, it will no through an import error. Here is the output of that command:
```
dag_id | filepath | owner | paused
============+================+=========+=======
hello_world | hello_world.py | airflow | False
```
Similarly, the `airflow dags list-import-errors` command returns this:
```
No data found
```
If I make the dag fail by giving it a bad import (say `from fruits2.apple import apple` in `dags/hello_world.py`, the expected result shows the traceback.
Interestingly, if I change both `helpers.py` files to `utils.py`, then I also get this error message in the Airflow webserver UI
```python
Broken plugin: [$AIRFLOW_HOME/plugins/fruits/apple.py] cannot import name 'greeting' from 'utils' ($AIRFLOW_HOME/plugins/fruits/utils.py)
```
### Airflow 2.3.4
When rerunning this test on Airflow 2.3.4, the dag was able to run properly. When the filename is `helpers.py`, there are no error messages. However, when the filename is `utils.py`, this error shows up in the UI even though the dag can run properly:
```python
Broken plugin: [$AIRFLOW_HOME/plugins/fruits/apple.py] cannot import name 'greeting' from 'utils' ($AIRFLOW_HOME/plugins/fruits/utils.py)
```
And in both cases, the `airflow dags list` and `airflow dags list-import-errors` commands return as if there are no errors.
### Operating System
Ubuntu 20.04 LTS
### Versions of Apache Airflow Providers
apache-airflow-providers-common-sql==1.3.4
apache-airflow-providers-ftp==3.3.1
apache-airflow-providers-http==4.2.0
apache-airflow-providers-imap==3.1.1
apache-airflow-providers-sqlite==3.3.1
### Deployment
Virtualenv installation
### Deployment details
Actually, just a conda environment - Python 3.8.12. Airflow package installed with pip.
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #30449: Nested plugins with the same name causes import error
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #30449:
URL: https://github.com/apache/airflow/issues/30449#issuecomment-1500412043
> So the expected outcome of a pr that resolves this issue should throw a more meaningful error of why the plugin import statement is invalid?
>
> And curious, why is this the expected behavior? In a normal Python module, you are allowed to have a filename in a submodule that matches a filename at the root of the module.
Because plugins are imported in effecively "sandboxed" environment, completely separately from regular Python import (for example relative imports do not work in plugins).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #30449: Nested plugins with the same name causes import error
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #30449:
URL: https://github.com/apache/airflow/issues/30449#issuecomment-1500412283
(and the reason is security).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Nested plugins with the same name causes import error [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #30449:
URL: https://github.com/apache/airflow/issues/30449#issuecomment-3986500368
@nyoungstudios We are unassigning you from this issue as part of our updated [assignment policy](https://github.com/apache/airflow/blob/main/contributing-docs/04_how_to_contribute.rst#contribute-code-changes).
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #30449: Nested plugins with the same name causes import error
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #30449:
URL: https://github.com/apache/airflow/issues/30449#issuecomment-1495830566
Correct. this is expected behaviour - you need to avoid doing it. But if you have an idea how to improve diagnostics in this case - feel free.
You will have to likely see if there are more packages/modules importable and do some importlib trickery to achieve it. Assigned you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #30449: Nested plugins with the same name causes import error
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #30449:
URL: https://github.com/apache/airflow/issues/30449#issuecomment-1495181142
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nyoungstudios commented on issue #30449: Nested plugins with the same name causes import error
Posted by "nyoungstudios (via GitHub)" <gi...@apache.org>.
nyoungstudios commented on issue #30449:
URL: https://github.com/apache/airflow/issues/30449#issuecomment-1500385997
So the expected outcome of a pr that resolves this issue should throw a more meaningful error of why the plugin import statement is invalid?
And curious, why is this the expected behavior? In a normal Python module, you are allowed to have a filename in a submodule that matches a filename at the root of the module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org