You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/09/30 08:36:05 UTC
[GitHub] [airflow] vavasilev opened a new issue #18629: No venv isolation between zipped workflows
vavasilev opened a new issue #18629:
URL: https://github.com/apache/airflow/issues/18629
### Apache Airflow version
2.0.1
### Operating System
ubunto 20.04
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### What happened
I am using zipped workflows so that I can have code-level isolation between them. For example I would like to use different versions of python code in different workflows.
When I upload a workflow with newer version of the code the airflow-web has no issues, but when I try to backfill in the scheduler, it tries to load the python libraries from the other workflow.
Example: I have 2 workflows: xxx_workflow_1_0 (the old one) and xxx_stockout_correlation_workflow_1_0 (the new one). The new workflow uses slightly different version of one library.
The web UI renders successfully both workflow. However when I try to backfill the new workflow with the following command:
airflow dags backfill --start-date 2021-03-03 --end-date 2021-03-03 --conf '{"execId": "stock_correlation_20210303"}' xxx_stockout_correlation_workflow_1_0'
I receive the following error:
ERROR - Failed to import: /opt/airflow/dags/xxx_stockout_correlation_1_0.zip
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dagbag.py", line 345, in _load_modules_from_zip
current_module = importlib.import_module(mod_name)
File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
File "<frozen zipimport>", line 259, in load_module
File "/opt/airflow/dags/xxx_stockout_correlation_1_0.zip/xxx_stockout_correlation_1_0.py", line 3, in <module>
from a4e_retail.a4e.retail.dags import retail_stockout_correlation_dag
ImportError: cannot import name 'retail_stockout_correlation_dag' from 'a4e_retail.a4e.retail.dags' (/opt/airflow/dags/xxx_1_0.zip/a4e_retail/a4e/retail/dags/__init__.py)
You can see it attempts to import from the wrong zip - xxx_1_0.zip. It should import from xxx_stockout_correlation_1_0.zip
When I delete the old workflow everything works properly with the new one
### What you expected to happen
Both workflows should coexist
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #18629: No venv isolation between zipped workflows
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #18629:
URL: https://github.com/apache/airflow/issues/18629
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18629: No venv isolation between zipped workflows
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18629:
URL: https://github.com/apache/airflow/issues/18629#issuecomment-931136791
For now the workaround will be to name different copy of the modules differently in the two zip files. I do not think we have a "guarantee" today that zipped workflows will be isolated from each other.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18629: No venv isolation between zipped workflows
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18629:
URL: https://github.com/apache/airflow/issues/18629#issuecomment-931134178
Interesting interaction I think that one might not be super-easy to fix and might require some more changes in the underlying way how backfill is done. Might be related with #18428 and #18317
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #18629: No venv isolation between zipped workflows
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #18629:
URL: https://github.com/apache/airflow/issues/18629#issuecomment-931038830
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org