You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/11 18:33:12 UTC
[GitHub] [airflow] potiuk opened a new issue #11433: More Pythonic way of package separation
potiuk opened a new issue #11433:
URL: https://github.com/apache/airflow/issues/11433
Currenlty - for convenience, we prepare packages using custom scripts and Breeze where we copy sources and prepare packages from the same sources.
We might want to make it in more pythonic way possibly where each package shares their own setup.py maybe, or at most symbolic link to the same setup.py.
We should make a POC to check if this is possible and what it gives us vs. the current solution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11433: More Pythonic way of package separation
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #11433:
URL: https://github.com/apache/airflow/issues/11433#issuecomment-711167926
As part of the #11546 fixing I prepared a PR #11630 where I think I got it really sorted out very well:
1) we generate setup.py and config.py dynamically based on provider and information found in the Airflow's setup.py (dependencies etrc.). Those are automatically generated from JINJA templates for those files the same way we generate README.md and CHANGES files.
2) In the procedure of preparing such provider releases, those files are regenerated (using ./breeze prepare-provider-readme) and committed to the repository in the appropriate "airfllow/providers/<PROVIDER>" dir. We have (for now) two versions of those setup.py, setup.cfg, README, CHANGES files (one for backports, and one for regular providers). Whe the backports reach end-of-life we will be able to get rid of the BACKPORT)ORIVDERS_ ones.
3) when prepping the packages, I copy the sources to "provider_packages" - one, because it is needed to do refactor for backports and two - because the "setuptools" only works when setup.py, setup,cfg, README.md, CHANGELOG,txt are named as such. If we do it in main directory, we will be overwriting airflow ones. And we cannot do it easily in "providers" folder, because also setuptools require "airflow/providers" subdir for python files. I think this can stay like that permanently.
The nice thing about the final result (sdist packages) is that those .tar.gz packages when uncompressed, only contain sources of the provider they relate to + setup.py + setup.cfg + README.md + CHANGELOG.txt + LICENCE + NOTICE that makes sense for that provider, so we are now generating packages that "look like" and "behave like" packages prepared from separate repositories. Which is cool.
I think - if everyone else agrees with me - that this one can be closed once we merge #11630
WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #11433: More Pythonic way of package separation
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #11433:
URL: https://github.com/apache/airflow/issues/11433#issuecomment-706748090
cc: @mik-laj - I believe you mentioned that this something possible and possibly you would like to take care of this when it gets to that. It might be implemented as late as before we release 2.0 so there is still time and I consider that as an optional task as the current solution is not the most beautiful, but it works.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11433: More Pythonic way of package separation
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #11433:
URL: https://github.com/apache/airflow/issues/11433#issuecomment-706748090
cc: @mik-laj - I beleve you metnioned that this something possible and possibly you would like to take care about this when it gets to that. It might be implemented as late as before we release 2.0 so there is still time and I consider that as an optional task as the current solution is not the most beautiful, but it works.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #11433: More Pythonic way of package separation
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #11433:
URL: https://github.com/apache/airflow/issues/11433
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org