You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2023/08/30 06:35:19 UTC
[GitHub] [airflow] potiuk opened a new issue, #33909: Split out providers into "standalone" python packages
potiuk opened a new issue, #33909:
URL: https://github.com/apache/airflow/issues/33909
### Body
As part of making it possible to move out of providers from Airflow core repository (we have not decided yet on it, we just want to make it possible) we should turn Airflow Providers into "real" packages.
Currently those packages are build "dynamically" -> https://github.com/apache/airflow/blob/main/dev/provider_packages/prepare_provider_packages.py is used as part of `breeze release-management prepare-provider-packages` to extract parts of the "airflow/providers/<provider>" sources dynamically, generate setup.* and pyproject.toml files dynamically and build the providers from those dynamically generated temporary folders.
This has some disadvantages - for example it does not make reproducible builds possible, and it requires complex `breeze` command, CI image and the python script in the image to build the packages (CI image is used to make sure all dependencies are installed and to provide isolation and cleanup between builds, also it allows to isolate (security) the host from container building the packages whenbuilding the providers in case of builds from contributor forks).
But this got us through last 3 years of releasing airflow and providers separately :).
With recent changes (#32604 upcoming #32048, the upcoming #33907 and a number of other changes already implemented in the past - we are quite close to make it possible to split out providers to "standalone" packages - where each provider is a separate "compliant with standard" package and has a complete independent directory where you can build the package without moving the sources around, using standard python tooling. This would require everything that relates to the providers to move to those directories (docs, tests are notably shared between airflow and providers in "docs" and "tests" and they should be moved around).
The early draft POC attempts to add scripts to automate such migration and result of it can be seen here:
* https://github.com/apache/airflow/pull/28291
* https://github.com/apache/airflow/pull/28292
However rhe challenge to solve for this one is to make it easy for contributors to contribute to airflow and providers together. We want to make it possible - simlarly as today to have an easy environment where you can edit both airflow and provider code and run tests, run airflow, run integration tests without extra hassle of installing and reinstalling the packages in editabkle mode.
Ideal workflow of the developer is where they can:
* run `breeze` and be able to edit any sources locally in the host - and airflow in breeze should automatically pick the changes in both airlfow and providers when interpreter is started
* install a local venv (in editable mode possibly) for airlfow and selected (all?) providers and the locally installed airflow should also pick changes for both airflow and providers when interpreter is started
The way how to do it and choice of (ideally) standard python tooling to make such move is not yet determined and is open for discussion, POCs and proposals.
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Split out providers into "standalone" python packages [airflow]
Posted by "shahar1 (via GitHub) via commits" <co...@airflow.apache.org>.
shahar1 commented on issue #33909:
URL: https://github.com/apache/airflow/issues/33909#issuecomment-6685361828
@potiuk I think that we could call it done? :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #33909: Split out providers into "standalone" python packages
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33909:
URL: https://github.com/apache/airflow/issues/33909#issuecomment-1698589033
cc: @eladkal @uranusjr @Taragolis -> some of the recent discussion from Slack: https://apache-airflow.slack.com/archives/CCPRP7943/p1692612215901809
At some point in time when (IF?) we have good idea how it can be done, we should bring it to the devlist.
I had a positive feedback on the "split" proposal quite some time ago https://lists.apache.org/thread/3s5tn1wnvo0cw9vofwmbjl0rkyvhrtbx but I do not want to make the discussion about it before we have a firm idea how we can achieve especially the point about "easy development" environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #33909: Split out providers into "standalone" python packages
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33909:
URL: https://github.com/apache/airflow/issues/33909#issuecomment-1698919217
> Ideally, there would also be no functional differences between Airflow running this way, and production image of Airflow - besides selection of installed/running providers.
Agree. But this one is tricky as we rely on entrypoints (and this is what ProvidersManager and INSTALL_PROVIDERS_FROM_SOURCES attempts to workaround). But if we find another solution, I would gladly remove those hacks. That's why I was eyeying Hatch https://github.com/pypa/hatch/issues/233 (hence my original comment about it).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #33909: Split out providers into "standalone" python packages
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33909:
URL: https://github.com/apache/airflow/issues/33909#issuecomment-1698619441
Another thought....
Maybe we do NOT have to solve the challenge of all-editable build for airflow for providers. I think we have a viable alternative....
* Breeze could remain an all-editable environment even after we move the providers to subfolders. It could read all the dependency information from pyproject.toml files instead of provider.yaml files (we can very easily change the way how `generated/provider_dependencies.json` are generated
* When/If we get to contributing @kaxil's `airflowctl` to become our "community" tool eventually- We could use it to have an easy local venv to install "airflow" in it and add an easy way to "inject" provider's editable sources (selected or all) and dependencies into such environment. While `airflowctl` is mainly targeted to support DAG development, we could make it easy to "inject" editable sources of providers too. That would be even better than what we have now - because contributors could test new providers also with old airflow version very easily (it's not easy currently).
Breeze could remain the "CI" driver and "All-editable" sources where latest source airflow + latest providers could be developed and tested (which is absolutely necessary BTW).
Also cc: @mobuchowski and @hussein-awala as I know they are interested in DevEX and I think it would be great to do some discussion/brainstorming running about that and potential future changes to it in a bit more of an "interest group".
Of course others are invited too.
We could design together the future of Airflow dev-env for that and once we have the providers as "standalone" packages, we could then start a discussion "Does it make sense/ Do we want to split providers to a separate repo". I have no ready answer for that (lots of thoughts though). But I think this dicussion should happen after we have the providers as "standalone" packages in our main "monorepo".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #33909: Split out providers into "standalone" python packages
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on issue #33909:
URL: https://github.com/apache/airflow/issues/33909#issuecomment-1698657277
Honestly I’m not particularly keen on moving the packages to separate repositories since that’d just make contributing more difficult instead of easier. But making providers separate packages from the core Airflow could be useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mobuchowski commented on issue #33909: Split out providers into "standalone" python packages
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on issue #33909:
URL: https://github.com/apache/airflow/issues/33909#issuecomment-1698815790
I like providers being standalone packages by default. In addition to the benefits described here, it would allow installing providers straight from github via pip.
I agree with @uranusjr that being a monorepo instead of multiple separate repos is a net benefit, especially when also thinking about local development process, not only CI.
>Ideal workflow of the developer is where they can:
> run breeze and be able to edit any sources locally in the host - and airflow in breeze should automatically pick the changes in both airlfow and providers when interpreter is started
> install a local venv (in editable mode possibly) for airlfow and selected (all?) providers and the locally installed airflow should also pick changes for both airflow and providers when interpreter is started
Ideally, there would also be no functional differences between Airflow running this way, and production image of Airflow - besides selection of installed/running providers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #33909: Split out providers into "standalone" python packages
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33909:
URL: https://github.com/apache/airflow/issues/33909#issuecomment-1698671744
> Honestly I’m not particularly keen on moving the packages to separate repositories since that’d just make contributing more difficult instead of easier. But making providers separate packages from the core Airflow could be useful.
I tend to agree on that statement. But that discussion is yet to happen :). It's a bit of theoretical possibility and I wanted to only start it after we see potentially how separate packages inside airflow repo work for us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Split out providers into "standalone" python packages [airflow]
Posted by "shahar1 (via GitHub) via commits" <co...@airflow.apache.org>.
shahar1 closed issue #33909: Split out providers into "standalone" python packages
URL: https://github.com/apache/airflow/issues/33909
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org