You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <ja...@potiuk.com> on 2025/03/14 19:27:46 UTC
[DISCUSS] Next steps of repo restructuring - airflow-core (finally!)
Hey here,
I have a first (very draft and still requires a number of changes) PR for
the final step of big refactoring of our projects and using workspace. This
is to let you know about the changes coming (so please take a look at the
consequences to not be surprised).
This is the most *scary* one -> moving all airflow code to "airflow-core".
And I have draft version of it in
https://github.com/apache/airflow/pull/47798
And it's not for the faint of heart :)
[image: image.png]
Note! It's not yet complete and unless you have some general comments, it's
likely not worth pointing to individual changes (yet) - it's more to take a
look at how things will look like eventually. I will work in the next two
days to get it to reviewable state, and will keep it rebased and running
till mid next-week. I would like to have it ready (including the release
process) for the fourth (and final?) beta).
Some resulting packaging changes:
*FOR DEVELOPMENT:*
* the pyproject.toml in the "root" of Airflow is still "apache-airflow"
package - but this will be an empty "meta" package that will install
together "apache-airflow-core", "apache-airflow-task-sdk" and optionally
providers (via extras)
* the airflow-core is a new "apache-airflow-core" distribution, where only
airflow dependencies and airflow "core" extras are configured (smtp/ otel,
pandas,rabbitmq etc) - I will likely cleanup some of those as well, some of
them are not needed. the nice thing is that this package has all
dependencies static (no hatch_build.py - everything is in pyproject.toml) -
which is pretty cool and allow us to better use dependabot for security
upgrades and notifications
The airflow-core structure is pretty standard:
airflow-core # <- this is folder where airflow-core distribution is
\- src
| \ airflow # <- This is airflow package
| \- api
| |- api_fastapi
| |- assets
| ...
|- tests
| \- always
| |- api
| ...
|- docs
|
|- pyproject.toml
|- README.md
* for development - i will describe later the `pypi` way, but with `uv`
things get simpler and we have a few new options (Dennis - this is
continuation of discussion on the uv sync commands, so it's worth to
look closely:
There are a number of ways you will be (eventually able to interact with
venv. After you checkout Airflow. You can change working directory and work
on different packages and depending on which directory you run `uv sync` -
uv (using workspace feature) will sync the **expected** dependencies.
It's best to get used to the fact that instead of one airflow project we
will have ~100 pretty independent projects, and while you can continue
working with all of them as a single huge "workspace", it is generally way
more convenient to change directory to the "distribution" you are working
on currently and do everything there - with isolated set of dependencies
required only for that "distribution" - "airflow-core", "task-sdk",
"providers/amazon", "providers/mongo" - those are all separate
distributions, and more and more we will be able to treat them as
independent projects (but we will conveniently keep the option to develop
and run tests in a joined "workspace" environment at the top of the project
where we can install and test everything together - that's a bit of `uv
workspace` magic in play.
Here are typical patterns:
1) Installing all development dependencies for everything (I.e complete
environment like in breeze) -- allows to run all tests for all airflow and
all providers
cd .
uv sync --all-packages
2) installing just airflow core with required dependencies (ready for most
core tests)
cd airflow-core
uv sync
3) installing airflow core with optional dependencies (should allow to run
all core tests - including for the optional core features such as otel etc).
cd airflow-core
uv sync --all-extras
4) installing individual provider dependencies (say amazon) - this allows
to run all tests of the provider you are working on - including installing
all dependencies from cross-provider dependencies (i.e. if you have google
tests in amazon provider, it will also install necessary google
dependencies).
cd providers/amazon
uv sync
Generally speaking - "airflow-core" will become (eventually) a truly
airflow-only distribution. It will have a few dependencies to "standard"
and "fab" providers - but I hope we will be able to get rid of those during
the resulting cleanup.
The IDE (IntelliJ) setting will just require "airflow-core/src" and
"airflow-core/tests" to be source/test roots as usual for other
distributions.
I will update the docs after I complete the PR, there are some small
variations on when to install which extras and I will play a bit to get to
the best developer experience and least surprises.
*FOR USERS*
For "installable" airflow (i.e. user's experience) - the changes will be
pretty much 100% transparent. When user will install "apache-airflow" or
"apache-airflow[google]" - things will work as they did before - only
instead of one "apache-airflow" distribution, they will have
"apache-airflow", "apache-airflow-core" and "apache-airflow-task-sdk"
installed.
Regarding version numbers etc., I will start a separate discussion - later
next week after we see how those packages will interact ("apache-airflow"
will only contain extras, but for compatibility reasons we likely want to
pin both "apache-airflow" and "apache-airflow-core" to each other, so that
users will be able to upgrade "core" by upgrading "apache-airflow" - we do
not want to change those habits likely.
The "apache-airflow-task-sdk" will be versioned separately.
Please take a look - also at the PR, see if you have any big
issues/questions/doubts - let's start discussion here - I am happy to
answer all general questions and adapt the PR to respond to
questions/suggestions.
In the meantime I will be working on making the PR green and adding missing
bits and pieces for the release process.
J.
Re: [DISCUSS] Next steps of repo restructuring - airflow-core (finally!)
Posted by Buğra Öztürk <oz...@gmail.com>.
Woohoo, amazing news! Sounds great! Another huge structural change that is
defining the `core` as reality, it won't be just definition anymore. This
will enable lots of possibilities. Thanks Jarek!
On Wed, 19 Mar 2025, 14:52 Jarek Potiuk, <ja...@potiuk.com> wrote:
> FYI. Likely today I am going to "undraft" :)
> https://github.com/apache/airflow/pull/47798.
>
> *TL;DR; My proposal on the process of merging "airflow-core" move (it will
> be slightly disturbing and require some local env adjustments by all
> people). Read on - or leave it for Friday morning lecture :)*
>
> I am trying to solve a few last riddles that Sphinx is asking me to solve,
> getting PROD image in order and having to make sure that several
> distributions produced are equivalent to the single distribution we had so
> far when installed together. That all seems doable today. Most of the
> tests, scripts, release process and tooling updates, development workflows,
> breeze image building and tests are already fixed in the PR. I will also
> add more detailed description what's "in" the PR - I had to make a number
> of small changes on how we are importing stuff and where dependencies are
> kept etc (hint - it's going to be much, much more standard) but those are
> mostly impacting development environment, rather than released
> distributions.
>
> I will have a few follow-up things after that - but they will be far
> smaller and far less - potentially - disruptive.
>
> I keep on rebasing the change and while it's not **too** difficult, it
> might be difficult for some people to rebase and resolve their conflicts,
> also I expect a few teething issues - mostly around CI/ breeze, so I think
> the following plan of making it happen is going to be the best:
>
> 1) I will make the PR green and hopefully reviewed by the end of tomorrow
> - or generally by the time last beta is released (it's mostly renames and
> fixing stuff to be used from new place - so even if it is a lot of
> changes/files, it should be somewhat easy to review)
> 2) I will merge it **right after** the beta is released. This means that I
> will merge and resolve conflicts for all PRs merged before beta3 (so hurry
> up! :) )
> 3) After that - everyone will be asked to do four things:
> a) rebase their opened PRs to latest main (and solve conflicts)
> b) run `*breeze ci image build --python 3.9*" (for doc building and
> some static checks)
> c) run `*uv self upgrade*` followed by `*uv sync --reinstall*`
> (possibly with `*--all-packages*` if they work on both - airflow core and
> providers at the same time.
> d) update their IDE settings - if they use IDEs - to make sure
> airflow-core/src and airflow/core/tests are available for code completion
> and import resolution (there will be updated screenshots on how to do it)
>
> 4) With that -> everything ***SHOULD*** work without big problems but I
> will make myself available to solve any teething issues (and I am sure
> other team members of the "CI/DEV task force" will help as well).
>
> Also I have been somewhat out (speaking and taking part in a great L8 Conf
> event in Warsaw Mon/Tue). I will be able to spend a lot of time Friday and
> weekend to fix any potential teething issues - so that starting Monday next
> week we can all continue working towards RC1 in an uninterrupted way.
>
> I am also going to be very available to fix any potential issues next week
> - while I will be completing some left-over tasks (I hope to get Sphinx a
> little less mysterious by moving the docs to a proper "devel-common" place
> for one and start identifying DB tests in providers and paving a path on
> how we can get rid of those).
>
> Let me know if you have any questions - I will do more docs and
> explanation and also happy to talk on slack and explain more of the changes
> once the PR is undrafted.
>
> J.
>
>
>
> On Fri, Mar 14, 2025 at 8:27 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Hey here,
>>
>> I have a first (very draft and still requires a number of changes) PR for
>> the final step of big refactoring of our projects and using workspace. This
>> is to let you know about the changes coming (so please take a look at the
>> consequences to not be surprised).
>>
>> This is the most *scary* one -> moving all airflow code to
>> "airflow-core". And I have draft version of it in
>> https://github.com/apache/airflow/pull/47798
>>
>> And it's not for the faint of heart :)
>>
>> [image: image.png]
>>
>> Note! It's not yet complete and unless you have some general comments,
>> it's likely not worth pointing to individual changes (yet) - it's more to
>> take a look at how things will look like eventually. I will work in the
>> next two days to get it to reviewable state, and will keep it rebased and
>> running till mid next-week. I would like to have it ready (including the
>> release process) for the fourth (and final?) beta).
>>
>> Some resulting packaging changes:
>>
>> *FOR DEVELOPMENT:*
>>
>> * the pyproject.toml in the "root" of Airflow is still "apache-airflow"
>> package - but this will be an empty "meta" package that will install
>> together "apache-airflow-core", "apache-airflow-task-sdk" and optionally
>> providers (via extras)
>>
>> * the airflow-core is a new "apache-airflow-core" distribution, where
>> only airflow dependencies and airflow "core" extras are configured (smtp/
>> otel, pandas,rabbitmq etc) - I will likely cleanup some of those as well,
>> some of them are not needed. the nice thing is that this package has all
>> dependencies static (no hatch_build.py - everything is in pyproject.toml) -
>> which is pretty cool and allow us to better use dependabot for security
>> upgrades and notifications
>>
>> The airflow-core structure is pretty standard:
>>
>> airflow-core # <- this is folder where airflow-core distribution is
>> \- src
>> | \ airflow # <- This is airflow package
>> | \- api
>> | |- api_fastapi
>> | |- assets
>> | ...
>> |- tests
>> | \- always
>> | |- api
>> | ...
>> |- docs
>> |
>> |- pyproject.toml
>> |- README.md
>>
>>
>> * for development - i will describe later the `pypi` way, but with `uv`
>> things get simpler and we have a few new options (Dennis - this is
>> continuation of discussion on the uv sync commands, so it's worth to
>> look closely:
>>
>> There are a number of ways you will be (eventually able to interact with
>> venv. After you checkout Airflow. You can change working directory and work
>> on different packages and depending on which directory you run `uv sync` -
>> uv (using workspace feature) will sync the **expected** dependencies.
>>
>> It's best to get used to the fact that instead of one airflow project we
>> will have ~100 pretty independent projects, and while you can continue
>> working with all of them as a single huge "workspace", it is generally way
>> more convenient to change directory to the "distribution" you are working
>> on currently and do everything there - with isolated set of dependencies
>> required only for that "distribution" - "airflow-core", "task-sdk",
>> "providers/amazon", "providers/mongo" - those are all separate
>> distributions, and more and more we will be able to treat them as
>> independent projects (but we will conveniently keep the option to develop
>> and run tests in a joined "workspace" environment at the top of the project
>> where we can install and test everything together - that's a bit of `uv
>> workspace` magic in play.
>>
>> Here are typical patterns:
>>
>> 1) Installing all development dependencies for everything (I.e complete
>> environment like in breeze) -- allows to run all tests for all airflow and
>> all providers
>>
>> cd .
>> uv sync --all-packages
>>
>> 2) installing just airflow core with required dependencies (ready for
>> most core tests)
>>
>> cd airflow-core
>> uv sync
>>
>> 3) installing airflow core with optional dependencies (should allow to
>> run all core tests - including for the optional core features such as otel
>> etc).
>>
>> cd airflow-core
>> uv sync --all-extras
>>
>> 4) installing individual provider dependencies (say amazon) - this allows
>> to run all tests of the provider you are working on - including installing
>> all dependencies from cross-provider dependencies (i.e. if you have google
>> tests in amazon provider, it will also install necessary google
>> dependencies).
>>
>> cd providers/amazon
>> uv sync
>>
>> Generally speaking - "airflow-core" will become (eventually) a truly
>> airflow-only distribution. It will have a few dependencies to "standard"
>> and "fab" providers - but I hope we will be able to get rid of those during
>> the resulting cleanup.
>>
>> The IDE (IntelliJ) setting will just require "airflow-core/src" and
>> "airflow-core/tests" to be source/test roots as usual for other
>> distributions.
>>
>> I will update the docs after I complete the PR, there are some small
>> variations on when to install which extras and I will play a bit to get to
>> the best developer experience and least surprises.
>>
>> *FOR USERS*
>>
>> For "installable" airflow (i.e. user's experience) - the changes will be
>> pretty much 100% transparent. When user will install "apache-airflow" or
>> "apache-airflow[google]" - things will work as they did before - only
>> instead of one "apache-airflow" distribution, they will have
>> "apache-airflow", "apache-airflow-core" and "apache-airflow-task-sdk"
>> installed.
>>
>> Regarding version numbers etc., I will start a separate discussion -
>> later next week after we see how those packages will interact
>> ("apache-airflow" will only contain extras, but for compatibility reasons
>> we likely want to pin both "apache-airflow" and "apache-airflow-core" to
>> each other, so that users will be able to upgrade "core" by upgrading
>> "apache-airflow" - we do not want to change those habits likely.
>>
>> The "apache-airflow-task-sdk" will be versioned separately.
>>
>> Please take a look - also at the PR, see if you have any big
>> issues/questions/doubts - let's start discussion here - I am happy to
>> answer all general questions and adapt the PR to respond to
>> questions/suggestions.
>>
>> In the meantime I will be working on making the PR green and adding
>> missing bits and pieces for the release process.
>>
>> J.
>>
>>
>>
>>
Re: [DISCUSS] Next steps of repo restructuring - airflow-core (finally!)
Posted by Jarek Potiuk <ja...@potiuk.com>.
FYI. Likely today I am going to "undraft" :)
https://github.com/apache/airflow/pull/47798.
*TL;DR; My proposal on the process of merging "airflow-core" move (it will
be slightly disturbing and require some local env adjustments by all
people). Read on - or leave it for Friday morning lecture :)*
I am trying to solve a few last riddles that Sphinx is asking me to solve,
getting PROD image in order and having to make sure that several
distributions produced are equivalent to the single distribution we had so
far when installed together. That all seems doable today. Most of the
tests, scripts, release process and tooling updates, development workflows,
breeze image building and tests are already fixed in the PR. I will also
add more detailed description what's "in" the PR - I had to make a number
of small changes on how we are importing stuff and where dependencies are
kept etc (hint - it's going to be much, much more standard) but those are
mostly impacting development environment, rather than released
distributions.
I will have a few follow-up things after that - but they will be far
smaller and far less - potentially - disruptive.
I keep on rebasing the change and while it's not **too** difficult, it
might be difficult for some people to rebase and resolve their conflicts,
also I expect a few teething issues - mostly around CI/ breeze, so I think
the following plan of making it happen is going to be the best:
1) I will make the PR green and hopefully reviewed by the end of tomorrow -
or generally by the time last beta is released (it's mostly renames and
fixing stuff to be used from new place - so even if it is a lot of
changes/files, it should be somewhat easy to review)
2) I will merge it **right after** the beta is released. This means that I
will merge and resolve conflicts for all PRs merged before beta3 (so hurry
up! :) )
3) After that - everyone will be asked to do four things:
a) rebase their opened PRs to latest main (and solve conflicts)
b) run `*breeze ci image build --python 3.9*" (for doc building and
some static checks)
c) run `*uv self upgrade*` followed by `*uv sync --reinstall*`
(possibly with `*--all-packages*` if they work on both - airflow core and
providers at the same time.
d) update their IDE settings - if they use IDEs - to make sure
airflow-core/src and airflow/core/tests are available for code completion
and import resolution (there will be updated screenshots on how to do it)
4) With that -> everything ***SHOULD*** work without big problems but I
will make myself available to solve any teething issues (and I am sure
other team members of the "CI/DEV task force" will help as well).
Also I have been somewhat out (speaking and taking part in a great L8 Conf
event in Warsaw Mon/Tue). I will be able to spend a lot of time Friday and
weekend to fix any potential teething issues - so that starting Monday next
week we can all continue working towards RC1 in an uninterrupted way.
I am also going to be very available to fix any potential issues next week
- while I will be completing some left-over tasks (I hope to get Sphinx a
little less mysterious by moving the docs to a proper "devel-common" place
for one and start identifying DB tests in providers and paving a path on
how we can get rid of those).
Let me know if you have any questions - I will do more docs and explanation
and also happy to talk on slack and explain more of the changes once the PR
is undrafted.
J.
On Fri, Mar 14, 2025 at 8:27 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hey here,
>
> I have a first (very draft and still requires a number of changes) PR for
> the final step of big refactoring of our projects and using workspace. This
> is to let you know about the changes coming (so please take a look at the
> consequences to not be surprised).
>
> This is the most *scary* one -> moving all airflow code to
> "airflow-core". And I have draft version of it in
> https://github.com/apache/airflow/pull/47798
>
> And it's not for the faint of heart :)
>
> [image: image.png]
>
> Note! It's not yet complete and unless you have some general comments,
> it's likely not worth pointing to individual changes (yet) - it's more to
> take a look at how things will look like eventually. I will work in the
> next two days to get it to reviewable state, and will keep it rebased and
> running till mid next-week. I would like to have it ready (including the
> release process) for the fourth (and final?) beta).
>
> Some resulting packaging changes:
>
> *FOR DEVELOPMENT:*
>
> * the pyproject.toml in the "root" of Airflow is still "apache-airflow"
> package - but this will be an empty "meta" package that will install
> together "apache-airflow-core", "apache-airflow-task-sdk" and optionally
> providers (via extras)
>
> * the airflow-core is a new "apache-airflow-core" distribution, where only
> airflow dependencies and airflow "core" extras are configured (smtp/ otel,
> pandas,rabbitmq etc) - I will likely cleanup some of those as well, some of
> them are not needed. the nice thing is that this package has all
> dependencies static (no hatch_build.py - everything is in pyproject.toml) -
> which is pretty cool and allow us to better use dependabot for security
> upgrades and notifications
>
> The airflow-core structure is pretty standard:
>
> airflow-core # <- this is folder where airflow-core distribution is
> \- src
> | \ airflow # <- This is airflow package
> | \- api
> | |- api_fastapi
> | |- assets
> | ...
> |- tests
> | \- always
> | |- api
> | ...
> |- docs
> |
> |- pyproject.toml
> |- README.md
>
>
> * for development - i will describe later the `pypi` way, but with `uv`
> things get simpler and we have a few new options (Dennis - this is
> continuation of discussion on the uv sync commands, so it's worth to
> look closely:
>
> There are a number of ways you will be (eventually able to interact with
> venv. After you checkout Airflow. You can change working directory and work
> on different packages and depending on which directory you run `uv sync` -
> uv (using workspace feature) will sync the **expected** dependencies.
>
> It's best to get used to the fact that instead of one airflow project we
> will have ~100 pretty independent projects, and while you can continue
> working with all of them as a single huge "workspace", it is generally way
> more convenient to change directory to the "distribution" you are working
> on currently and do everything there - with isolated set of dependencies
> required only for that "distribution" - "airflow-core", "task-sdk",
> "providers/amazon", "providers/mongo" - those are all separate
> distributions, and more and more we will be able to treat them as
> independent projects (but we will conveniently keep the option to develop
> and run tests in a joined "workspace" environment at the top of the project
> where we can install and test everything together - that's a bit of `uv
> workspace` magic in play.
>
> Here are typical patterns:
>
> 1) Installing all development dependencies for everything (I.e complete
> environment like in breeze) -- allows to run all tests for all airflow and
> all providers
>
> cd .
> uv sync --all-packages
>
> 2) installing just airflow core with required dependencies (ready for most
> core tests)
>
> cd airflow-core
> uv sync
>
> 3) installing airflow core with optional dependencies (should allow to run
> all core tests - including for the optional core features such as otel etc).
>
> cd airflow-core
> uv sync --all-extras
>
> 4) installing individual provider dependencies (say amazon) - this allows
> to run all tests of the provider you are working on - including installing
> all dependencies from cross-provider dependencies (i.e. if you have google
> tests in amazon provider, it will also install necessary google
> dependencies).
>
> cd providers/amazon
> uv sync
>
> Generally speaking - "airflow-core" will become (eventually) a truly
> airflow-only distribution. It will have a few dependencies to "standard"
> and "fab" providers - but I hope we will be able to get rid of those during
> the resulting cleanup.
>
> The IDE (IntelliJ) setting will just require "airflow-core/src" and
> "airflow-core/tests" to be source/test roots as usual for other
> distributions.
>
> I will update the docs after I complete the PR, there are some small
> variations on when to install which extras and I will play a bit to get to
> the best developer experience and least surprises.
>
> *FOR USERS*
>
> For "installable" airflow (i.e. user's experience) - the changes will be
> pretty much 100% transparent. When user will install "apache-airflow" or
> "apache-airflow[google]" - things will work as they did before - only
> instead of one "apache-airflow" distribution, they will have
> "apache-airflow", "apache-airflow-core" and "apache-airflow-task-sdk"
> installed.
>
> Regarding version numbers etc., I will start a separate discussion - later
> next week after we see how those packages will interact ("apache-airflow"
> will only contain extras, but for compatibility reasons we likely want to
> pin both "apache-airflow" and "apache-airflow-core" to each other, so that
> users will be able to upgrade "core" by upgrading "apache-airflow" - we do
> not want to change those habits likely.
>
> The "apache-airflow-task-sdk" will be versioned separately.
>
> Please take a look - also at the PR, see if you have any big
> issues/questions/doubts - let's start discussion here - I am happy to
> answer all general questions and adapt the PR to respond to
> questions/suggestions.
>
> In the meantime I will be working on making the PR green and adding
> missing bits and pieces for the release process.
>
> J.
>
>
>
>
Re: [DISCUSS] Next steps of repo restructuring - airflow-core (finally!)
Posted by Jarek Potiuk <ja...@potiuk.com>.
Ok. The PR https://github.com/apache/airflow/pull/47798 is "green" (minus
failing main issue with microsoft libraries solved separately and randomly
failing k8s tests that we are fighting with).
I also added a description of the changes and happy to take any comments
and reviews. Would be great to get it merged **right** after the beta
release, to not disturb the release but also to get as many open PRs as
possible before the merge to minimize the number of conflicts YOU will have
to solve (at the expense of ME solving them :) ).
I would like to have a small discussion afterwards on the exact way we will
treat `uv sync` and dependencies - including pre-installed providers, but I
would like to have this discussion later, I do not want to "muddy the
waters" right now. After we merge it and get some teething problems sorted
out, I will start a discussion thread about it. In short we can still
decide and move around thing such as - how many extras are installed by
default with `uv sync`, where we keep pre-installed providers definition -
is it in `airflow-core` or `airflow` and whether we want to keep with `pip`
way of doing things or can we entirely rely on `uv` for development (the
latter would simplify some of the hatch_build_* logic and allow us to have
more static dependency definition.
But let's leave that discussion for next week. I will set the stage for it
today at the dev call, before I send the email with a more detailed
description of options and dependencies we have - but that should not stop
the "small :) " PR of mine to be merged:
[image: image.png]
J.
On Wed, Mar 19, 2025 at 9:12 PM Shahar Epstein <sh...@apache.org> wrote:
> That's hardcore (pun intended) :D
> Great work and good luck merging it!
>
> On Fri, Mar 14, 2025 at 9:28 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Hey here,
> >
> > I have a first (very draft and still requires a number of changes) PR for
> > the final step of big refactoring of our projects and using workspace.
> This
> > is to let you know about the changes coming (so please take a look at the
> > consequences to not be surprised).
> >
> > This is the most *scary* one -> moving all airflow code to
> > "airflow-core". And I have draft version of it in
> > https://github.com/apache/airflow/pull/47798
> >
> > And it's not for the faint of heart :)
> >
> > [image: image.png]
> >
> > Note! It's not yet complete and unless you have some general comments,
> > it's likely not worth pointing to individual changes (yet) - it's more to
> > take a look at how things will look like eventually. I will work in the
> > next two days to get it to reviewable state, and will keep it rebased
> and
> > running till mid next-week. I would like to have it ready (including the
> > release process) for the fourth (and final?) beta).
> >
> > Some resulting packaging changes:
> >
> > *FOR DEVELOPMENT:*
> >
> > * the pyproject.toml in the "root" of Airflow is still "apache-airflow"
> > package - but this will be an empty "meta" package that will install
> > together "apache-airflow-core", "apache-airflow-task-sdk" and optionally
> > providers (via extras)
> >
> > * the airflow-core is a new "apache-airflow-core" distribution, where
> only
> > airflow dependencies and airflow "core" extras are configured (smtp/
> otel,
> > pandas,rabbitmq etc) - I will likely cleanup some of those as well, some
> of
> > them are not needed. the nice thing is that this package has all
> > dependencies static (no hatch_build.py - everything is in
> pyproject.toml) -
> > which is pretty cool and allow us to better use dependabot for security
> > upgrades and notifications
> >
> > The airflow-core structure is pretty standard:
> >
> > airflow-core # <- this is folder where airflow-core distribution is
> > \- src
> > | \ airflow # <- This is airflow package
> > | \- api
> > | |- api_fastapi
> > | |- assets
> > | ...
> > |- tests
> > | \- always
> > | |- api
> > | ...
> > |- docs
> > |
> > |- pyproject.toml
> > |- README.md
> >
> >
> > * for development - i will describe later the `pypi` way, but with `uv`
> > things get simpler and we have a few new options (Dennis - this is
> > continuation of discussion on the uv sync commands, so it's worth to
> > look closely:
> >
> > There are a number of ways you will be (eventually able to interact with
> > venv. After you checkout Airflow. You can change working directory and
> work
> > on different packages and depending on which directory you run `uv sync`
> -
> > uv (using workspace feature) will sync the **expected** dependencies.
> >
> > It's best to get used to the fact that instead of one airflow project we
> > will have ~100 pretty independent projects, and while you can continue
> > working with all of them as a single huge "workspace", it is generally
> way
> > more convenient to change directory to the "distribution" you are working
> > on currently and do everything there - with isolated set of dependencies
> > required only for that "distribution" - "airflow-core", "task-sdk",
> > "providers/amazon", "providers/mongo" - those are all separate
> > distributions, and more and more we will be able to treat them as
> > independent projects (but we will conveniently keep the option to develop
> > and run tests in a joined "workspace" environment at the top of the
> project
> > where we can install and test everything together - that's a bit of `uv
> > workspace` magic in play.
> >
> > Here are typical patterns:
> >
> > 1) Installing all development dependencies for everything (I.e complete
> > environment like in breeze) -- allows to run all tests for all airflow
> and
> > all providers
> >
> > cd .
> > uv sync --all-packages
> >
> > 2) installing just airflow core with required dependencies (ready for
> most
> > core tests)
> >
> > cd airflow-core
> > uv sync
> >
> > 3) installing airflow core with optional dependencies (should allow to
> run
> > all core tests - including for the optional core features such as otel
> etc).
> >
> > cd airflow-core
> > uv sync --all-extras
> >
> > 4) installing individual provider dependencies (say amazon) - this allows
> > to run all tests of the provider you are working on - including
> installing
> > all dependencies from cross-provider dependencies (i.e. if you have
> google
> > tests in amazon provider, it will also install necessary google
> > dependencies).
> >
> > cd providers/amazon
> > uv sync
> >
> > Generally speaking - "airflow-core" will become (eventually) a truly
> > airflow-only distribution. It will have a few dependencies to "standard"
> > and "fab" providers - but I hope we will be able to get rid of those
> during
> > the resulting cleanup.
> >
> > The IDE (IntelliJ) setting will just require "airflow-core/src" and
> > "airflow-core/tests" to be source/test roots as usual for other
> > distributions.
> >
> > I will update the docs after I complete the PR, there are some small
> > variations on when to install which extras and I will play a bit to get
> to
> > the best developer experience and least surprises.
> >
> > *FOR USERS*
> >
> > For "installable" airflow (i.e. user's experience) - the changes will be
> > pretty much 100% transparent. When user will install "apache-airflow" or
> > "apache-airflow[google]" - things will work as they did before - only
> > instead of one "apache-airflow" distribution, they will have
> > "apache-airflow", "apache-airflow-core" and "apache-airflow-task-sdk"
> > installed.
> >
> > Regarding version numbers etc., I will start a separate discussion -
> later
> > next week after we see how those packages will interact ("apache-airflow"
> > will only contain extras, but for compatibility reasons we likely want to
> > pin both "apache-airflow" and "apache-airflow-core" to each other, so
> that
> > users will be able to upgrade "core" by upgrading "apache-airflow" - we
> do
> > not want to change those habits likely.
> >
> > The "apache-airflow-task-sdk" will be versioned separately.
> >
> > Please take a look - also at the PR, see if you have any big
> > issues/questions/doubts - let's start discussion here - I am happy to
> > answer all general questions and adapt the PR to respond to
> > questions/suggestions.
> >
> > In the meantime I will be working on making the PR green and adding
> > missing bits and pieces for the release process.
> >
> > J.
> >
> >
> >
> >
>
Re: [DISCUSS] Next steps of repo restructuring - airflow-core (finally!)
Posted by Shahar Epstein <sh...@apache.org>.
That's hardcore (pun intended) :D
Great work and good luck merging it!
On Fri, Mar 14, 2025 at 9:28 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hey here,
>
> I have a first (very draft and still requires a number of changes) PR for
> the final step of big refactoring of our projects and using workspace. This
> is to let you know about the changes coming (so please take a look at the
> consequences to not be surprised).
>
> This is the most *scary* one -> moving all airflow code to
> "airflow-core". And I have draft version of it in
> https://github.com/apache/airflow/pull/47798
>
> And it's not for the faint of heart :)
>
> [image: image.png]
>
> Note! It's not yet complete and unless you have some general comments,
> it's likely not worth pointing to individual changes (yet) - it's more to
> take a look at how things will look like eventually. I will work in the
> next two days to get it to reviewable state, and will keep it rebased and
> running till mid next-week. I would like to have it ready (including the
> release process) for the fourth (and final?) beta).
>
> Some resulting packaging changes:
>
> *FOR DEVELOPMENT:*
>
> * the pyproject.toml in the "root" of Airflow is still "apache-airflow"
> package - but this will be an empty "meta" package that will install
> together "apache-airflow-core", "apache-airflow-task-sdk" and optionally
> providers (via extras)
>
> * the airflow-core is a new "apache-airflow-core" distribution, where only
> airflow dependencies and airflow "core" extras are configured (smtp/ otel,
> pandas,rabbitmq etc) - I will likely cleanup some of those as well, some of
> them are not needed. the nice thing is that this package has all
> dependencies static (no hatch_build.py - everything is in pyproject.toml) -
> which is pretty cool and allow us to better use dependabot for security
> upgrades and notifications
>
> The airflow-core structure is pretty standard:
>
> airflow-core # <- this is folder where airflow-core distribution is
> \- src
> | \ airflow # <- This is airflow package
> | \- api
> | |- api_fastapi
> | |- assets
> | ...
> |- tests
> | \- always
> | |- api
> | ...
> |- docs
> |
> |- pyproject.toml
> |- README.md
>
>
> * for development - i will describe later the `pypi` way, but with `uv`
> things get simpler and we have a few new options (Dennis - this is
> continuation of discussion on the uv sync commands, so it's worth to
> look closely:
>
> There are a number of ways you will be (eventually able to interact with
> venv. After you checkout Airflow. You can change working directory and work
> on different packages and depending on which directory you run `uv sync` -
> uv (using workspace feature) will sync the **expected** dependencies.
>
> It's best to get used to the fact that instead of one airflow project we
> will have ~100 pretty independent projects, and while you can continue
> working with all of them as a single huge "workspace", it is generally way
> more convenient to change directory to the "distribution" you are working
> on currently and do everything there - with isolated set of dependencies
> required only for that "distribution" - "airflow-core", "task-sdk",
> "providers/amazon", "providers/mongo" - those are all separate
> distributions, and more and more we will be able to treat them as
> independent projects (but we will conveniently keep the option to develop
> and run tests in a joined "workspace" environment at the top of the project
> where we can install and test everything together - that's a bit of `uv
> workspace` magic in play.
>
> Here are typical patterns:
>
> 1) Installing all development dependencies for everything (I.e complete
> environment like in breeze) -- allows to run all tests for all airflow and
> all providers
>
> cd .
> uv sync --all-packages
>
> 2) installing just airflow core with required dependencies (ready for most
> core tests)
>
> cd airflow-core
> uv sync
>
> 3) installing airflow core with optional dependencies (should allow to run
> all core tests - including for the optional core features such as otel etc).
>
> cd airflow-core
> uv sync --all-extras
>
> 4) installing individual provider dependencies (say amazon) - this allows
> to run all tests of the provider you are working on - including installing
> all dependencies from cross-provider dependencies (i.e. if you have google
> tests in amazon provider, it will also install necessary google
> dependencies).
>
> cd providers/amazon
> uv sync
>
> Generally speaking - "airflow-core" will become (eventually) a truly
> airflow-only distribution. It will have a few dependencies to "standard"
> and "fab" providers - but I hope we will be able to get rid of those during
> the resulting cleanup.
>
> The IDE (IntelliJ) setting will just require "airflow-core/src" and
> "airflow-core/tests" to be source/test roots as usual for other
> distributions.
>
> I will update the docs after I complete the PR, there are some small
> variations on when to install which extras and I will play a bit to get to
> the best developer experience and least surprises.
>
> *FOR USERS*
>
> For "installable" airflow (i.e. user's experience) - the changes will be
> pretty much 100% transparent. When user will install "apache-airflow" or
> "apache-airflow[google]" - things will work as they did before - only
> instead of one "apache-airflow" distribution, they will have
> "apache-airflow", "apache-airflow-core" and "apache-airflow-task-sdk"
> installed.
>
> Regarding version numbers etc., I will start a separate discussion - later
> next week after we see how those packages will interact ("apache-airflow"
> will only contain extras, but for compatibility reasons we likely want to
> pin both "apache-airflow" and "apache-airflow-core" to each other, so that
> users will be able to upgrade "core" by upgrading "apache-airflow" - we do
> not want to change those habits likely.
>
> The "apache-airflow-task-sdk" will be versioned separately.
>
> Please take a look - also at the PR, see if you have any big
> issues/questions/doubts - let's start discussion here - I am happy to
> answer all general questions and adapt the PR to respond to
> questions/suggestions.
>
> In the meantime I will be working on making the PR green and adding
> missing bits and pieces for the release process.
>
> J.
>
>
>
>