You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <ja...@potiuk.com> on 2025/02/16 13:12:06 UTC
[DISCUSS] Turn "tests_common" into separate distribution for development
Hello here,
Next phase of the cleanup - it's been sped up by the comment from @kxepal
- https://github.com/apache/airflow/pull/46801#issuecomment-2661415731 - I
have planned to do it a bit later this week, but maybe indeed it's a good
idea to start a discussion now so that people are not confused.
Currently we are using "from tests_common" to reuse code in various
providers and airflow, and this is fine, with the exception that
"tests_common" is currently just a package in "airflow" main project. But
it does not have to be (or rather it should not be).
We should have it in a separate sub-project - similarly as providers/* and
task_sdk are now separate projects - and we should add dependency to the
common test code distribution from within all the projects that use it (via
workspace feature).
My proposal is to name it "common_test_code" (but I am open to any
suggestions): It will look like this
.
|- airflow
|- common_test_code
| pyproject.toml
| src
| tests_common
The code in airflow and providers will not change, they will continue to
use "from tests_common import ...".
* uv will work as it used to work, no changes will be needed - uv sync will
automatically install the common_test_code as editable project
* additionally - (and that's a big plus) after this change you should be
able to run `uv sync` in a provider folder and have provider-specific
separate venv, and basically you should be able to run tests for a provider
using that .venv and generally treat provider project as a "standalone" one
- this is what I hinted at in the previous mail.
* people who do not use uv but pip for example will have to manually add
the `common_test_code` as an editable install in their venv. For example:
pip install -e . -e ./task_sdk -e. ./providers/google -e ./common_test_code
This is the next step in standardizing the layout of the Airflow project
using workspaces.
J.
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Jens Scheffler <j_...@gmx.de.INVALID>.
+1
Just wondernig... would an optional dependency not be the right place to
describe that apache-airflow-providers-google[tests] would have an
dependency to the common_tests subproject?
Would mean you would need to install via
pip install -e . -e ./task_sdk[tests] -e. ./providers/google[tests]
On 16.02.25 14:12, Jarek Potiuk wrote:
> Hello here,
>
> Next phase of the cleanup - it's been sped up by the comment from @kxepal
> - https://github.com/apache/airflow/pull/46801#issuecomment-2661415731 - I
> have planned to do it a bit later this week, but maybe indeed it's a good
> idea to start a discussion now so that people are not confused.
>
> Currently we are using "from tests_common" to reuse code in various
> providers and airflow, and this is fine, with the exception that
> "tests_common" is currently just a package in "airflow" main project. But
> it does not have to be (or rather it should not be).
>
> We should have it in a separate sub-project - similarly as providers/* and
> task_sdk are now separate projects - and we should add dependency to the
> common test code distribution from within all the projects that use it (via
> workspace feature).
>
> My proposal is to name it "common_test_code" (but I am open to any
> suggestions): It will look like this
>
> .
> |- airflow
> |- common_test_code
> | pyproject.toml
> | src
> | tests_common
>
> The code in airflow and providers will not change, they will continue to
> use "from tests_common import ...".
>
> * uv will work as it used to work, no changes will be needed - uv sync will
> automatically install the common_test_code as editable project
>
> * additionally - (and that's a big plus) after this change you should be
> able to run `uv sync` in a provider folder and have provider-specific
> separate venv, and basically you should be able to run tests for a provider
> using that .venv and generally treat provider project as a "standalone" one
> - this is what I hinted at in the previous mail.
>
> * people who do not use uv but pip for example will have to manually add
> the `common_test_code` as an editable install in their venv. For example:
>
> pip install -e . -e ./task_sdk -e. ./providers/google -e ./common_test_code
>
> This is the next step in standardizing the layout of the Airflow project
> using workspaces.
>
> J.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Jens Scheffler <j_...@gmx.de.INVALID>.
+1 for devel-common
On 04.03.25 20:21, Ferruzzi, Dennis wrote:
> devel-common sounds reasonable
>
>
> - ferruzzi
>
>
> ________________________________
> From: Jarek Potiuk <ja...@potiuk.com>
> Sent: Tuesday, March 4, 2025 10:53 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXT] [DISCUSS] Turn "tests_common" into separate distribution for development
>
>
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.
>
>
> I am doing a bit more cleanup, and I have found that the easier way to fix some of the remaining issues will be to clean-up (and remove) the remaining editable devel dependencies and incorporate them all in the "tests-common" package.
>
> You can take a look at the PR: https://github.com/apache/airflow/pull/47281 - but basically what it means is:
>
> * all "devel" dependencies are added as required dependencies of "tests-common" (except "doc" - I will treat doc separately).
> * I removed all "legacy" extras from the "airflow" package including "bundle" extras: "devel-ci", "devel-db" and a few others - except installing "all" dependencies as they were pretty useless. Except "editable" all only - see below - we will have no more devel and bundle extras (Ash - I guess this is what you were looking forward to :) )
> * Instead we have one "all" extra that is available only in editable mode - it's not documented in user documentation and it is really only useful to install everything with `pip` (with uv you get the same with `uv sync --all-extras`) - this is still used internally in the CI image to run `uv pip install .[all] --constraints` until we switch to use "uv.lock" in the future
> * hatch_build.py is significantly simpler and easier to understand now - with all the bundle removal and moving all dependencies to ./tests-common
> * we still need dynamic dependencies and ./hatch_build.py - but less and less, with PEP735 (https://peps.python.org/pep-0735/) implemented in pip in April we will likely be able to turn our optional dependencies into static pyproject.toml deps, and with https://peps.python.org/pep-0771/ (needs approval and implementation) we will likely be able to have static pyproject.toml required dependencies as well.
> * I updated install and contributing docs to be "uv first" - presenting as recommended and the first option to go with `uv` - as it is becoming deceptively simple now to work with both - airflow and providers (and it will be even simpler after few next PRs)
>
> Now. THE BIG QUESTION - naming again. With all those changes..... `tests-common` is becoming more of a `devel-common` package - because what it will do - it will contribute to all other sub-projects all the development tooling that is needed for those other projects to be developed.
>
> Shall we name it "devel-common" instead of `tests-common`?
>
> Part of why I think it makes sense is this specification in `tests-common/pyproject.toml` (see attachment) - also https://ibb.co/1cwZQRm if you do not see attachment. Note that these are generally "common" devel dependencies - and each of the packages can contribute their own (including task-sdk, airflow-core (future), every provider etc.)
>
> [Screenshot from 2025-03-04 19-50-32.png]
>
>
> J.
>
>
>
> On Mon, Mar 3, 2025 at 4:14 AM Jarek Potiuk <ja...@potiuk.com>> wrote:
> Hello everyone.
>
> I created the PR for that https://github.com/apache/airflow/pull/47281.
>
> It's even nicer than I anticipated. I love the new super-simple workflows this restructuring finally enabled.
>
> With `uv` and workspace, and the new structure of tests, developing and running tests for providers, task-sdk or any of the other future sub-projects becomes very, very straightforward, we avoid duplication of pytest options and switching between airflow, task-sdk, providers tests will be simple and straightforward.
>
> 1) First of all with this change, we remove `devel-tests` extra. It was always needed in local venv to install all test dependencies, But this is completely gone right now. Test dependencies are automatically installed now when you run `uv sync` - and you do not need to specify `--extra devel-tests`. If you are still using `pip` (I strongly recommend switching to uv) - you just install `pip install -e ./tests-common`.
>
> 2) the previous way of syncing and running tests in "everything installed" mode works as it worked before:
>
> uv sync --all-extras
>
> This installs all possible extras of airflow, allows you to run tests for all providers, activate the venv in `.venv` and run `pytests tests/always` or `pytest providers/mongo/tests` or `pytest task_sdk/tests and it should all work fine as it did before
>
> 3) you can also install dependencies of a selected provider (or a few of those) in your .venv
>
> uv sync --extra mongo
>
> 3) All the `breeze testing provider-tests --test-type "Providers[mongo]"" and generally all the ways to replicate what we run in CI inside breeze containers with selective checks will also continue to work - no changes here.
>
> But then, a little bit of standardization, a bit of pre-commits that automatically update `pyproject.toml` files of ours with cross-provider dependencies and a little of `uv` workspace magic sprinkled over the repo of ours and now we can do something much more straightforward:
>
> You can change directory to task_sdk, or provider folder of your choice and simply run:
>
> cd providers/mongo
> uv run pytest
>
> or
>
> cd task_sdk
> uv run pytest
>
> And it will do what you would expect to do - even if this is the first thing you do after checking out Airflow's repo without any earlier syncing or preparation. Yes. It's that simple.
>
> The `uv run pytest` command will now automatically: install python, create a venv, sync your venv to ONLY have dependencies needed for your provider or any of the providers imported by your provider, and all test dependencies needed, it will find only non-system, non-integration tests and run them. That's it. If you want to only develop one provider - you will not have to worry any more about breeze image or setting up all other dependencies.
>
> Go ahead and try it. It's unbelievably simple.
>
> J.
>
>
>
> On Tue, Feb 25, 2025 at 5:45 PM Ash Berlin-Taylor <as...@apache.org>> wrote:
> I like this approach, lets do it!
>
>> On 25 Feb 2025, at 16:02, Jarek Potiuk <ja...@potiuk.com>> wrote:
>>
>> airflow-core
>> task-sdk
>> tests-common
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Jarek Potiuk <ja...@potiuk.com>.
Two small things (aka possible teething issues):
* *uv self update* <- once you do it, in the future you will get a clear
error when you should update your uv if you have not done so
* *uv sync --reinstall* <- forces reinstallation of airflow package. Uv
does it periodically with some heuristics, but this one will force it after
restructuring
J.
On Wed, Mar 5, 2025 at 11:13 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> The PR is now merged,,, we have 'devel-common` now.
>
> The important bit now for everyone... It's easier than ever to have
> dev-env running for airflow...
>
> 1) uv sync -> and your .venv env allows you to run all tests for
> airflow/task-sdk
> 2) uv sync --all-packages -> and your .venv allows you to also run tests
> for all providers
> 3)
> cd providers/PROVIDER
> uv sync
>
> gives env ready to run all tests for the provider of your choice
>
> uv run pytest
>
> does it.
>
> There are likely a few missing per-provider deps but we will find and
> squash them soon !
>
> Have fun!
>
> J.
>
>
>
> On Wed, Mar 5, 2025 at 12:48 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Seems like the PR is getting to "green" zone - so one last push - and I
>> am changing it to "devel-common" unless I hear strong NOOOOOOO!
>>
>> On Tue, Mar 4, 2025 at 10:36 PM Vincent Beck <vi...@apache.org> wrote:
>>
>>> devel-common makes sense to me
>>>
>>> On 2025/03/04 21:13:47 "Oliveira, Niko" wrote:
>>> > +1 to devel-common from me
>>> >
>>> > ________________________________
>>> > From: Ferruzzi, Dennis <fe...@amazon.com.INVALID>
>>> > Sent: Tuesday, March 4, 2025 11:21:20 AM
>>> > To: dev@airflow.apache.org
>>> > Subject: RE: [EXT] [DISCUSS] Turn "tests_common" into separate
>>> distribution for development
>>> >
>>> > CAUTION: This email originated from outside of the organization. Do
>>> not click links or open attachments unless you can confirm the sender and
>>> know the content is safe.
>>> >
>>> >
>>> >
>>> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
>>> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous
>>> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
>>> certain que le contenu ne présente aucun risque.
>>> >
>>> >
>>> >
>>> > devel-common sounds reasonable
>>> >
>>> >
>>> > - ferruzzi
>>> >
>>> >
>>> > ________________________________
>>> > From: Jarek Potiuk <ja...@potiuk.com>
>>> > Sent: Tuesday, March 4, 2025 10:53 AM
>>> > To: dev@airflow.apache.org
>>> > Subject: RE: [EXT] [DISCUSS] Turn "tests_common" into separate
>>> distribution for development
>>> >
>>> >
>>> > CAUTION: This email originated from outside of the organization. Do
>>> not click links or open attachments unless you can confirm the sender and
>>> know the content is safe.
>>> >
>>> >
>>> >
>>> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
>>> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous
>>> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
>>> certain que le contenu ne présente aucun risque.
>>> >
>>> >
>>> > I am doing a bit more cleanup, and I have found that the easier way to
>>> fix some of the remaining issues will be to clean-up (and remove) the
>>> remaining editable devel dependencies and incorporate them all in the
>>> "tests-common" package.
>>> >
>>> > You can take a look at the PR:
>>> https://github.com/apache/airflow/pull/47281 - but basically what it
>>> means is:
>>> >
>>> > * all "devel" dependencies are added as required dependencies of
>>> "tests-common" (except "doc" - I will treat doc separately).
>>> > * I removed all "legacy" extras from the "airflow" package including
>>> "bundle" extras: "devel-ci", "devel-db" and a few others - except
>>> installing "all" dependencies as they were pretty useless. Except
>>> "editable" all only - see below - we will have no more devel and bundle
>>> extras (Ash - I guess this is what you were looking forward to :) )
>>> > * Instead we have one "all" extra that is available only in editable
>>> mode - it's not documented in user documentation and it is really only
>>> useful to install everything with `pip` (with uv you get the same with `uv
>>> sync --all-extras`) - this is still used internally in the CI image to run
>>> `uv pip install .[all] --constraints` until we switch to use "uv.lock" in
>>> the future
>>> > * hatch_build.py is significantly simpler and easier to understand now
>>> - with all the bundle removal and moving all dependencies to ./tests-common
>>> > * we still need dynamic dependencies and ./hatch_build.py - but less
>>> and less, with PEP735 (https://peps.python.org/pep-0735/) implemented
>>> in pip in April we will likely be able to turn our optional dependencies
>>> into static pyproject.toml deps, and with
>>> https://peps.python.org/pep-0771/ (needs approval and implementation)
>>> we will likely be able to have static pyproject.toml required dependencies
>>> as well.
>>> > * I updated install and contributing docs to be "uv first" -
>>> presenting as recommended and the first option to go with `uv` - as it is
>>> becoming deceptively simple now to work with both - airflow and providers
>>> (and it will be even simpler after few next PRs)
>>> >
>>> > Now. THE BIG QUESTION - naming again. With all those changes.....
>>> `tests-common` is becoming more of a `devel-common` package - because what
>>> it will do - it will contribute to all other sub-projects all the
>>> development tooling that is needed for those other projects to be developed.
>>> >
>>> > Shall we name it "devel-common" instead of `tests-common`?
>>> >
>>> > Part of why I think it makes sense is this specification in
>>> `tests-common/pyproject.toml` (see attachment) - also
>>> https://ibb.co/1cwZQRm if you do not see attachment. Note that these
>>> are generally "common" devel dependencies - and each of the packages can
>>> contribute their own (including task-sdk, airflow-core (future), every
>>> provider etc.)
>>> >
>>> > [Screenshot from 2025-03-04 19-50-32.png]
>>> >
>>> >
>>> > J.
>>> >
>>> >
>>> >
>>> > On Mon, Mar 3, 2025 at 4:14 AM Jarek Potiuk <jarek@potiuk.com<mailto:
>>> jarek@potiuk.com>> wrote:
>>> > Hello everyone.
>>> >
>>> > I created the PR for that https://github.com/apache/airflow/pull/47281
>>> .
>>> >
>>> > It's even nicer than I anticipated. I love the new super-simple
>>> workflows this restructuring finally enabled.
>>> >
>>> > With `uv` and workspace, and the new structure of tests, developing
>>> and running tests for providers, task-sdk or any of the other future
>>> sub-projects becomes very, very straightforward, we avoid duplication of
>>> pytest options and switching between airflow, task-sdk, providers tests
>>> will be simple and straightforward.
>>> >
>>> > 1) First of all with this change, we remove `devel-tests` extra. It
>>> was always needed in local venv to install all test dependencies, But this
>>> is completely gone right now. Test dependencies are automatically installed
>>> now when you run `uv sync` - and you do not need to specify `--extra
>>> devel-tests`. If you are still using `pip` (I strongly recommend switching
>>> to uv) - you just install `pip install -e ./tests-common`.
>>> >
>>> > 2) the previous way of syncing and running tests in "everything
>>> installed" mode works as it worked before:
>>> >
>>> > uv sync --all-extras
>>> >
>>> > This installs all possible extras of airflow, allows you to run tests
>>> for all providers, activate the venv in `.venv` and run `pytests
>>> tests/always` or `pytest providers/mongo/tests` or `pytest task_sdk/tests
>>> and it should all work fine as it did before
>>> >
>>> > 3) you can also install dependencies of a selected provider (or a few
>>> of those) in your .venv
>>> >
>>> > uv sync --extra mongo
>>> >
>>> > 3) All the `breeze testing provider-tests --test-type
>>> "Providers[mongo]"" and generally all the ways to replicate what we run in
>>> CI inside breeze containers with selective checks will also continue to
>>> work - no changes here.
>>> >
>>> > But then, a little bit of standardization, a bit of pre-commits that
>>> automatically update `pyproject.toml` files of ours with cross-provider
>>> dependencies and a little of `uv` workspace magic sprinkled over the repo
>>> of ours and now we can do something much more straightforward:
>>> >
>>> > You can change directory to task_sdk, or provider folder of your
>>> choice and simply run:
>>> >
>>> > cd providers/mongo
>>> > uv run pytest
>>> >
>>> > or
>>> >
>>> > cd task_sdk
>>> > uv run pytest
>>> >
>>> > And it will do what you would expect to do - even if this is the first
>>> thing you do after checking out Airflow's repo without any earlier syncing
>>> or preparation. Yes. It's that simple.
>>> >
>>> > The `uv run pytest` command will now automatically: install python,
>>> create a venv, sync your venv to ONLY have dependencies needed for your
>>> provider or any of the providers imported by your provider, and all test
>>> dependencies needed, it will find only non-system, non-integration tests
>>> and run them. That's it. If you want to only develop one provider - you
>>> will not have to worry any more about breeze image or setting up all other
>>> dependencies.
>>> >
>>> > Go ahead and try it. It's unbelievably simple.
>>> >
>>> > J.
>>> >
>>> >
>>> >
>>> > On Tue, Feb 25, 2025 at 5:45 PM Ash Berlin-Taylor <ash@apache.org
>>> <ma...@apache.org>> wrote:
>>> > I like this approach, lets do it!
>>> >
>>> > > On 25 Feb 2025, at 16:02, Jarek Potiuk <jarek@potiuk.com<mailto:
>>> jarek@potiuk.com>> wrote:
>>> > >
>>> > > airflow-core
>>> > > task-sdk
>>> > > tests-common
>>> >
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>
>>>
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Jarek Potiuk <ja...@potiuk.com>.
The PR is now merged,,, we have 'devel-common` now.
The important bit now for everyone... It's easier than ever to have dev-env
running for airflow...
1) uv sync -> and your .venv env allows you to run all tests for
airflow/task-sdk
2) uv sync --all-packages -> and your .venv allows you to also run tests
for all providers
3)
cd providers/PROVIDER
uv sync
gives env ready to run all tests for the provider of your choice
uv run pytest
does it.
There are likely a few missing per-provider deps but we will find and
squash them soon !
Have fun!
J.
On Wed, Mar 5, 2025 at 12:48 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Seems like the PR is getting to "green" zone - so one last push - and I am
> changing it to "devel-common" unless I hear strong NOOOOOOO!
>
> On Tue, Mar 4, 2025 at 10:36 PM Vincent Beck <vi...@apache.org> wrote:
>
>> devel-common makes sense to me
>>
>> On 2025/03/04 21:13:47 "Oliveira, Niko" wrote:
>> > +1 to devel-common from me
>> >
>> > ________________________________
>> > From: Ferruzzi, Dennis <fe...@amazon.com.INVALID>
>> > Sent: Tuesday, March 4, 2025 11:21:20 AM
>> > To: dev@airflow.apache.org
>> > Subject: RE: [EXT] [DISCUSS] Turn "tests_common" into separate
>> distribution for development
>> >
>> > CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>> >
>> >
>> >
>> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
>> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous
>> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
>> certain que le contenu ne présente aucun risque.
>> >
>> >
>> >
>> > devel-common sounds reasonable
>> >
>> >
>> > - ferruzzi
>> >
>> >
>> > ________________________________
>> > From: Jarek Potiuk <ja...@potiuk.com>
>> > Sent: Tuesday, March 4, 2025 10:53 AM
>> > To: dev@airflow.apache.org
>> > Subject: RE: [EXT] [DISCUSS] Turn "tests_common" into separate
>> distribution for development
>> >
>> >
>> > CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>> >
>> >
>> >
>> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
>> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous
>> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
>> certain que le contenu ne présente aucun risque.
>> >
>> >
>> > I am doing a bit more cleanup, and I have found that the easier way to
>> fix some of the remaining issues will be to clean-up (and remove) the
>> remaining editable devel dependencies and incorporate them all in the
>> "tests-common" package.
>> >
>> > You can take a look at the PR:
>> https://github.com/apache/airflow/pull/47281 - but basically what it
>> means is:
>> >
>> > * all "devel" dependencies are added as required dependencies of
>> "tests-common" (except "doc" - I will treat doc separately).
>> > * I removed all "legacy" extras from the "airflow" package including
>> "bundle" extras: "devel-ci", "devel-db" and a few others - except
>> installing "all" dependencies as they were pretty useless. Except
>> "editable" all only - see below - we will have no more devel and bundle
>> extras (Ash - I guess this is what you were looking forward to :) )
>> > * Instead we have one "all" extra that is available only in editable
>> mode - it's not documented in user documentation and it is really only
>> useful to install everything with `pip` (with uv you get the same with `uv
>> sync --all-extras`) - this is still used internally in the CI image to run
>> `uv pip install .[all] --constraints` until we switch to use "uv.lock" in
>> the future
>> > * hatch_build.py is significantly simpler and easier to understand now
>> - with all the bundle removal and moving all dependencies to ./tests-common
>> > * we still need dynamic dependencies and ./hatch_build.py - but less
>> and less, with PEP735 (https://peps.python.org/pep-0735/) implemented in
>> pip in April we will likely be able to turn our optional dependencies into
>> static pyproject.toml deps, and with https://peps.python.org/pep-0771/
>> (needs approval and implementation) we will likely be able to have static
>> pyproject.toml required dependencies as well.
>> > * I updated install and contributing docs to be "uv first" - presenting
>> as recommended and the first option to go with `uv` - as it is becoming
>> deceptively simple now to work with both - airflow and providers (and it
>> will be even simpler after few next PRs)
>> >
>> > Now. THE BIG QUESTION - naming again. With all those changes.....
>> `tests-common` is becoming more of a `devel-common` package - because what
>> it will do - it will contribute to all other sub-projects all the
>> development tooling that is needed for those other projects to be developed.
>> >
>> > Shall we name it "devel-common" instead of `tests-common`?
>> >
>> > Part of why I think it makes sense is this specification in
>> `tests-common/pyproject.toml` (see attachment) - also
>> https://ibb.co/1cwZQRm if you do not see attachment. Note that these are
>> generally "common" devel dependencies - and each of the packages can
>> contribute their own (including task-sdk, airflow-core (future), every
>> provider etc.)
>> >
>> > [Screenshot from 2025-03-04 19-50-32.png]
>> >
>> >
>> > J.
>> >
>> >
>> >
>> > On Mon, Mar 3, 2025 at 4:14 AM Jarek Potiuk <jarek@potiuk.com<mailto:
>> jarek@potiuk.com>> wrote:
>> > Hello everyone.
>> >
>> > I created the PR for that https://github.com/apache/airflow/pull/47281.
>> >
>> > It's even nicer than I anticipated. I love the new super-simple
>> workflows this restructuring finally enabled.
>> >
>> > With `uv` and workspace, and the new structure of tests, developing and
>> running tests for providers, task-sdk or any of the other future
>> sub-projects becomes very, very straightforward, we avoid duplication of
>> pytest options and switching between airflow, task-sdk, providers tests
>> will be simple and straightforward.
>> >
>> > 1) First of all with this change, we remove `devel-tests` extra. It was
>> always needed in local venv to install all test dependencies, But this is
>> completely gone right now. Test dependencies are automatically installed
>> now when you run `uv sync` - and you do not need to specify `--extra
>> devel-tests`. If you are still using `pip` (I strongly recommend switching
>> to uv) - you just install `pip install -e ./tests-common`.
>> >
>> > 2) the previous way of syncing and running tests in "everything
>> installed" mode works as it worked before:
>> >
>> > uv sync --all-extras
>> >
>> > This installs all possible extras of airflow, allows you to run tests
>> for all providers, activate the venv in `.venv` and run `pytests
>> tests/always` or `pytest providers/mongo/tests` or `pytest task_sdk/tests
>> and it should all work fine as it did before
>> >
>> > 3) you can also install dependencies of a selected provider (or a few
>> of those) in your .venv
>> >
>> > uv sync --extra mongo
>> >
>> > 3) All the `breeze testing provider-tests --test-type
>> "Providers[mongo]"" and generally all the ways to replicate what we run in
>> CI inside breeze containers with selective checks will also continue to
>> work - no changes here.
>> >
>> > But then, a little bit of standardization, a bit of pre-commits that
>> automatically update `pyproject.toml` files of ours with cross-provider
>> dependencies and a little of `uv` workspace magic sprinkled over the repo
>> of ours and now we can do something much more straightforward:
>> >
>> > You can change directory to task_sdk, or provider folder of your choice
>> and simply run:
>> >
>> > cd providers/mongo
>> > uv run pytest
>> >
>> > or
>> >
>> > cd task_sdk
>> > uv run pytest
>> >
>> > And it will do what you would expect to do - even if this is the first
>> thing you do after checking out Airflow's repo without any earlier syncing
>> or preparation. Yes. It's that simple.
>> >
>> > The `uv run pytest` command will now automatically: install python,
>> create a venv, sync your venv to ONLY have dependencies needed for your
>> provider or any of the providers imported by your provider, and all test
>> dependencies needed, it will find only non-system, non-integration tests
>> and run them. That's it. If you want to only develop one provider - you
>> will not have to worry any more about breeze image or setting up all other
>> dependencies.
>> >
>> > Go ahead and try it. It's unbelievably simple.
>> >
>> > J.
>> >
>> >
>> >
>> > On Tue, Feb 25, 2025 at 5:45 PM Ash Berlin-Taylor <ash@apache.org
>> <ma...@apache.org>> wrote:
>> > I like this approach, lets do it!
>> >
>> > > On 25 Feb 2025, at 16:02, Jarek Potiuk <jarek@potiuk.com<mailto:
>> jarek@potiuk.com>> wrote:
>> > >
>> > > airflow-core
>> > > task-sdk
>> > > tests-common
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> For additional commands, e-mail: dev-help@airflow.apache.org
>>
>>
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Jarek Potiuk <ja...@potiuk.com>.
Seems like the PR is getting to "green" zone - so one last push - and I am
changing it to "devel-common" unless I hear strong NOOOOOOO!
On Tue, Mar 4, 2025 at 10:36 PM Vincent Beck <vi...@apache.org> wrote:
> devel-common makes sense to me
>
> On 2025/03/04 21:13:47 "Oliveira, Niko" wrote:
> > +1 to devel-common from me
> >
> > ________________________________
> > From: Ferruzzi, Dennis <fe...@amazon.com.INVALID>
> > Sent: Tuesday, March 4, 2025 11:21:20 AM
> > To: dev@airflow.apache.org
> > Subject: RE: [EXT] [DISCUSS] Turn "tests_common" into separate
> distribution for development
> >
> > CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
> >
> >
> >
> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous
> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> certain que le contenu ne présente aucun risque.
> >
> >
> >
> > devel-common sounds reasonable
> >
> >
> > - ferruzzi
> >
> >
> > ________________________________
> > From: Jarek Potiuk <ja...@potiuk.com>
> > Sent: Tuesday, March 4, 2025 10:53 AM
> > To: dev@airflow.apache.org
> > Subject: RE: [EXT] [DISCUSS] Turn "tests_common" into separate
> distribution for development
> >
> >
> > CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
> >
> >
> >
> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous
> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> certain que le contenu ne présente aucun risque.
> >
> >
> > I am doing a bit more cleanup, and I have found that the easier way to
> fix some of the remaining issues will be to clean-up (and remove) the
> remaining editable devel dependencies and incorporate them all in the
> "tests-common" package.
> >
> > You can take a look at the PR:
> https://github.com/apache/airflow/pull/47281 - but basically what it
> means is:
> >
> > * all "devel" dependencies are added as required dependencies of
> "tests-common" (except "doc" - I will treat doc separately).
> > * I removed all "legacy" extras from the "airflow" package including
> "bundle" extras: "devel-ci", "devel-db" and a few others - except
> installing "all" dependencies as they were pretty useless. Except
> "editable" all only - see below - we will have no more devel and bundle
> extras (Ash - I guess this is what you were looking forward to :) )
> > * Instead we have one "all" extra that is available only in editable
> mode - it's not documented in user documentation and it is really only
> useful to install everything with `pip` (with uv you get the same with `uv
> sync --all-extras`) - this is still used internally in the CI image to run
> `uv pip install .[all] --constraints` until we switch to use "uv.lock" in
> the future
> > * hatch_build.py is significantly simpler and easier to understand now -
> with all the bundle removal and moving all dependencies to ./tests-common
> > * we still need dynamic dependencies and ./hatch_build.py - but less and
> less, with PEP735 (https://peps.python.org/pep-0735/) implemented in pip
> in April we will likely be able to turn our optional dependencies into
> static pyproject.toml deps, and with https://peps.python.org/pep-0771/
> (needs approval and implementation) we will likely be able to have static
> pyproject.toml required dependencies as well.
> > * I updated install and contributing docs to be "uv first" - presenting
> as recommended and the first option to go with `uv` - as it is becoming
> deceptively simple now to work with both - airflow and providers (and it
> will be even simpler after few next PRs)
> >
> > Now. THE BIG QUESTION - naming again. With all those changes.....
> `tests-common` is becoming more of a `devel-common` package - because what
> it will do - it will contribute to all other sub-projects all the
> development tooling that is needed for those other projects to be developed.
> >
> > Shall we name it "devel-common" instead of `tests-common`?
> >
> > Part of why I think it makes sense is this specification in
> `tests-common/pyproject.toml` (see attachment) - also
> https://ibb.co/1cwZQRm if you do not see attachment. Note that these are
> generally "common" devel dependencies - and each of the packages can
> contribute their own (including task-sdk, airflow-core (future), every
> provider etc.)
> >
> > [Screenshot from 2025-03-04 19-50-32.png]
> >
> >
> > J.
> >
> >
> >
> > On Mon, Mar 3, 2025 at 4:14 AM Jarek Potiuk <jarek@potiuk.com<mailto:
> jarek@potiuk.com>> wrote:
> > Hello everyone.
> >
> > I created the PR for that https://github.com/apache/airflow/pull/47281.
> >
> > It's even nicer than I anticipated. I love the new super-simple
> workflows this restructuring finally enabled.
> >
> > With `uv` and workspace, and the new structure of tests, developing and
> running tests for providers, task-sdk or any of the other future
> sub-projects becomes very, very straightforward, we avoid duplication of
> pytest options and switching between airflow, task-sdk, providers tests
> will be simple and straightforward.
> >
> > 1) First of all with this change, we remove `devel-tests` extra. It was
> always needed in local venv to install all test dependencies, But this is
> completely gone right now. Test dependencies are automatically installed
> now when you run `uv sync` - and you do not need to specify `--extra
> devel-tests`. If you are still using `pip` (I strongly recommend switching
> to uv) - you just install `pip install -e ./tests-common`.
> >
> > 2) the previous way of syncing and running tests in "everything
> installed" mode works as it worked before:
> >
> > uv sync --all-extras
> >
> > This installs all possible extras of airflow, allows you to run tests
> for all providers, activate the venv in `.venv` and run `pytests
> tests/always` or `pytest providers/mongo/tests` or `pytest task_sdk/tests
> and it should all work fine as it did before
> >
> > 3) you can also install dependencies of a selected provider (or a few of
> those) in your .venv
> >
> > uv sync --extra mongo
> >
> > 3) All the `breeze testing provider-tests --test-type
> "Providers[mongo]"" and generally all the ways to replicate what we run in
> CI inside breeze containers with selective checks will also continue to
> work - no changes here.
> >
> > But then, a little bit of standardization, a bit of pre-commits that
> automatically update `pyproject.toml` files of ours with cross-provider
> dependencies and a little of `uv` workspace magic sprinkled over the repo
> of ours and now we can do something much more straightforward:
> >
> > You can change directory to task_sdk, or provider folder of your choice
> and simply run:
> >
> > cd providers/mongo
> > uv run pytest
> >
> > or
> >
> > cd task_sdk
> > uv run pytest
> >
> > And it will do what you would expect to do - even if this is the first
> thing you do after checking out Airflow's repo without any earlier syncing
> or preparation. Yes. It's that simple.
> >
> > The `uv run pytest` command will now automatically: install python,
> create a venv, sync your venv to ONLY have dependencies needed for your
> provider or any of the providers imported by your provider, and all test
> dependencies needed, it will find only non-system, non-integration tests
> and run them. That's it. If you want to only develop one provider - you
> will not have to worry any more about breeze image or setting up all other
> dependencies.
> >
> > Go ahead and try it. It's unbelievably simple.
> >
> > J.
> >
> >
> >
> > On Tue, Feb 25, 2025 at 5:45 PM Ash Berlin-Taylor <ash@apache.org
> <ma...@apache.org>> wrote:
> > I like this approach, lets do it!
> >
> > > On 25 Feb 2025, at 16:02, Jarek Potiuk <jarek@potiuk.com<mailto:
> jarek@potiuk.com>> wrote:
> > >
> > > airflow-core
> > > task-sdk
> > > tests-common
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Vincent Beck <vi...@apache.org>.
devel-common makes sense to me
On 2025/03/04 21:13:47 "Oliveira, Niko" wrote:
> +1 to devel-common from me
>
> ________________________________
> From: Ferruzzi, Dennis <fe...@amazon.com.INVALID>
> Sent: Tuesday, March 4, 2025 11:21:20 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXT] [DISCUSS] Turn "tests_common" into separate distribution for development
>
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.
>
>
>
> devel-common sounds reasonable
>
>
> - ferruzzi
>
>
> ________________________________
> From: Jarek Potiuk <ja...@potiuk.com>
> Sent: Tuesday, March 4, 2025 10:53 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXT] [DISCUSS] Turn "tests_common" into separate distribution for development
>
>
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.
>
>
> I am doing a bit more cleanup, and I have found that the easier way to fix some of the remaining issues will be to clean-up (and remove) the remaining editable devel dependencies and incorporate them all in the "tests-common" package.
>
> You can take a look at the PR: https://github.com/apache/airflow/pull/47281 - but basically what it means is:
>
> * all "devel" dependencies are added as required dependencies of "tests-common" (except "doc" - I will treat doc separately).
> * I removed all "legacy" extras from the "airflow" package including "bundle" extras: "devel-ci", "devel-db" and a few others - except installing "all" dependencies as they were pretty useless. Except "editable" all only - see below - we will have no more devel and bundle extras (Ash - I guess this is what you were looking forward to :) )
> * Instead we have one "all" extra that is available only in editable mode - it's not documented in user documentation and it is really only useful to install everything with `pip` (with uv you get the same with `uv sync --all-extras`) - this is still used internally in the CI image to run `uv pip install .[all] --constraints` until we switch to use "uv.lock" in the future
> * hatch_build.py is significantly simpler and easier to understand now - with all the bundle removal and moving all dependencies to ./tests-common
> * we still need dynamic dependencies and ./hatch_build.py - but less and less, with PEP735 (https://peps.python.org/pep-0735/) implemented in pip in April we will likely be able to turn our optional dependencies into static pyproject.toml deps, and with https://peps.python.org/pep-0771/ (needs approval and implementation) we will likely be able to have static pyproject.toml required dependencies as well.
> * I updated install and contributing docs to be "uv first" - presenting as recommended and the first option to go with `uv` - as it is becoming deceptively simple now to work with both - airflow and providers (and it will be even simpler after few next PRs)
>
> Now. THE BIG QUESTION - naming again. With all those changes..... `tests-common` is becoming more of a `devel-common` package - because what it will do - it will contribute to all other sub-projects all the development tooling that is needed for those other projects to be developed.
>
> Shall we name it "devel-common" instead of `tests-common`?
>
> Part of why I think it makes sense is this specification in `tests-common/pyproject.toml` (see attachment) - also https://ibb.co/1cwZQRm if you do not see attachment. Note that these are generally "common" devel dependencies - and each of the packages can contribute their own (including task-sdk, airflow-core (future), every provider etc.)
>
> [Screenshot from 2025-03-04 19-50-32.png]
>
>
> J.
>
>
>
> On Mon, Mar 3, 2025 at 4:14 AM Jarek Potiuk <ja...@potiuk.com>> wrote:
> Hello everyone.
>
> I created the PR for that https://github.com/apache/airflow/pull/47281.
>
> It's even nicer than I anticipated. I love the new super-simple workflows this restructuring finally enabled.
>
> With `uv` and workspace, and the new structure of tests, developing and running tests for providers, task-sdk or any of the other future sub-projects becomes very, very straightforward, we avoid duplication of pytest options and switching between airflow, task-sdk, providers tests will be simple and straightforward.
>
> 1) First of all with this change, we remove `devel-tests` extra. It was always needed in local venv to install all test dependencies, But this is completely gone right now. Test dependencies are automatically installed now when you run `uv sync` - and you do not need to specify `--extra devel-tests`. If you are still using `pip` (I strongly recommend switching to uv) - you just install `pip install -e ./tests-common`.
>
> 2) the previous way of syncing and running tests in "everything installed" mode works as it worked before:
>
> uv sync --all-extras
>
> This installs all possible extras of airflow, allows you to run tests for all providers, activate the venv in `.venv` and run `pytests tests/always` or `pytest providers/mongo/tests` or `pytest task_sdk/tests and it should all work fine as it did before
>
> 3) you can also install dependencies of a selected provider (or a few of those) in your .venv
>
> uv sync --extra mongo
>
> 3) All the `breeze testing provider-tests --test-type "Providers[mongo]"" and generally all the ways to replicate what we run in CI inside breeze containers with selective checks will also continue to work - no changes here.
>
> But then, a little bit of standardization, a bit of pre-commits that automatically update `pyproject.toml` files of ours with cross-provider dependencies and a little of `uv` workspace magic sprinkled over the repo of ours and now we can do something much more straightforward:
>
> You can change directory to task_sdk, or provider folder of your choice and simply run:
>
> cd providers/mongo
> uv run pytest
>
> or
>
> cd task_sdk
> uv run pytest
>
> And it will do what you would expect to do - even if this is the first thing you do after checking out Airflow's repo without any earlier syncing or preparation. Yes. It's that simple.
>
> The `uv run pytest` command will now automatically: install python, create a venv, sync your venv to ONLY have dependencies needed for your provider or any of the providers imported by your provider, and all test dependencies needed, it will find only non-system, non-integration tests and run them. That's it. If you want to only develop one provider - you will not have to worry any more about breeze image or setting up all other dependencies.
>
> Go ahead and try it. It's unbelievably simple.
>
> J.
>
>
>
> On Tue, Feb 25, 2025 at 5:45 PM Ash Berlin-Taylor <as...@apache.org>> wrote:
> I like this approach, lets do it!
>
> > On 25 Feb 2025, at 16:02, Jarek Potiuk <ja...@potiuk.com>> wrote:
> >
> > airflow-core
> > task-sdk
> > tests-common
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by "Oliveira, Niko" <on...@amazon.com.INVALID>.
+1 to devel-common from me
________________________________
From: Ferruzzi, Dennis <fe...@amazon.com.INVALID>
Sent: Tuesday, March 4, 2025 11:21:20 AM
To: dev@airflow.apache.org
Subject: RE: [EXT] [DISCUSS] Turn "tests_common" into separate distribution for development
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.
devel-common sounds reasonable
- ferruzzi
________________________________
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Tuesday, March 4, 2025 10:53 AM
To: dev@airflow.apache.org
Subject: RE: [EXT] [DISCUSS] Turn "tests_common" into separate distribution for development
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.
I am doing a bit more cleanup, and I have found that the easier way to fix some of the remaining issues will be to clean-up (and remove) the remaining editable devel dependencies and incorporate them all in the "tests-common" package.
You can take a look at the PR: https://github.com/apache/airflow/pull/47281 - but basically what it means is:
* all "devel" dependencies are added as required dependencies of "tests-common" (except "doc" - I will treat doc separately).
* I removed all "legacy" extras from the "airflow" package including "bundle" extras: "devel-ci", "devel-db" and a few others - except installing "all" dependencies as they were pretty useless. Except "editable" all only - see below - we will have no more devel and bundle extras (Ash - I guess this is what you were looking forward to :) )
* Instead we have one "all" extra that is available only in editable mode - it's not documented in user documentation and it is really only useful to install everything with `pip` (with uv you get the same with `uv sync --all-extras`) - this is still used internally in the CI image to run `uv pip install .[all] --constraints` until we switch to use "uv.lock" in the future
* hatch_build.py is significantly simpler and easier to understand now - with all the bundle removal and moving all dependencies to ./tests-common
* we still need dynamic dependencies and ./hatch_build.py - but less and less, with PEP735 (https://peps.python.org/pep-0735/) implemented in pip in April we will likely be able to turn our optional dependencies into static pyproject.toml deps, and with https://peps.python.org/pep-0771/ (needs approval and implementation) we will likely be able to have static pyproject.toml required dependencies as well.
* I updated install and contributing docs to be "uv first" - presenting as recommended and the first option to go with `uv` - as it is becoming deceptively simple now to work with both - airflow and providers (and it will be even simpler after few next PRs)
Now. THE BIG QUESTION - naming again. With all those changes..... `tests-common` is becoming more of a `devel-common` package - because what it will do - it will contribute to all other sub-projects all the development tooling that is needed for those other projects to be developed.
Shall we name it "devel-common" instead of `tests-common`?
Part of why I think it makes sense is this specification in `tests-common/pyproject.toml` (see attachment) - also https://ibb.co/1cwZQRm if you do not see attachment. Note that these are generally "common" devel dependencies - and each of the packages can contribute their own (including task-sdk, airflow-core (future), every provider etc.)
[Screenshot from 2025-03-04 19-50-32.png]
J.
On Mon, Mar 3, 2025 at 4:14 AM Jarek Potiuk <ja...@potiuk.com>> wrote:
Hello everyone.
I created the PR for that https://github.com/apache/airflow/pull/47281.
It's even nicer than I anticipated. I love the new super-simple workflows this restructuring finally enabled.
With `uv` and workspace, and the new structure of tests, developing and running tests for providers, task-sdk or any of the other future sub-projects becomes very, very straightforward, we avoid duplication of pytest options and switching between airflow, task-sdk, providers tests will be simple and straightforward.
1) First of all with this change, we remove `devel-tests` extra. It was always needed in local venv to install all test dependencies, But this is completely gone right now. Test dependencies are automatically installed now when you run `uv sync` - and you do not need to specify `--extra devel-tests`. If you are still using `pip` (I strongly recommend switching to uv) - you just install `pip install -e ./tests-common`.
2) the previous way of syncing and running tests in "everything installed" mode works as it worked before:
uv sync --all-extras
This installs all possible extras of airflow, allows you to run tests for all providers, activate the venv in `.venv` and run `pytests tests/always` or `pytest providers/mongo/tests` or `pytest task_sdk/tests and it should all work fine as it did before
3) you can also install dependencies of a selected provider (or a few of those) in your .venv
uv sync --extra mongo
3) All the `breeze testing provider-tests --test-type "Providers[mongo]"" and generally all the ways to replicate what we run in CI inside breeze containers with selective checks will also continue to work - no changes here.
But then, a little bit of standardization, a bit of pre-commits that automatically update `pyproject.toml` files of ours with cross-provider dependencies and a little of `uv` workspace magic sprinkled over the repo of ours and now we can do something much more straightforward:
You can change directory to task_sdk, or provider folder of your choice and simply run:
cd providers/mongo
uv run pytest
or
cd task_sdk
uv run pytest
And it will do what you would expect to do - even if this is the first thing you do after checking out Airflow's repo without any earlier syncing or preparation. Yes. It's that simple.
The `uv run pytest` command will now automatically: install python, create a venv, sync your venv to ONLY have dependencies needed for your provider or any of the providers imported by your provider, and all test dependencies needed, it will find only non-system, non-integration tests and run them. That's it. If you want to only develop one provider - you will not have to worry any more about breeze image or setting up all other dependencies.
Go ahead and try it. It's unbelievably simple.
J.
On Tue, Feb 25, 2025 at 5:45 PM Ash Berlin-Taylor <as...@apache.org>> wrote:
I like this approach, lets do it!
> On 25 Feb 2025, at 16:02, Jarek Potiuk <ja...@potiuk.com>> wrote:
>
> airflow-core
> task-sdk
> tests-common
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by "Ferruzzi, Dennis" <fe...@amazon.com.INVALID>.
devel-common sounds reasonable
- ferruzzi
________________________________
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Tuesday, March 4, 2025 10:53 AM
To: dev@airflow.apache.org
Subject: RE: [EXT] [DISCUSS] Turn "tests_common" into separate distribution for development
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.
I am doing a bit more cleanup, and I have found that the easier way to fix some of the remaining issues will be to clean-up (and remove) the remaining editable devel dependencies and incorporate them all in the "tests-common" package.
You can take a look at the PR: https://github.com/apache/airflow/pull/47281 - but basically what it means is:
* all "devel" dependencies are added as required dependencies of "tests-common" (except "doc" - I will treat doc separately).
* I removed all "legacy" extras from the "airflow" package including "bundle" extras: "devel-ci", "devel-db" and a few others - except installing "all" dependencies as they were pretty useless. Except "editable" all only - see below - we will have no more devel and bundle extras (Ash - I guess this is what you were looking forward to :) )
* Instead we have one "all" extra that is available only in editable mode - it's not documented in user documentation and it is really only useful to install everything with `pip` (with uv you get the same with `uv sync --all-extras`) - this is still used internally in the CI image to run `uv pip install .[all] --constraints` until we switch to use "uv.lock" in the future
* hatch_build.py is significantly simpler and easier to understand now - with all the bundle removal and moving all dependencies to ./tests-common
* we still need dynamic dependencies and ./hatch_build.py - but less and less, with PEP735 (https://peps.python.org/pep-0735/) implemented in pip in April we will likely be able to turn our optional dependencies into static pyproject.toml deps, and with https://peps.python.org/pep-0771/ (needs approval and implementation) we will likely be able to have static pyproject.toml required dependencies as well.
* I updated install and contributing docs to be "uv first" - presenting as recommended and the first option to go with `uv` - as it is becoming deceptively simple now to work with both - airflow and providers (and it will be even simpler after few next PRs)
Now. THE BIG QUESTION - naming again. With all those changes..... `tests-common` is becoming more of a `devel-common` package - because what it will do - it will contribute to all other sub-projects all the development tooling that is needed for those other projects to be developed.
Shall we name it "devel-common" instead of `tests-common`?
Part of why I think it makes sense is this specification in `tests-common/pyproject.toml` (see attachment) - also https://ibb.co/1cwZQRm if you do not see attachment. Note that these are generally "common" devel dependencies - and each of the packages can contribute their own (including task-sdk, airflow-core (future), every provider etc.)
[Screenshot from 2025-03-04 19-50-32.png]
J.
On Mon, Mar 3, 2025 at 4:14 AM Jarek Potiuk <ja...@potiuk.com>> wrote:
Hello everyone.
I created the PR for that https://github.com/apache/airflow/pull/47281.
It's even nicer than I anticipated. I love the new super-simple workflows this restructuring finally enabled.
With `uv` and workspace, and the new structure of tests, developing and running tests for providers, task-sdk or any of the other future sub-projects becomes very, very straightforward, we avoid duplication of pytest options and switching between airflow, task-sdk, providers tests will be simple and straightforward.
1) First of all with this change, we remove `devel-tests` extra. It was always needed in local venv to install all test dependencies, But this is completely gone right now. Test dependencies are automatically installed now when you run `uv sync` - and you do not need to specify `--extra devel-tests`. If you are still using `pip` (I strongly recommend switching to uv) - you just install `pip install -e ./tests-common`.
2) the previous way of syncing and running tests in "everything installed" mode works as it worked before:
uv sync --all-extras
This installs all possible extras of airflow, allows you to run tests for all providers, activate the venv in `.venv` and run `pytests tests/always` or `pytest providers/mongo/tests` or `pytest task_sdk/tests and it should all work fine as it did before
3) you can also install dependencies of a selected provider (or a few of those) in your .venv
uv sync --extra mongo
3) All the `breeze testing provider-tests --test-type "Providers[mongo]"" and generally all the ways to replicate what we run in CI inside breeze containers with selective checks will also continue to work - no changes here.
But then, a little bit of standardization, a bit of pre-commits that automatically update `pyproject.toml` files of ours with cross-provider dependencies and a little of `uv` workspace magic sprinkled over the repo of ours and now we can do something much more straightforward:
You can change directory to task_sdk, or provider folder of your choice and simply run:
cd providers/mongo
uv run pytest
or
cd task_sdk
uv run pytest
And it will do what you would expect to do - even if this is the first thing you do after checking out Airflow's repo without any earlier syncing or preparation. Yes. It's that simple.
The `uv run pytest` command will now automatically: install python, create a venv, sync your venv to ONLY have dependencies needed for your provider or any of the providers imported by your provider, and all test dependencies needed, it will find only non-system, non-integration tests and run them. That's it. If you want to only develop one provider - you will not have to worry any more about breeze image or setting up all other dependencies.
Go ahead and try it. It's unbelievably simple.
J.
On Tue, Feb 25, 2025 at 5:45 PM Ash Berlin-Taylor <as...@apache.org>> wrote:
I like this approach, lets do it!
> On 25 Feb 2025, at 16:02, Jarek Potiuk <ja...@potiuk.com>> wrote:
>
> airflow-core
> task-sdk
> tests-common
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Jarek Potiuk <ja...@potiuk.com>.
I am doing a bit more cleanup, and I have found that the easier way to fix
some of the remaining issues will be to clean-up (and remove) the remaining
editable devel dependencies and incorporate them all in the "tests-common"
package.
You can take a look at the PR: https://github.com/apache/airflow/pull/47281 -
but basically what it means is:
* all "devel" dependencies are added as required dependencies of
"tests-common" (except "doc" - I will treat doc separately).
* I removed all "legacy" extras from the "airflow" package including
"bundle" extras: "devel-ci", "devel-db" and a few others - except
installing "all" dependencies as they were pretty useless. Except
"editable" all only - see below - we will have no more devel and bundle
extras (Ash - I guess this is what you were looking forward to :) )
* Instead we have one "all" extra that is available only in editable mode -
it's not documented in user documentation and it is really only useful to
install everything with `pip` (with uv you get the same with `uv sync
--all-extras`) - this is still used internally in the CI image to run `uv
pip install .[all] --constraints` until we switch to use "uv.lock" in the
future
* hatch_build.py is significantly simpler and easier to understand now -
with all the bundle removal and moving all dependencies to ./tests-common
* we still need dynamic dependencies and ./hatch_build.py - but less and
less, with PEP735 (https://peps.python.org/pep-0735/) implemented in pip in
April we will likely be able to turn our optional dependencies into static
pyproject.toml deps, and with https://peps.python.org/pep-0771/ (needs
approval and implementation) we will likely be able to have static
pyproject.toml required dependencies as well.
* I updated install and contributing docs to be "uv first" - presenting as
recommended and the first option to go with `uv` - as it is becoming
deceptively simple now to work with both - airflow and providers (and it
will be even simpler after few next PRs)
Now. THE BIG QUESTION - naming again. With all those changes.....
`tests-common` is becoming more of a `devel-common` package - because what
it will do - it will contribute to all other sub-projects all the
development tooling that is needed for those other projects to be
developed.
*Shall we name it "devel-common" instead of `tests-common`? *
Part of why I think it makes sense is this specification in
`tests-common/pyproject.toml` (see attachment) - also https://ibb.co/1cwZQRm
if you do not see attachment. Note that these are generally "common" devel
dependencies - and each of the packages can contribute their own (including
task-sdk, airflow-core (future), every provider etc.)
[image: Screenshot from 2025-03-04 19-50-32.png]
J.
On Mon, Mar 3, 2025 at 4:14 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hello everyone.
>
> I created the PR for that https://github.com/apache/airflow/pull/47281.
>
> It's even nicer than I anticipated. I love the new super-simple workflows
> this restructuring finally enabled.
>
> With `uv` and workspace, and the new structure of tests, developing and
> running tests for providers, task-sdk or any of the other future
> sub-projects becomes very, very straightforward, we avoid duplication of
> pytest options and switching between airflow, task-sdk, providers tests
> will be simple and straightforward.
>
> 1) First of all with this change, we remove `devel-tests` extra. It was
> always needed in local venv to install all test dependencies, But this is
> completely gone right now. Test dependencies are automatically installed
> now when you run `uv sync` - and you do not need to specify `--extra
> devel-tests`. If you are still using `pip` (I strongly recommend switching
> to uv) - you just install `pip install -e ./tests-common`.
>
> 2) the previous way of syncing and running tests in "everything installed"
> mode works as it worked before:
>
> uv sync --all-extras
>
> This installs all possible extras of airflow, allows you to run tests for
> all providers, activate the venv in `.venv` and run `pytests tests/always`
> or `pytest providers/mongo/tests` or `pytest task_sdk/tests and it should
> all work fine as it did before
>
> 3) you can also install dependencies of a selected provider (or a few of
> those) in your .venv
>
> uv sync --extra mongo
>
> 3) All the `breeze testing provider-tests --test-type "Providers[mongo]""
> and generally all the ways to replicate what we run in CI inside breeze
> containers with selective checks will also continue to work - no changes
> here.
>
> But then, a little bit of standardization, a bit of pre-commits that
> automatically update `pyproject.toml` files of ours with cross-provider
> dependencies and a little of `uv` workspace magic sprinkled over the repo
> of ours and now we can do something much more straightforward:
>
> *You can change directory to task_sdk, or provider folder of your choice
> and simply run:*
>
> *cd providers/mongo*
> *uv run pytest*
>
> or
>
> *cd task_sdk*
> *uv run pytest *
>
> And it will do what you would expect to do - even if this is the first
> thing you do after checking out Airflow's repo without any earlier syncing
> or preparation. Yes. It's that simple.
>
> The `uv run pytest` command will now automatically: install python, create
> a venv, sync your venv to ONLY have dependencies needed for your provider
> or any of the providers imported by your provider, and all test
> dependencies needed, it will find only non-system, non-integration tests
> and run them. That's it. If you want to only develop one provider - you
> will not have to worry any more about breeze image or setting up all other
> dependencies.
>
> Go ahead and try it. It's unbelievably simple.
>
> J.
>
>
>
> On Tue, Feb 25, 2025 at 5:45 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
>> I like this approach, lets do it!
>>
>> > On 25 Feb 2025, at 16:02, Jarek Potiuk <ja...@potiuk.com> wrote:
>> >
>> > airflow-core
>> > task-sdk
>> > tests-common
>>
>>
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Jarek Potiuk <ja...@potiuk.com>.
Hello everyone.
I created the PR for that https://github.com/apache/airflow/pull/47281.
It's even nicer than I anticipated. I love the new super-simple workflows
this restructuring finally enabled.
With `uv` and workspace, and the new structure of tests, developing and
running tests for providers, task-sdk or any of the other future
sub-projects becomes very, very straightforward, we avoid duplication of
pytest options and switching between airflow, task-sdk, providers tests
will be simple and straightforward.
1) First of all with this change, we remove `devel-tests` extra. It was
always needed in local venv to install all test dependencies, But this is
completely gone right now. Test dependencies are automatically installed
now when you run `uv sync` - and you do not need to specify `--extra
devel-tests`. If you are still using `pip` (I strongly recommend switching
to uv) - you just install `pip install -e ./tests-common`.
2) the previous way of syncing and running tests in "everything installed"
mode works as it worked before:
uv sync --all-extras
This installs all possible extras of airflow, allows you to run tests for
all providers, activate the venv in `.venv` and run `pytests tests/always`
or `pytest providers/mongo/tests` or `pytest task_sdk/tests and it should
all work fine as it did before
3) you can also install dependencies of a selected provider (or a few of
those) in your .venv
uv sync --extra mongo
3) All the `breeze testing provider-tests --test-type "Providers[mongo]""
and generally all the ways to replicate what we run in CI inside breeze
containers with selective checks will also continue to work - no changes
here.
But then, a little bit of standardization, a bit of pre-commits that
automatically update `pyproject.toml` files of ours with cross-provider
dependencies and a little of `uv` workspace magic sprinkled over the repo
of ours and now we can do something much more straightforward:
*You can change directory to task_sdk, or provider folder of your choice
and simply run:*
*cd providers/mongo*
*uv run pytest*
or
*cd task_sdk*
*uv run pytest *
And it will do what you would expect to do - even if this is the first
thing you do after checking out Airflow's repo without any earlier syncing
or preparation. Yes. It's that simple.
The `uv run pytest` command will now automatically: install python, create
a venv, sync your venv to ONLY have dependencies needed for your provider
or any of the providers imported by your provider, and all test
dependencies needed, it will find only non-system, non-integration tests
and run them. That's it. If you want to only develop one provider - you
will not have to worry any more about breeze image or setting up all other
dependencies.
Go ahead and try it. It's unbelievably simple.
J.
On Tue, Feb 25, 2025 at 5:45 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> I like this approach, lets do it!
>
> > On 25 Feb 2025, at 16:02, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > airflow-core
> > task-sdk
> > tests-common
>
>
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Ash Berlin-Taylor <as...@apache.org>.
I like this approach, lets do it!
> On 25 Feb 2025, at 16:02, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> airflow-core
> task-sdk
> tests-common
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Jarek Potiuk <ja...@potiuk.com>.
And we could do the same with others:
airflow-core
task-sdk
tests-common
That would guarantee that those will be pure folders and not accidentally
turn into python packages.
For example now we have this:
task_sdk
\__init_.py
src
\ - airflow
\ sdk
And with implicit namespaces, someone could easily do :
`from task_sdk.src.airflow.sdk`
In fact we are already doing it for task_sdk tests:
from task_sdk.tests.api.test_client import make_client
One reason was conftest.py confusion when we had providers but we should be
able to remove it when we move tests_common.
By having the "root" of each of the sub-project the same as
"pyproject.toml` we got it way easier to import and work on
independently from the whole "airflow" repo.
J.
On Tue, Feb 25, 2025 at 4:53 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Actually - after experiencing unrelated error in my PR for "legacy"
> namespace packages -
> https://github.com/apache/airflow/actions/runs/13524796336/job/37793501020?pr=47064#step:6:36
> I have a way better idea:
>
> ```
> tests-common <- folder
> \pyproject.toml
> \src
> \tests_common <- Python package
> ```
>
> tests-common (note `-` not `_`) - is not a valid python package name, but
> it's a perfectly valid folder name. And `from tests_common` will never be
> misinterpreted this way,
>
> On Tue, Feb 25, 2025 at 4:52 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
>> Yeah, but why would we confuse things more by putting it in the already
>> messy airflow namespace? That feels like a step backwards to me.
>>
>> > On 25 Feb 2025, at 15:23, Jarek Potiuk <ja...@potiuk.com> wrote:
>> >
>> >> This is the only bit I’m confused about — what the `airflow.` prefix?
>> >
>> > Precisely to avoid that `from tests_common" will be used from the root
>> of
>> > the project (commonly added to PYTHONPATH).
>> >
>> > The whole discussion is about how to avoid this.
>> >
>> > J.
>> >
>> >
>> >
>> >
>> > On Tue, Feb 25, 2025 at 3:22 PM Ash Berlin-Taylor <as...@apache.org>
>> wrote:
>> >
>> >> This is the only bit I’m confused about — what the `airflow.` prefix?
>> >>
>> >>> On 25 Feb 2025, at 13:40, Jarek Potiuk <ja...@potiuk.com> wrote:
>> >>>
>> >>> from airflow.tests_common
>> >>
>> >>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> For additional commands, e-mail: dev-help@airflow.apache.org
>>
>>
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Jarek Potiuk <ja...@potiuk.com>.
Actually - after experiencing unrelated error in my PR for "legacy"
namespace packages -
https://github.com/apache/airflow/actions/runs/13524796336/job/37793501020?pr=47064#step:6:36
I have a way better idea:
```
tests-common <- folder
\pyproject.toml
\src
\tests_common <- Python package
```
tests-common (note `-` not `_`) - is not a valid python package name, but
it's a perfectly valid folder name. And `from tests_common` will never be
misinterpreted this way,
On Tue, Feb 25, 2025 at 4:52 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> Yeah, but why would we confuse things more by putting it in the already
> messy airflow namespace? That feels like a step backwards to me.
>
> > On 25 Feb 2025, at 15:23, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> >> This is the only bit I’m confused about — what the `airflow.` prefix?
> >
> > Precisely to avoid that `from tests_common" will be used from the root of
> > the project (commonly added to PYTHONPATH).
> >
> > The whole discussion is about how to avoid this.
> >
> > J.
> >
> >
> >
> >
> > On Tue, Feb 25, 2025 at 3:22 PM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> >
> >> This is the only bit I’m confused about — what the `airflow.` prefix?
> >>
> >>> On 25 Feb 2025, at 13:40, Jarek Potiuk <ja...@potiuk.com> wrote:
> >>>
> >>> from airflow.tests_common
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Ash Berlin-Taylor <as...@apache.org>.
Yeah, but why would we confuse things more by putting it in the already messy airflow namespace? That feels like a step backwards to me.
> On 25 Feb 2025, at 15:23, Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> This is the only bit I’m confused about — what the `airflow.` prefix?
>
> Precisely to avoid that `from tests_common" will be used from the root of
> the project (commonly added to PYTHONPATH).
>
> The whole discussion is about how to avoid this.
>
> J.
>
>
>
>
> On Tue, Feb 25, 2025 at 3:22 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
>> This is the only bit I’m confused about — what the `airflow.` prefix?
>>
>>> On 25 Feb 2025, at 13:40, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>> from airflow.tests_common
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Jarek Potiuk <ja...@potiuk.com>.
> This is the only bit I’m confused about — what the `airflow.` prefix?
Precisely to avoid that `from tests_common" will be used from the root of
the project (commonly added to PYTHONPATH).
The whole discussion is about how to avoid this.
J.
On Tue, Feb 25, 2025 at 3:22 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> This is the only bit I’m confused about — what the `airflow.` prefix?
>
> > On 25 Feb 2025, at 13:40, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > from airflow.tests_common
>
>
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Ash Berlin-Taylor <as...@apache.org>.
This is the only bit I’m confused about — what the `airflow.` prefix?
> On 25 Feb 2025, at 13:40, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> from airflow.tests_common
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Jarek Potiuk <ja...@potiuk.com>.
Ok. I described a bit more in slack discussions why repeating the folders
is not a good idea (it's prone to tooling that might have different ways of
handling PYTHONPATH for development).
But yeah - having the same name, for distribution and package makes sense.
-> so maybe this will be a good structure:
Maybe a good proposal will be to just :
tests_common <- folder
\pyproject.toml
\src
\airflow <-- Airflow Python package
\ __init__.py <- legacy namespace package with __path__
= __import__("pkgutil").extend_path(__path__, __name__)
\tests_common <- Python package
That will avoid ambiguity in imports. But this change will be bigger (but
only once) and `from airflow.tests_common import *` should be used
everywhere.
If there will be no objections, I will make a draft PR right after we
remove the old UI.
J.
On Tue, Feb 18, 2025 at 6:28 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> BTW. After reading the proposed structure again python-subfolder -> that
> is the biggest point to discuss it, and I would rather (at least now) first
> move airflow_core to a sub-directory, not necessarily move all "python"
> projects to a sub-folder. There is very little value in such grouping, and
> even if we add new languages for sdks, at least initially, having them at
> top-level as sub-folders, does not seem too problematic. But that should be
> another discussion and conclusion as I said, let's incrementally change
> what we have now to be "good pyproject.toml and workspace" support, and
> then we can decide what to do next. once we do it, just moving folders will
> be way easier.
>
> On Tue, Feb 18, 2025 at 6:20 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> > One thing I would like to avoid is having the `[test]` extra show up in
>> the released packages though (it’s not important, just would be nice if we
>> can avoid that)
>>
>> That is already not happening (and won't, all those devel extras in
>> dependencies (e is gone at the moment I added hatchling, it handles dynamic
>> extras removal for wheel packages, so almost all devel kind of extras have
>> been removed back then - at the beginning of 2024 - I think it was Airflow
>> 2.7? 2.8? can't remember). The only remaining extra in release package was
>> "devel-ci" used exclusively for optimising our image building, but we can
>> remove it in airflow 3 now, because that has been replaced with "stash"
>> action usage and utilising it to reuse `uv` cache across the builds, so we
>> do not need `devel-ci` any more. That is one of the next cleanups that are
>> possible to be done now after all the "foundational" cleanups in the
>> packaging :D. Airflow is getting cleaner :)
>>
>> Also - as explained before in the near future, we will be able to get rid
>> of all the devel extras even for "editable" installs - once `pip` will get
>> support for "dependency-groups" https://peps.python.org/pep-0735/
>> approved - we will be able to fully switch to dependency groups for both
>> `uv` and `pip` installation, so the `devel` kind of extras will be gone
>> completely - not only from wheels, but also from editable installations.
>> And even more - next step, once https://peps.python.org/pep-0771/ Default
>> Extras for Python Software Packages (currently Draft) will be accepted and
>> implemented (this time it will likely just have to be implemented by
>> hatchling backend or whatever backend we will choose) - then we will likely
>> be able to simplify the dynamic dependencies we have now - and bring them
>> back from being calculated in hatch_build.py to Airflow's
>> pyproject.toml (actually likely we should be able to make
>> optional-dependencies non-dynamic even before PEP-0771 is implemented -
>> because I think dependency groups are enough to make our optional
>> dependencies non-dynamic.
>>
>> > I wonder if now is also the time to move all the python code under a
>> sub-folder
>>
>> Absolutely. That was absolutely my goal as well. And I did not need
>> "dagster" inspiration :). it's a very natural consequence of all
>> other moves (and yes `uv` supports both types of workspaces - one where the
>> top project is in the root and one when the root is just the workspace with
>> no code, or just a bootstrap, which is pretty cool).
>>
>> For now I was holding up this discussion to the next step, but yes,
>> absolutely, that would be the next topic to discuss and absolutely the way
>> to go. I think though it will require a bit more discussion on consequences
>> and transition process - especially that we are in the middle of
>> super-heavy modifications to Airflow-core, also some removals (old UI!
>> YAY!) are about to happen that will make this move quite a bit easier.
>> Moving tests_common as an incremental step will help us also to learn what
>> it means - to move a "root" Python package to a "folder/src". I expect a
>> few surprises, (so called "unknown unknowns") and I think it's better to
>> solve them now with tests_common, which is far less "busy" than airflow,
>> and then discussion about moving "airflow" to its own sub-folder would be
>> much more informed.
>>
>> One thing-at-a-time is my motto here. We have at least a few dozen people
>> actively contributing to airflow repo and whatever we break, we should
>> break a little, incrementally. And we will break things, I am sure.
>>
>> > This extra top level python_modules folder might help with the
>> PYTHONPATH issue?
>>
>> I am afraid not entirely. It might or might not and it will also depend
>> very much on the tooling used by our contributors. It really depends on
>> interpretation of tools, IDEs. For example IntelliJ/Pycharm will (as far as
>> I know) always add content root to the PYTHONPATH, regardless, and there
>> are also some default python behaviours (scripts adding parent folder of
>> the script to PYTHONPATH for example) that are somewhat unexpected and I
>> find using different "folder" name and "top package name" as the easiest
>> and most robust way to prevent those difficult-to-debug things happening.
>> I'd prefer to prevent those issues rather than having to diagnose and fix
>> them :).
>>
>> J.
>>
>>
>>
>>
>> On Tue, Feb 18, 2025 at 10:06 AM Ash Berlin-Taylor <as...@apache.org>
>> wrote:
>>
>>> One thing I would like to avoid is having the `[test]` extra show up in
>>> the released packages though (it’s not important, just would be nice if we
>>> can avoid that)
>>>
>>> I wonder if now is also the time to move all the python code under a
>>> sub-folder
>>>
>>> So something like this
>>>
>>> Python_modules/
>>> Airflow_core/
>>> Providers/
>>> task-sdk/
>>> Helm/
>>> Dev/
>>> Breeze/
>>>
>>> Etc. (Casing not relevant, just auto-correct)
>>>
>>> This extra top level python_modules folder might help with the
>>> PYTHONPATH issue?
>>>
>>> (Yes, I’m using dragster somewhat as inspiration for this
>>> https://github.com/dagster-io/dagster/ )
>>>
>>>
>>> > On 18 Feb 2025, at 04:51, Amogh Desai <am...@gmail.com>
>>> wrote:
>>> >
>>> > +1 to this idea overall.
>>> >
>>> > A bit torn on naming it "common_test_code" -- no strong reason for it
>>> but
>>> > names like: `airflow_test_utils` or
>>> > `airflow_test_shared` sound better to me. No strong objection though.
>>> >
>>> > Thanks & Regards,
>>> > Amogh Desai
>>> >
>>> >
>>> > On Mon, Feb 17, 2025 at 11:16 PM Jarek Potiuk <ja...@potiuk.com>
>>> wrote:
>>> >
>>> >> Main reason is that this might avoid duplication and remove ambiguity
>>> of
>>> >> what is being imported. If we keep the same name, we will have to have
>>> >> something like that:
>>> >>
>>> >> a) folder where project is
>>> >> b) python package we import
>>> >>
>>> >>
>>> >> So ...if we do tests_common, we will have to do:
>>> >>
>>> >> tests_common <- folder
>>> >> \pyproject.toml
>>> >> \src
>>> >> \tests_common <- Python package
>>> >>
>>> >>
>>> >> And when you do:
>>> >>
>>> >> import tests_common
>>> >>
>>> >> Depending on the tooling, PYTHONPATH, whether implicit python
>>> packages are
>>> >> enabled, WEIRD things might happen. We've seen it when we had "smtp"
>>> >> provider name and it clashed with built-in package and that's why we
>>> have
>>> >> now "unit.smtp", "integrations.smtp" and "system.smtp" as "canonical
>>> >> imports" to avoid this problem.
>>> >>
>>> >> This is similar - importing tests_common in this case might behave
>>> >> differently if we keep the same name. That was my main motivation to
>>> have a
>>> >> "different" name - whether it's "common_test_code" or something else,
>>> does
>>> >> not matter, it just has to be significantly different.
>>> >>
>>> >> Another approach could be what we did in providers, add extra package
>>> >> inside the project, and import "tests_common" with a package prefix
>>> (for
>>> >> example `airflow_tests`):
>>> >>
>>> >>
>>> >> tests_common <- folder
>>> >> \pyproject.toml
>>> >> \src
>>> >> \airflow_tests <- Python package
>>> >> \tests_common <- Python package
>>> >>
>>> >> But then we would have to change all the tests that import
>>> tests_common to:
>>> >>
>>> >> from `airflow_tests.tests_common`
>>> >>
>>> >> Easy to do, but I wanted to avoid another 1000+ files PR for our poor
>>> >> reviewers :)
>>> >>
>>> >> So basically, I think we have two options:
>>> >>
>>> >> a) different "folder" name where we keep the project
>>> >> b) adding parent package to "tests_common"
>>> >>
>>> >> We could do both, I am happy with either approach but maybe we should
>>> do a
>>> >> small unscientific poll, and have others propose better names (you
>>> know,
>>> >> naming is hard :D).
>>> >>
>>> >> J
>>> >>
>>> >>
>>> >> pon., 17 lut 2025, 16:43 użytkownik Vincent Beck <vincbeck@apache.org
>>> >
>>> >> napisał:
>>> >>
>>> >>> Overall +1 on this one. Regarding the naming, why not keeping
>>> >>> "tests_common" instead of "common_test_code"? I am not a big fan of
>>> >>> "common_test_code" but it is obviously just a personal opinion (as
>>> it is
>>> >>> always with naming :))
>>> >>>
>>> >>> On 2025/02/16 13:30:09 Jarek Potiuk wrote:
>>> >>>>> Just wondernig... would an optional dependency not be the right
>>> place
>>> >>> to
>>> >>>> describe that apache-airflow-providers-google[tests] would have an
>>> >>>> dependency to the common_tests subproject?
>>> >>>>> Would mean you would need to install via
>>> >>>>> pip install -e . -e ./task_sdk[tests] -e. ./providers/google[tests]
>>> >>>>
>>> >>>> Something like that. This is more of a details of workspace but we
>>> are
>>> >>>> going to use "dev" dependency group for that - rather than extra.
>>> >> Details
>>> >>>> to be worked out how pip interaction will look like (pip does not
>>> have
>>> >>>> support for dependency groups yet - they are coming in 25.1 - they
>>> are
>>> >>>> already merged in main)
>>> >>>>
>>> >>>> With `uv` dependency groups can be used today and "dev" dependency
>>> >> group
>>> >>> is
>>> >>>> installed automatically when you run uv sync or `uv pip install
>>> -e.` ->
>>> >>> so
>>> >>>> we will follow this along
>>> >>>>
>>> >>>> See details about "development dependencies" in uv here
>>> >>>>
>>> >>>
>>> >>
>>> https://docs.astral.sh/uv/concepts/projects/dependencies/#development-dependencies
>>> >>>>
>>> >>>> Dependency groups are already approved via PEP-735
>>> >>>> https://peps.python.org/pep-0735/ and as mentioned - we are a
>>> release
>>> >>> away
>>> >>>> from having them released in `pip`, so for now we will have to
>>> emulate
>>> >> it
>>> >>>> with extras for pip case I think but we will be able to remove it
>>> when
>>> >>> pip
>>> >>>> 25.1 is released and matures a bit.
>>> >>>>
>>> >>>> J.
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> On Sun, Feb 16, 2025 at 2:23 PM Jarek Potiuk <ja...@potiuk.com>
>>> wrote:
>>> >>>>
>>> >>>>>> I would love to see some airflow_testing package which will be
>>> >>> useful for
>>> >>>>> testing airflow-related projects and involve independently.
>>> >>>>>
>>> >>>>>> Certainly, it's not a good thing to have tests import something
>>> >> from
>>> >>>>> tests.
>>> >>>>> New packages as projects are cheap and provide more flexibility and
>>> >> are
>>> >>>>> useful from outside of the project.
>>> >>>>>
>>> >>>>> I already explained in the PR, but let me repeat my opinion here:
>>> >>>>>
>>> >>>>> IMHO It's extremely unlikely we are going to release and publish
>>> the
>>> >>>>> common test code / fixtures in any way. They will continue to be in
>>> >>>>> development-only-distribution and they will be treated as "internal
>>> >>> detail".
>>> >>>>>
>>> >>>>> If we decide to release and publish them, we will have to maintain
>>> >>>>> backwards compatibility and account for our users (like you) using
>>> >>> them for
>>> >>>>> their own purpose. That would block us or make it very difficult to
>>> >>> make
>>> >>>>> breaking changes in them.
>>> >>>>>
>>> >>>>> So while you will be free to continue copying the whole
>>> distribution
>>> >>> and
>>> >>>>> use it in your tests as you want (our licence allows that) - I
>>> >>> seriously
>>> >>>>> doubt we will ever release and publish it in "reusable form" with
>>> >>>>> "compatibility guarantees". It's far more efficient if people like
>>> >> you
>>> >>> just
>>> >>>>> copy them and are aware that they can change any time.
>>> >>>>>
>>> >>>>> J.
>>> >>>>>
>>> >>>>> On Sun, Feb 16, 2025 at 2:21 PM Alexander Shorin <
>>> kxepal@apache.org>
>>> >>>>> wrote:
>>> >>>>>
>>> >>>>>> I would love to see some airflow_testing package which will be
>>> >> useful
>>> >>> for
>>> >>>>>> testing airflow-related projects and involve independently.
>>> >>>>>>
>>> >>>>>> Certainly, it's not a good thing to have tests import something
>>> from
>>> >>>>>> tests.
>>> >>>>>> New packages as projects are cheap and provide more flexibility
>>> and
>>> >>> are
>>> >>>>>> useful from outside of the project.
>>> >>>>>>
>>> >>>>>> Also this project could have a future for testing, compatibility,
>>> >>> quality
>>> >>>>>> and rest of measuring.
>>> >>>>>>
>>> >>>>>> --
>>> >>>>>> ,,,^..^,,,
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Sun, Feb 16, 2025 at 4:12 PM Jarek Potiuk <ja...@potiuk.com>
>>> >>> wrote:
>>> >>>>>>
>>> >>>>>>> Hello here,
>>> >>>>>>>
>>> >>>>>>> Next phase of the cleanup - it's been sped up by the comment
>>> from
>>> >>>>>> @kxepal
>>> >>>>>>> -
>>> >>> https://github.com/apache/airflow/pull/46801#issuecomment-2661415731
>>> >>>>>> -
>>> >>>>>>> I
>>> >>>>>>> have planned to do it a bit later this week, but maybe indeed
>>> >> it's a
>>> >>>>>> good
>>> >>>>>>> idea to start a discussion now so that people are not confused.
>>> >>>>>>>
>>> >>>>>>> Currently we are using "from tests_common" to reuse code in
>>> >> various
>>> >>>>>>> providers and airflow, and this is fine, with the exception that
>>> >>>>>>> "tests_common" is currently just a package in "airflow" main
>>> >>> project.
>>> >>>>>> But
>>> >>>>>>> it does not have to be (or rather it should not be).
>>> >>>>>>>
>>> >>>>>>> We should have it in a separate sub-project - similarly as
>>> >>> providers/*
>>> >>>>>> and
>>> >>>>>>> task_sdk are now separate projects - and we should add dependency
>>> >>> to the
>>> >>>>>>> common test code distribution from within all the projects that
>>> >> use
>>> >>> it
>>> >>>>>> (via
>>> >>>>>>> workspace feature).
>>> >>>>>>>
>>> >>>>>>> My proposal is to name it "common_test_code" (but I am open to
>>> any
>>> >>>>>>> suggestions): It will look like this
>>> >>>>>>>
>>> >>>>>>> .
>>> >>>>>>> |- airflow
>>> >>>>>>> |- common_test_code
>>> >>>>>>> | pyproject.toml
>>> >>>>>>> | src
>>> >>>>>>> | tests_common
>>> >>>>>>>
>>> >>>>>>> The code in airflow and providers will not change, they will
>>> >>> continue to
>>> >>>>>>> use "from tests_common import ...".
>>> >>>>>>>
>>> >>>>>>> * uv will work as it used to work, no changes will be needed - uv
>>> >>> sync
>>> >>>>>> will
>>> >>>>>>> automatically install the common_test_code as editable project
>>> >>>>>>>
>>> >>>>>>> * additionally - (and that's a big plus) after this change you
>>> >>> should be
>>> >>>>>>> able to run `uv sync` in a provider folder and have
>>> >>> provider-specific
>>> >>>>>>> separate venv, and basically you should be able to run tests for
>>> a
>>> >>>>>> provider
>>> >>>>>>> using that .venv and generally treat provider project as a
>>> >>> "standalone"
>>> >>>>>> one
>>> >>>>>>> - this is what I hinted at in the previous mail.
>>> >>>>>>>
>>> >>>>>>> * people who do not use uv but pip for example will have to
>>> >>> manually add
>>> >>>>>>> the `common_test_code` as an editable install in their venv. For
>>> >>>>>> example:
>>> >>>>>>>
>>> >>>>>>> pip install -e . -e ./task_sdk -e. ./providers/google -e
>>> >>>>>> ./common_test_code
>>> >>>>>>>
>>> >>>>>>> This is the next step in standardizing the layout of the Airflow
>>> >>> project
>>> >>>>>>> using workspaces.
>>> >>>>>>>
>>> >>>>>>> J.
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>> ---------------------------------------------------------------------
>>> >>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>> >>> For additional commands, e-mail: dev-help@airflow.apache.org
>>> >>>
>>> >>>
>>> >>
>>>
>>>
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Jarek Potiuk <ja...@potiuk.com>.
BTW. After reading the proposed structure again python-subfolder -> that is
the biggest point to discuss it, and I would rather (at least now) first
move airflow_core to a sub-directory, not necessarily move all "python"
projects to a sub-folder. There is very little value in such grouping, and
even if we add new languages for sdks, at least initially, having them at
top-level as sub-folders, does not seem too problematic. But that should be
another discussion and conclusion as I said, let's incrementally change
what we have now to be "good pyproject.toml and workspace" support, and
then we can decide what to do next. once we do it, just moving folders will
be way easier.
On Tue, Feb 18, 2025 at 6:20 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > One thing I would like to avoid is having the `[test]` extra show up in
> the released packages though (it’s not important, just would be nice if we
> can avoid that)
>
> That is already not happening (and won't, all those devel extras in
> dependencies (e is gone at the moment I added hatchling, it handles dynamic
> extras removal for wheel packages, so almost all devel kind of extras have
> been removed back then - at the beginning of 2024 - I think it was Airflow
> 2.7? 2.8? can't remember). The only remaining extra in release package was
> "devel-ci" used exclusively for optimising our image building, but we can
> remove it in airflow 3 now, because that has been replaced with "stash"
> action usage and utilising it to reuse `uv` cache across the builds, so we
> do not need `devel-ci` any more. That is one of the next cleanups that are
> possible to be done now after all the "foundational" cleanups in the
> packaging :D. Airflow is getting cleaner :)
>
> Also - as explained before in the near future, we will be able to get rid
> of all the devel extras even for "editable" installs - once `pip` will get
> support for "dependency-groups" https://peps.python.org/pep-0735/
> approved - we will be able to fully switch to dependency groups for both
> `uv` and `pip` installation, so the `devel` kind of extras will be gone
> completely - not only from wheels, but also from editable installations.
> And even more - next step, once https://peps.python.org/pep-0771/ Default
> Extras for Python Software Packages (currently Draft) will be accepted and
> implemented (this time it will likely just have to be implemented by
> hatchling backend or whatever backend we will choose) - then we will likely
> be able to simplify the dynamic dependencies we have now - and bring them
> back from being calculated in hatch_build.py to Airflow's
> pyproject.toml (actually likely we should be able to make
> optional-dependencies non-dynamic even before PEP-0771 is implemented -
> because I think dependency groups are enough to make our optional
> dependencies non-dynamic.
>
> > I wonder if now is also the time to move all the python code under a
> sub-folder
>
> Absolutely. That was absolutely my goal as well. And I did not need
> "dagster" inspiration :). it's a very natural consequence of all
> other moves (and yes `uv` supports both types of workspaces - one where the
> top project is in the root and one when the root is just the workspace with
> no code, or just a bootstrap, which is pretty cool).
>
> For now I was holding up this discussion to the next step, but yes,
> absolutely, that would be the next topic to discuss and absolutely the way
> to go. I think though it will require a bit more discussion on consequences
> and transition process - especially that we are in the middle of
> super-heavy modifications to Airflow-core, also some removals (old UI!
> YAY!) are about to happen that will make this move quite a bit easier.
> Moving tests_common as an incremental step will help us also to learn what
> it means - to move a "root" Python package to a "folder/src". I expect a
> few surprises, (so called "unknown unknowns") and I think it's better to
> solve them now with tests_common, which is far less "busy" than airflow,
> and then discussion about moving "airflow" to its own sub-folder would be
> much more informed.
>
> One thing-at-a-time is my motto here. We have at least a few dozen people
> actively contributing to airflow repo and whatever we break, we should
> break a little, incrementally. And we will break things, I am sure.
>
> > This extra top level python_modules folder might help with the
> PYTHONPATH issue?
>
> I am afraid not entirely. It might or might not and it will also depend
> very much on the tooling used by our contributors. It really depends on
> interpretation of tools, IDEs. For example IntelliJ/Pycharm will (as far as
> I know) always add content root to the PYTHONPATH, regardless, and there
> are also some default python behaviours (scripts adding parent folder of
> the script to PYTHONPATH for example) that are somewhat unexpected and I
> find using different "folder" name and "top package name" as the easiest
> and most robust way to prevent those difficult-to-debug things happening.
> I'd prefer to prevent those issues rather than having to diagnose and fix
> them :).
>
> J.
>
>
>
>
> On Tue, Feb 18, 2025 at 10:06 AM Ash Berlin-Taylor <as...@apache.org> wrote:
>
>> One thing I would like to avoid is having the `[test]` extra show up in
>> the released packages though (it’s not important, just would be nice if we
>> can avoid that)
>>
>> I wonder if now is also the time to move all the python code under a
>> sub-folder
>>
>> So something like this
>>
>> Python_modules/
>> Airflow_core/
>> Providers/
>> task-sdk/
>> Helm/
>> Dev/
>> Breeze/
>>
>> Etc. (Casing not relevant, just auto-correct)
>>
>> This extra top level python_modules folder might help with the PYTHONPATH
>> issue?
>>
>> (Yes, I’m using dragster somewhat as inspiration for this
>> https://github.com/dagster-io/dagster/ )
>>
>>
>> > On 18 Feb 2025, at 04:51, Amogh Desai <am...@gmail.com> wrote:
>> >
>> > +1 to this idea overall.
>> >
>> > A bit torn on naming it "common_test_code" -- no strong reason for it
>> but
>> > names like: `airflow_test_utils` or
>> > `airflow_test_shared` sound better to me. No strong objection though.
>> >
>> > Thanks & Regards,
>> > Amogh Desai
>> >
>> >
>> > On Mon, Feb 17, 2025 at 11:16 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>> >
>> >> Main reason is that this might avoid duplication and remove ambiguity
>> of
>> >> what is being imported. If we keep the same name, we will have to have
>> >> something like that:
>> >>
>> >> a) folder where project is
>> >> b) python package we import
>> >>
>> >>
>> >> So ...if we do tests_common, we will have to do:
>> >>
>> >> tests_common <- folder
>> >> \pyproject.toml
>> >> \src
>> >> \tests_common <- Python package
>> >>
>> >>
>> >> And when you do:
>> >>
>> >> import tests_common
>> >>
>> >> Depending on the tooling, PYTHONPATH, whether implicit python packages
>> are
>> >> enabled, WEIRD things might happen. We've seen it when we had "smtp"
>> >> provider name and it clashed with built-in package and that's why we
>> have
>> >> now "unit.smtp", "integrations.smtp" and "system.smtp" as "canonical
>> >> imports" to avoid this problem.
>> >>
>> >> This is similar - importing tests_common in this case might behave
>> >> differently if we keep the same name. That was my main motivation to
>> have a
>> >> "different" name - whether it's "common_test_code" or something else,
>> does
>> >> not matter, it just has to be significantly different.
>> >>
>> >> Another approach could be what we did in providers, add extra package
>> >> inside the project, and import "tests_common" with a package prefix
>> (for
>> >> example `airflow_tests`):
>> >>
>> >>
>> >> tests_common <- folder
>> >> \pyproject.toml
>> >> \src
>> >> \airflow_tests <- Python package
>> >> \tests_common <- Python package
>> >>
>> >> But then we would have to change all the tests that import
>> tests_common to:
>> >>
>> >> from `airflow_tests.tests_common`
>> >>
>> >> Easy to do, but I wanted to avoid another 1000+ files PR for our poor
>> >> reviewers :)
>> >>
>> >> So basically, I think we have two options:
>> >>
>> >> a) different "folder" name where we keep the project
>> >> b) adding parent package to "tests_common"
>> >>
>> >> We could do both, I am happy with either approach but maybe we should
>> do a
>> >> small unscientific poll, and have others propose better names (you
>> know,
>> >> naming is hard :D).
>> >>
>> >> J
>> >>
>> >>
>> >> pon., 17 lut 2025, 16:43 użytkownik Vincent Beck <vi...@apache.org>
>> >> napisał:
>> >>
>> >>> Overall +1 on this one. Regarding the naming, why not keeping
>> >>> "tests_common" instead of "common_test_code"? I am not a big fan of
>> >>> "common_test_code" but it is obviously just a personal opinion (as it
>> is
>> >>> always with naming :))
>> >>>
>> >>> On 2025/02/16 13:30:09 Jarek Potiuk wrote:
>> >>>>> Just wondernig... would an optional dependency not be the right
>> place
>> >>> to
>> >>>> describe that apache-airflow-providers-google[tests] would have an
>> >>>> dependency to the common_tests subproject?
>> >>>>> Would mean you would need to install via
>> >>>>> pip install -e . -e ./task_sdk[tests] -e. ./providers/google[tests]
>> >>>>
>> >>>> Something like that. This is more of a details of workspace but we
>> are
>> >>>> going to use "dev" dependency group for that - rather than extra.
>> >> Details
>> >>>> to be worked out how pip interaction will look like (pip does not
>> have
>> >>>> support for dependency groups yet - they are coming in 25.1 - they
>> are
>> >>>> already merged in main)
>> >>>>
>> >>>> With `uv` dependency groups can be used today and "dev" dependency
>> >> group
>> >>> is
>> >>>> installed automatically when you run uv sync or `uv pip install -e.`
>> ->
>> >>> so
>> >>>> we will follow this along
>> >>>>
>> >>>> See details about "development dependencies" in uv here
>> >>>>
>> >>>
>> >>
>> https://docs.astral.sh/uv/concepts/projects/dependencies/#development-dependencies
>> >>>>
>> >>>> Dependency groups are already approved via PEP-735
>> >>>> https://peps.python.org/pep-0735/ and as mentioned - we are a
>> release
>> >>> away
>> >>>> from having them released in `pip`, so for now we will have to
>> emulate
>> >> it
>> >>>> with extras for pip case I think but we will be able to remove it
>> when
>> >>> pip
>> >>>> 25.1 is released and matures a bit.
>> >>>>
>> >>>> J.
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Sun, Feb 16, 2025 at 2:23 PM Jarek Potiuk <ja...@potiuk.com>
>> wrote:
>> >>>>
>> >>>>>> I would love to see some airflow_testing package which will be
>> >>> useful for
>> >>>>> testing airflow-related projects and involve independently.
>> >>>>>
>> >>>>>> Certainly, it's not a good thing to have tests import something
>> >> from
>> >>>>> tests.
>> >>>>> New packages as projects are cheap and provide more flexibility and
>> >> are
>> >>>>> useful from outside of the project.
>> >>>>>
>> >>>>> I already explained in the PR, but let me repeat my opinion here:
>> >>>>>
>> >>>>> IMHO It's extremely unlikely we are going to release and publish the
>> >>>>> common test code / fixtures in any way. They will continue to be in
>> >>>>> development-only-distribution and they will be treated as "internal
>> >>> detail".
>> >>>>>
>> >>>>> If we decide to release and publish them, we will have to maintain
>> >>>>> backwards compatibility and account for our users (like you) using
>> >>> them for
>> >>>>> their own purpose. That would block us or make it very difficult to
>> >>> make
>> >>>>> breaking changes in them.
>> >>>>>
>> >>>>> So while you will be free to continue copying the whole distribution
>> >>> and
>> >>>>> use it in your tests as you want (our licence allows that) - I
>> >>> seriously
>> >>>>> doubt we will ever release and publish it in "reusable form" with
>> >>>>> "compatibility guarantees". It's far more efficient if people like
>> >> you
>> >>> just
>> >>>>> copy them and are aware that they can change any time.
>> >>>>>
>> >>>>> J.
>> >>>>>
>> >>>>> On Sun, Feb 16, 2025 at 2:21 PM Alexander Shorin <kxepal@apache.org
>> >
>> >>>>> wrote:
>> >>>>>
>> >>>>>> I would love to see some airflow_testing package which will be
>> >> useful
>> >>> for
>> >>>>>> testing airflow-related projects and involve independently.
>> >>>>>>
>> >>>>>> Certainly, it's not a good thing to have tests import something
>> from
>> >>>>>> tests.
>> >>>>>> New packages as projects are cheap and provide more flexibility and
>> >>> are
>> >>>>>> useful from outside of the project.
>> >>>>>>
>> >>>>>> Also this project could have a future for testing, compatibility,
>> >>> quality
>> >>>>>> and rest of measuring.
>> >>>>>>
>> >>>>>> --
>> >>>>>> ,,,^..^,,,
>> >>>>>>
>> >>>>>>
>> >>>>>> On Sun, Feb 16, 2025 at 4:12 PM Jarek Potiuk <ja...@potiuk.com>
>> >>> wrote:
>> >>>>>>
>> >>>>>>> Hello here,
>> >>>>>>>
>> >>>>>>> Next phase of the cleanup - it's been sped up by the comment from
>> >>>>>> @kxepal
>> >>>>>>> -
>> >>> https://github.com/apache/airflow/pull/46801#issuecomment-2661415731
>> >>>>>> -
>> >>>>>>> I
>> >>>>>>> have planned to do it a bit later this week, but maybe indeed
>> >> it's a
>> >>>>>> good
>> >>>>>>> idea to start a discussion now so that people are not confused.
>> >>>>>>>
>> >>>>>>> Currently we are using "from tests_common" to reuse code in
>> >> various
>> >>>>>>> providers and airflow, and this is fine, with the exception that
>> >>>>>>> "tests_common" is currently just a package in "airflow" main
>> >>> project.
>> >>>>>> But
>> >>>>>>> it does not have to be (or rather it should not be).
>> >>>>>>>
>> >>>>>>> We should have it in a separate sub-project - similarly as
>> >>> providers/*
>> >>>>>> and
>> >>>>>>> task_sdk are now separate projects - and we should add dependency
>> >>> to the
>> >>>>>>> common test code distribution from within all the projects that
>> >> use
>> >>> it
>> >>>>>> (via
>> >>>>>>> workspace feature).
>> >>>>>>>
>> >>>>>>> My proposal is to name it "common_test_code" (but I am open to any
>> >>>>>>> suggestions): It will look like this
>> >>>>>>>
>> >>>>>>> .
>> >>>>>>> |- airflow
>> >>>>>>> |- common_test_code
>> >>>>>>> | pyproject.toml
>> >>>>>>> | src
>> >>>>>>> | tests_common
>> >>>>>>>
>> >>>>>>> The code in airflow and providers will not change, they will
>> >>> continue to
>> >>>>>>> use "from tests_common import ...".
>> >>>>>>>
>> >>>>>>> * uv will work as it used to work, no changes will be needed - uv
>> >>> sync
>> >>>>>> will
>> >>>>>>> automatically install the common_test_code as editable project
>> >>>>>>>
>> >>>>>>> * additionally - (and that's a big plus) after this change you
>> >>> should be
>> >>>>>>> able to run `uv sync` in a provider folder and have
>> >>> provider-specific
>> >>>>>>> separate venv, and basically you should be able to run tests for a
>> >>>>>> provider
>> >>>>>>> using that .venv and generally treat provider project as a
>> >>> "standalone"
>> >>>>>> one
>> >>>>>>> - this is what I hinted at in the previous mail.
>> >>>>>>>
>> >>>>>>> * people who do not use uv but pip for example will have to
>> >>> manually add
>> >>>>>>> the `common_test_code` as an editable install in their venv. For
>> >>>>>> example:
>> >>>>>>>
>> >>>>>>> pip install -e . -e ./task_sdk -e. ./providers/google -e
>> >>>>>> ./common_test_code
>> >>>>>>>
>> >>>>>>> This is the next step in standardizing the layout of the Airflow
>> >>> project
>> >>>>>>> using workspaces.
>> >>>>>>>
>> >>>>>>> J.
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> >>> For additional commands, e-mail: dev-help@airflow.apache.org
>> >>>
>> >>>
>> >>
>>
>>
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Jarek Potiuk <ja...@potiuk.com>.
> One thing I would like to avoid is having the `[test]` extra show up in
the released packages though (it’s not important, just would be nice if we
can avoid that)
That is already not happening (and won't, all those devel extras in
dependencies (e is gone at the moment I added hatchling, it handles dynamic
extras removal for wheel packages, so almost all devel kind of extras have
been removed back then - at the beginning of 2024 - I think it was Airflow
2.7? 2.8? can't remember). The only remaining extra in release package was
"devel-ci" used exclusively for optimising our image building, but we can
remove it in airflow 3 now, because that has been replaced with "stash"
action usage and utilising it to reuse `uv` cache across the builds, so we
do not need `devel-ci` any more. That is one of the next cleanups that are
possible to be done now after all the "foundational" cleanups in the
packaging :D. Airflow is getting cleaner :)
Also - as explained before in the near future, we will be able to get rid
of all the devel extras even for "editable" installs - once `pip` will get
support for "dependency-groups" https://peps.python.org/pep-0735/ approved
- we will be able to fully switch to dependency groups for both `uv` and
`pip` installation, so the `devel` kind of extras will be gone completely -
not only from wheels, but also from editable installations. And even more -
next step, once https://peps.python.org/pep-0771/ Default Extras for Python
Software Packages (currently Draft) will be accepted and implemented (this
time it will likely just have to be implemented by hatchling backend or
whatever backend we will choose) - then we will likely be able to simplify
the dynamic dependencies we have now - and bring them back from being
calculated in hatch_build.py to Airflow's pyproject.toml (actually likely
we should be able to make optional-dependencies non-dynamic even before
PEP-0771 is implemented - because I think dependency groups are enough to
make our optional dependencies non-dynamic.
> I wonder if now is also the time to move all the python code under a
sub-folder
Absolutely. That was absolutely my goal as well. And I did not need
"dagster" inspiration :). it's a very natural consequence of all
other moves (and yes `uv` supports both types of workspaces - one where the
top project is in the root and one when the root is just the workspace with
no code, or just a bootstrap, which is pretty cool).
For now I was holding up this discussion to the next step, but yes,
absolutely, that would be the next topic to discuss and absolutely the way
to go. I think though it will require a bit more discussion on consequences
and transition process - especially that we are in the middle of
super-heavy modifications to Airflow-core, also some removals (old UI!
YAY!) are about to happen that will make this move quite a bit easier.
Moving tests_common as an incremental step will help us also to learn what
it means - to move a "root" Python package to a "folder/src". I expect a
few surprises, (so called "unknown unknowns") and I think it's better to
solve them now with tests_common, which is far less "busy" than airflow,
and then discussion about moving "airflow" to its own sub-folder would be
much more informed.
One thing-at-a-time is my motto here. We have at least a few dozen people
actively contributing to airflow repo and whatever we break, we should
break a little, incrementally. And we will break things, I am sure.
> This extra top level python_modules folder might help with the PYTHONPATH
issue?
I am afraid not entirely. It might or might not and it will also depend
very much on the tooling used by our contributors. It really depends on
interpretation of tools, IDEs. For example IntelliJ/Pycharm will (as far as
I know) always add content root to the PYTHONPATH, regardless, and there
are also some default python behaviours (scripts adding parent folder of
the script to PYTHONPATH for example) that are somewhat unexpected and I
find using different "folder" name and "top package name" as the easiest
and most robust way to prevent those difficult-to-debug things happening.
I'd prefer to prevent those issues rather than having to diagnose and fix
them :).
J.
On Tue, Feb 18, 2025 at 10:06 AM Ash Berlin-Taylor <as...@apache.org> wrote:
> One thing I would like to avoid is having the `[test]` extra show up in
> the released packages though (it’s not important, just would be nice if we
> can avoid that)
>
> I wonder if now is also the time to move all the python code under a
> sub-folder
>
> So something like this
>
> Python_modules/
> Airflow_core/
> Providers/
> task-sdk/
> Helm/
> Dev/
> Breeze/
>
> Etc. (Casing not relevant, just auto-correct)
>
> This extra top level python_modules folder might help with the PYTHONPATH
> issue?
>
> (Yes, I’m using dragster somewhat as inspiration for this
> https://github.com/dagster-io/dagster/ )
>
>
> > On 18 Feb 2025, at 04:51, Amogh Desai <am...@gmail.com> wrote:
> >
> > +1 to this idea overall.
> >
> > A bit torn on naming it "common_test_code" -- no strong reason for it but
> > names like: `airflow_test_utils` or
> > `airflow_test_shared` sound better to me. No strong objection though.
> >
> > Thanks & Regards,
> > Amogh Desai
> >
> >
> > On Mon, Feb 17, 2025 at 11:16 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> >> Main reason is that this might avoid duplication and remove ambiguity of
> >> what is being imported. If we keep the same name, we will have to have
> >> something like that:
> >>
> >> a) folder where project is
> >> b) python package we import
> >>
> >>
> >> So ...if we do tests_common, we will have to do:
> >>
> >> tests_common <- folder
> >> \pyproject.toml
> >> \src
> >> \tests_common <- Python package
> >>
> >>
> >> And when you do:
> >>
> >> import tests_common
> >>
> >> Depending on the tooling, PYTHONPATH, whether implicit python packages
> are
> >> enabled, WEIRD things might happen. We've seen it when we had "smtp"
> >> provider name and it clashed with built-in package and that's why we
> have
> >> now "unit.smtp", "integrations.smtp" and "system.smtp" as "canonical
> >> imports" to avoid this problem.
> >>
> >> This is similar - importing tests_common in this case might behave
> >> differently if we keep the same name. That was my main motivation to
> have a
> >> "different" name - whether it's "common_test_code" or something else,
> does
> >> not matter, it just has to be significantly different.
> >>
> >> Another approach could be what we did in providers, add extra package
> >> inside the project, and import "tests_common" with a package prefix (for
> >> example `airflow_tests`):
> >>
> >>
> >> tests_common <- folder
> >> \pyproject.toml
> >> \src
> >> \airflow_tests <- Python package
> >> \tests_common <- Python package
> >>
> >> But then we would have to change all the tests that import tests_common
> to:
> >>
> >> from `airflow_tests.tests_common`
> >>
> >> Easy to do, but I wanted to avoid another 1000+ files PR for our poor
> >> reviewers :)
> >>
> >> So basically, I think we have two options:
> >>
> >> a) different "folder" name where we keep the project
> >> b) adding parent package to "tests_common"
> >>
> >> We could do both, I am happy with either approach but maybe we should
> do a
> >> small unscientific poll, and have others propose better names (you know,
> >> naming is hard :D).
> >>
> >> J
> >>
> >>
> >> pon., 17 lut 2025, 16:43 użytkownik Vincent Beck <vi...@apache.org>
> >> napisał:
> >>
> >>> Overall +1 on this one. Regarding the naming, why not keeping
> >>> "tests_common" instead of "common_test_code"? I am not a big fan of
> >>> "common_test_code" but it is obviously just a personal opinion (as it
> is
> >>> always with naming :))
> >>>
> >>> On 2025/02/16 13:30:09 Jarek Potiuk wrote:
> >>>>> Just wondernig... would an optional dependency not be the right place
> >>> to
> >>>> describe that apache-airflow-providers-google[tests] would have an
> >>>> dependency to the common_tests subproject?
> >>>>> Would mean you would need to install via
> >>>>> pip install -e . -e ./task_sdk[tests] -e. ./providers/google[tests]
> >>>>
> >>>> Something like that. This is more of a details of workspace but we are
> >>>> going to use "dev" dependency group for that - rather than extra.
> >> Details
> >>>> to be worked out how pip interaction will look like (pip does not have
> >>>> support for dependency groups yet - they are coming in 25.1 - they are
> >>>> already merged in main)
> >>>>
> >>>> With `uv` dependency groups can be used today and "dev" dependency
> >> group
> >>> is
> >>>> installed automatically when you run uv sync or `uv pip install -e.`
> ->
> >>> so
> >>>> we will follow this along
> >>>>
> >>>> See details about "development dependencies" in uv here
> >>>>
> >>>
> >>
> https://docs.astral.sh/uv/concepts/projects/dependencies/#development-dependencies
> >>>>
> >>>> Dependency groups are already approved via PEP-735
> >>>> https://peps.python.org/pep-0735/ and as mentioned - we are a release
> >>> away
> >>>> from having them released in `pip`, so for now we will have to emulate
> >> it
> >>>> with extras for pip case I think but we will be able to remove it when
> >>> pip
> >>>> 25.1 is released and matures a bit.
> >>>>
> >>>> J.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Sun, Feb 16, 2025 at 2:23 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> >>>>
> >>>>>> I would love to see some airflow_testing package which will be
> >>> useful for
> >>>>> testing airflow-related projects and involve independently.
> >>>>>
> >>>>>> Certainly, it's not a good thing to have tests import something
> >> from
> >>>>> tests.
> >>>>> New packages as projects are cheap and provide more flexibility and
> >> are
> >>>>> useful from outside of the project.
> >>>>>
> >>>>> I already explained in the PR, but let me repeat my opinion here:
> >>>>>
> >>>>> IMHO It's extremely unlikely we are going to release and publish the
> >>>>> common test code / fixtures in any way. They will continue to be in
> >>>>> development-only-distribution and they will be treated as "internal
> >>> detail".
> >>>>>
> >>>>> If we decide to release and publish them, we will have to maintain
> >>>>> backwards compatibility and account for our users (like you) using
> >>> them for
> >>>>> their own purpose. That would block us or make it very difficult to
> >>> make
> >>>>> breaking changes in them.
> >>>>>
> >>>>> So while you will be free to continue copying the whole distribution
> >>> and
> >>>>> use it in your tests as you want (our licence allows that) - I
> >>> seriously
> >>>>> doubt we will ever release and publish it in "reusable form" with
> >>>>> "compatibility guarantees". It's far more efficient if people like
> >> you
> >>> just
> >>>>> copy them and are aware that they can change any time.
> >>>>>
> >>>>> J.
> >>>>>
> >>>>> On Sun, Feb 16, 2025 at 2:21 PM Alexander Shorin <kx...@apache.org>
> >>>>> wrote:
> >>>>>
> >>>>>> I would love to see some airflow_testing package which will be
> >> useful
> >>> for
> >>>>>> testing airflow-related projects and involve independently.
> >>>>>>
> >>>>>> Certainly, it's not a good thing to have tests import something from
> >>>>>> tests.
> >>>>>> New packages as projects are cheap and provide more flexibility and
> >>> are
> >>>>>> useful from outside of the project.
> >>>>>>
> >>>>>> Also this project could have a future for testing, compatibility,
> >>> quality
> >>>>>> and rest of measuring.
> >>>>>>
> >>>>>> --
> >>>>>> ,,,^..^,,,
> >>>>>>
> >>>>>>
> >>>>>> On Sun, Feb 16, 2025 at 4:12 PM Jarek Potiuk <ja...@potiuk.com>
> >>> wrote:
> >>>>>>
> >>>>>>> Hello here,
> >>>>>>>
> >>>>>>> Next phase of the cleanup - it's been sped up by the comment from
> >>>>>> @kxepal
> >>>>>>> -
> >>> https://github.com/apache/airflow/pull/46801#issuecomment-2661415731
> >>>>>> -
> >>>>>>> I
> >>>>>>> have planned to do it a bit later this week, but maybe indeed
> >> it's a
> >>>>>> good
> >>>>>>> idea to start a discussion now so that people are not confused.
> >>>>>>>
> >>>>>>> Currently we are using "from tests_common" to reuse code in
> >> various
> >>>>>>> providers and airflow, and this is fine, with the exception that
> >>>>>>> "tests_common" is currently just a package in "airflow" main
> >>> project.
> >>>>>> But
> >>>>>>> it does not have to be (or rather it should not be).
> >>>>>>>
> >>>>>>> We should have it in a separate sub-project - similarly as
> >>> providers/*
> >>>>>> and
> >>>>>>> task_sdk are now separate projects - and we should add dependency
> >>> to the
> >>>>>>> common test code distribution from within all the projects that
> >> use
> >>> it
> >>>>>> (via
> >>>>>>> workspace feature).
> >>>>>>>
> >>>>>>> My proposal is to name it "common_test_code" (but I am open to any
> >>>>>>> suggestions): It will look like this
> >>>>>>>
> >>>>>>> .
> >>>>>>> |- airflow
> >>>>>>> |- common_test_code
> >>>>>>> | pyproject.toml
> >>>>>>> | src
> >>>>>>> | tests_common
> >>>>>>>
> >>>>>>> The code in airflow and providers will not change, they will
> >>> continue to
> >>>>>>> use "from tests_common import ...".
> >>>>>>>
> >>>>>>> * uv will work as it used to work, no changes will be needed - uv
> >>> sync
> >>>>>> will
> >>>>>>> automatically install the common_test_code as editable project
> >>>>>>>
> >>>>>>> * additionally - (and that's a big plus) after this change you
> >>> should be
> >>>>>>> able to run `uv sync` in a provider folder and have
> >>> provider-specific
> >>>>>>> separate venv, and basically you should be able to run tests for a
> >>>>>> provider
> >>>>>>> using that .venv and generally treat provider project as a
> >>> "standalone"
> >>>>>> one
> >>>>>>> - this is what I hinted at in the previous mail.
> >>>>>>>
> >>>>>>> * people who do not use uv but pip for example will have to
> >>> manually add
> >>>>>>> the `common_test_code` as an editable install in their venv. For
> >>>>>> example:
> >>>>>>>
> >>>>>>> pip install -e . -e ./task_sdk -e. ./providers/google -e
> >>>>>> ./common_test_code
> >>>>>>>
> >>>>>>> This is the next step in standardizing the layout of the Airflow
> >>> project
> >>>>>>> using workspaces.
> >>>>>>>
> >>>>>>> J.
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >>> For additional commands, e-mail: dev-help@airflow.apache.org
> >>>
> >>>
> >>
>
>
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Ash Berlin-Taylor <as...@apache.org>.
One thing I would like to avoid is having the `[test]` extra show up in the released packages though (it’s not important, just would be nice if we can avoid that)
I wonder if now is also the time to move all the python code under a sub-folder
So something like this
Python_modules/
Airflow_core/
Providers/
task-sdk/
Helm/
Dev/
Breeze/
Etc. (Casing not relevant, just auto-correct)
This extra top level python_modules folder might help with the PYTHONPATH issue?
(Yes, I’m using dragster somewhat as inspiration for this https://github.com/dagster-io/dagster/ )
> On 18 Feb 2025, at 04:51, Amogh Desai <am...@gmail.com> wrote:
>
> +1 to this idea overall.
>
> A bit torn on naming it "common_test_code" -- no strong reason for it but
> names like: `airflow_test_utils` or
> `airflow_test_shared` sound better to me. No strong objection though.
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Mon, Feb 17, 2025 at 11:16 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Main reason is that this might avoid duplication and remove ambiguity of
>> what is being imported. If we keep the same name, we will have to have
>> something like that:
>>
>> a) folder where project is
>> b) python package we import
>>
>>
>> So ...if we do tests_common, we will have to do:
>>
>> tests_common <- folder
>> \pyproject.toml
>> \src
>> \tests_common <- Python package
>>
>>
>> And when you do:
>>
>> import tests_common
>>
>> Depending on the tooling, PYTHONPATH, whether implicit python packages are
>> enabled, WEIRD things might happen. We've seen it when we had "smtp"
>> provider name and it clashed with built-in package and that's why we have
>> now "unit.smtp", "integrations.smtp" and "system.smtp" as "canonical
>> imports" to avoid this problem.
>>
>> This is similar - importing tests_common in this case might behave
>> differently if we keep the same name. That was my main motivation to have a
>> "different" name - whether it's "common_test_code" or something else, does
>> not matter, it just has to be significantly different.
>>
>> Another approach could be what we did in providers, add extra package
>> inside the project, and import "tests_common" with a package prefix (for
>> example `airflow_tests`):
>>
>>
>> tests_common <- folder
>> \pyproject.toml
>> \src
>> \airflow_tests <- Python package
>> \tests_common <- Python package
>>
>> But then we would have to change all the tests that import tests_common to:
>>
>> from `airflow_tests.tests_common`
>>
>> Easy to do, but I wanted to avoid another 1000+ files PR for our poor
>> reviewers :)
>>
>> So basically, I think we have two options:
>>
>> a) different "folder" name where we keep the project
>> b) adding parent package to "tests_common"
>>
>> We could do both, I am happy with either approach but maybe we should do a
>> small unscientific poll, and have others propose better names (you know,
>> naming is hard :D).
>>
>> J
>>
>>
>> pon., 17 lut 2025, 16:43 użytkownik Vincent Beck <vi...@apache.org>
>> napisał:
>>
>>> Overall +1 on this one. Regarding the naming, why not keeping
>>> "tests_common" instead of "common_test_code"? I am not a big fan of
>>> "common_test_code" but it is obviously just a personal opinion (as it is
>>> always with naming :))
>>>
>>> On 2025/02/16 13:30:09 Jarek Potiuk wrote:
>>>>> Just wondernig... would an optional dependency not be the right place
>>> to
>>>> describe that apache-airflow-providers-google[tests] would have an
>>>> dependency to the common_tests subproject?
>>>>> Would mean you would need to install via
>>>>> pip install -e . -e ./task_sdk[tests] -e. ./providers/google[tests]
>>>>
>>>> Something like that. This is more of a details of workspace but we are
>>>> going to use "dev" dependency group for that - rather than extra.
>> Details
>>>> to be worked out how pip interaction will look like (pip does not have
>>>> support for dependency groups yet - they are coming in 25.1 - they are
>>>> already merged in main)
>>>>
>>>> With `uv` dependency groups can be used today and "dev" dependency
>> group
>>> is
>>>> installed automatically when you run uv sync or `uv pip install -e.` ->
>>> so
>>>> we will follow this along
>>>>
>>>> See details about "development dependencies" in uv here
>>>>
>>>
>> https://docs.astral.sh/uv/concepts/projects/dependencies/#development-dependencies
>>>>
>>>> Dependency groups are already approved via PEP-735
>>>> https://peps.python.org/pep-0735/ and as mentioned - we are a release
>>> away
>>>> from having them released in `pip`, so for now we will have to emulate
>> it
>>>> with extras for pip case I think but we will be able to remove it when
>>> pip
>>>> 25.1 is released and matures a bit.
>>>>
>>>> J.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Sun, Feb 16, 2025 at 2:23 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>
>>>>>> I would love to see some airflow_testing package which will be
>>> useful for
>>>>> testing airflow-related projects and involve independently.
>>>>>
>>>>>> Certainly, it's not a good thing to have tests import something
>> from
>>>>> tests.
>>>>> New packages as projects are cheap and provide more flexibility and
>> are
>>>>> useful from outside of the project.
>>>>>
>>>>> I already explained in the PR, but let me repeat my opinion here:
>>>>>
>>>>> IMHO It's extremely unlikely we are going to release and publish the
>>>>> common test code / fixtures in any way. They will continue to be in
>>>>> development-only-distribution and they will be treated as "internal
>>> detail".
>>>>>
>>>>> If we decide to release and publish them, we will have to maintain
>>>>> backwards compatibility and account for our users (like you) using
>>> them for
>>>>> their own purpose. That would block us or make it very difficult to
>>> make
>>>>> breaking changes in them.
>>>>>
>>>>> So while you will be free to continue copying the whole distribution
>>> and
>>>>> use it in your tests as you want (our licence allows that) - I
>>> seriously
>>>>> doubt we will ever release and publish it in "reusable form" with
>>>>> "compatibility guarantees". It's far more efficient if people like
>> you
>>> just
>>>>> copy them and are aware that they can change any time.
>>>>>
>>>>> J.
>>>>>
>>>>> On Sun, Feb 16, 2025 at 2:21 PM Alexander Shorin <kx...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> I would love to see some airflow_testing package which will be
>> useful
>>> for
>>>>>> testing airflow-related projects and involve independently.
>>>>>>
>>>>>> Certainly, it's not a good thing to have tests import something from
>>>>>> tests.
>>>>>> New packages as projects are cheap and provide more flexibility and
>>> are
>>>>>> useful from outside of the project.
>>>>>>
>>>>>> Also this project could have a future for testing, compatibility,
>>> quality
>>>>>> and rest of measuring.
>>>>>>
>>>>>> --
>>>>>> ,,,^..^,,,
>>>>>>
>>>>>>
>>>>>> On Sun, Feb 16, 2025 at 4:12 PM Jarek Potiuk <ja...@potiuk.com>
>>> wrote:
>>>>>>
>>>>>>> Hello here,
>>>>>>>
>>>>>>> Next phase of the cleanup - it's been sped up by the comment from
>>>>>> @kxepal
>>>>>>> -
>>> https://github.com/apache/airflow/pull/46801#issuecomment-2661415731
>>>>>> -
>>>>>>> I
>>>>>>> have planned to do it a bit later this week, but maybe indeed
>> it's a
>>>>>> good
>>>>>>> idea to start a discussion now so that people are not confused.
>>>>>>>
>>>>>>> Currently we are using "from tests_common" to reuse code in
>> various
>>>>>>> providers and airflow, and this is fine, with the exception that
>>>>>>> "tests_common" is currently just a package in "airflow" main
>>> project.
>>>>>> But
>>>>>>> it does not have to be (or rather it should not be).
>>>>>>>
>>>>>>> We should have it in a separate sub-project - similarly as
>>> providers/*
>>>>>> and
>>>>>>> task_sdk are now separate projects - and we should add dependency
>>> to the
>>>>>>> common test code distribution from within all the projects that
>> use
>>> it
>>>>>> (via
>>>>>>> workspace feature).
>>>>>>>
>>>>>>> My proposal is to name it "common_test_code" (but I am open to any
>>>>>>> suggestions): It will look like this
>>>>>>>
>>>>>>> .
>>>>>>> |- airflow
>>>>>>> |- common_test_code
>>>>>>> | pyproject.toml
>>>>>>> | src
>>>>>>> | tests_common
>>>>>>>
>>>>>>> The code in airflow and providers will not change, they will
>>> continue to
>>>>>>> use "from tests_common import ...".
>>>>>>>
>>>>>>> * uv will work as it used to work, no changes will be needed - uv
>>> sync
>>>>>> will
>>>>>>> automatically install the common_test_code as editable project
>>>>>>>
>>>>>>> * additionally - (and that's a big plus) after this change you
>>> should be
>>>>>>> able to run `uv sync` in a provider folder and have
>>> provider-specific
>>>>>>> separate venv, and basically you should be able to run tests for a
>>>>>> provider
>>>>>>> using that .venv and generally treat provider project as a
>>> "standalone"
>>>>>> one
>>>>>>> - this is what I hinted at in the previous mail.
>>>>>>>
>>>>>>> * people who do not use uv but pip for example will have to
>>> manually add
>>>>>>> the `common_test_code` as an editable install in their venv. For
>>>>>> example:
>>>>>>>
>>>>>>> pip install -e . -e ./task_sdk -e. ./providers/google -e
>>>>>> ./common_test_code
>>>>>>>
>>>>>>> This is the next step in standardizing the layout of the Airflow
>>> project
>>>>>>> using workspaces.
>>>>>>>
>>>>>>> J.
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>
>>>
>>
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Amogh Desai <am...@gmail.com>.
+1 to this idea overall.
A bit torn on naming it "common_test_code" -- no strong reason for it but
names like: `airflow_test_utils` or
`airflow_test_shared` sound better to me. No strong objection though.
Thanks & Regards,
Amogh Desai
On Mon, Feb 17, 2025 at 11:16 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Main reason is that this might avoid duplication and remove ambiguity of
> what is being imported. If we keep the same name, we will have to have
> something like that:
>
> a) folder where project is
> b) python package we import
>
>
> So ...if we do tests_common, we will have to do:
>
> tests_common <- folder
> \pyproject.toml
> \src
> \tests_common <- Python package
>
>
> And when you do:
>
> import tests_common
>
> Depending on the tooling, PYTHONPATH, whether implicit python packages are
> enabled, WEIRD things might happen. We've seen it when we had "smtp"
> provider name and it clashed with built-in package and that's why we have
> now "unit.smtp", "integrations.smtp" and "system.smtp" as "canonical
> imports" to avoid this problem.
>
> This is similar - importing tests_common in this case might behave
> differently if we keep the same name. That was my main motivation to have a
> "different" name - whether it's "common_test_code" or something else, does
> not matter, it just has to be significantly different.
>
> Another approach could be what we did in providers, add extra package
> inside the project, and import "tests_common" with a package prefix (for
> example `airflow_tests`):
>
>
> tests_common <- folder
> \pyproject.toml
> \src
> \airflow_tests <- Python package
> \tests_common <- Python package
>
> But then we would have to change all the tests that import tests_common to:
>
> from `airflow_tests.tests_common`
>
> Easy to do, but I wanted to avoid another 1000+ files PR for our poor
> reviewers :)
>
> So basically, I think we have two options:
>
> a) different "folder" name where we keep the project
> b) adding parent package to "tests_common"
>
> We could do both, I am happy with either approach but maybe we should do a
> small unscientific poll, and have others propose better names (you know,
> naming is hard :D).
>
> J
>
>
> pon., 17 lut 2025, 16:43 użytkownik Vincent Beck <vi...@apache.org>
> napisał:
>
> > Overall +1 on this one. Regarding the naming, why not keeping
> > "tests_common" instead of "common_test_code"? I am not a big fan of
> > "common_test_code" but it is obviously just a personal opinion (as it is
> > always with naming :))
> >
> > On 2025/02/16 13:30:09 Jarek Potiuk wrote:
> > > > Just wondernig... would an optional dependency not be the right place
> > to
> > > describe that apache-airflow-providers-google[tests] would have an
> > > dependency to the common_tests subproject?
> > > > Would mean you would need to install via
> > > > pip install -e . -e ./task_sdk[tests] -e. ./providers/google[tests]
> > >
> > > Something like that. This is more of a details of workspace but we are
> > > going to use "dev" dependency group for that - rather than extra.
> Details
> > > to be worked out how pip interaction will look like (pip does not have
> > > support for dependency groups yet - they are coming in 25.1 - they are
> > > already merged in main)
> > >
> > > With `uv` dependency groups can be used today and "dev" dependency
> group
> > is
> > > installed automatically when you run uv sync or `uv pip install -e.` ->
> > so
> > > we will follow this along
> > >
> > > See details about "development dependencies" in uv here
> > >
> >
> https://docs.astral.sh/uv/concepts/projects/dependencies/#development-dependencies
> > >
> > > Dependency groups are already approved via PEP-735
> > > https://peps.python.org/pep-0735/ and as mentioned - we are a release
> > away
> > > from having them released in `pip`, so for now we will have to emulate
> it
> > > with extras for pip case I think but we will be able to remove it when
> > pip
> > > 25.1 is released and matures a bit.
> > >
> > > J.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Sun, Feb 16, 2025 at 2:23 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > > I would love to see some airflow_testing package which will be
> > useful for
> > > > testing airflow-related projects and involve independently.
> > > >
> > > > > Certainly, it's not a good thing to have tests import something
> from
> > > > tests.
> > > > New packages as projects are cheap and provide more flexibility and
> are
> > > > useful from outside of the project.
> > > >
> > > > I already explained in the PR, but let me repeat my opinion here:
> > > >
> > > > IMHO It's extremely unlikely we are going to release and publish the
> > > > common test code / fixtures in any way. They will continue to be in
> > > > development-only-distribution and they will be treated as "internal
> > detail".
> > > >
> > > > If we decide to release and publish them, we will have to maintain
> > > > backwards compatibility and account for our users (like you) using
> > them for
> > > > their own purpose. That would block us or make it very difficult to
> > make
> > > > breaking changes in them.
> > > >
> > > > So while you will be free to continue copying the whole distribution
> > and
> > > > use it in your tests as you want (our licence allows that) - I
> > seriously
> > > > doubt we will ever release and publish it in "reusable form" with
> > > > "compatibility guarantees". It's far more efficient if people like
> you
> > just
> > > > copy them and are aware that they can change any time.
> > > >
> > > > J.
> > > >
> > > > On Sun, Feb 16, 2025 at 2:21 PM Alexander Shorin <kx...@apache.org>
> > > > wrote:
> > > >
> > > >> I would love to see some airflow_testing package which will be
> useful
> > for
> > > >> testing airflow-related projects and involve independently.
> > > >>
> > > >> Certainly, it's not a good thing to have tests import something from
> > > >> tests.
> > > >> New packages as projects are cheap and provide more flexibility and
> > are
> > > >> useful from outside of the project.
> > > >>
> > > >> Also this project could have a future for testing, compatibility,
> > quality
> > > >> and rest of measuring.
> > > >>
> > > >> --
> > > >> ,,,^..^,,,
> > > >>
> > > >>
> > > >> On Sun, Feb 16, 2025 at 4:12 PM Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > > >>
> > > >> > Hello here,
> > > >> >
> > > >> > Next phase of the cleanup - it's been sped up by the comment from
> > > >> @kxepal
> > > >> > -
> > https://github.com/apache/airflow/pull/46801#issuecomment-2661415731
> > > >> -
> > > >> > I
> > > >> > have planned to do it a bit later this week, but maybe indeed
> it's a
> > > >> good
> > > >> > idea to start a discussion now so that people are not confused.
> > > >> >
> > > >> > Currently we are using "from tests_common" to reuse code in
> various
> > > >> > providers and airflow, and this is fine, with the exception that
> > > >> > "tests_common" is currently just a package in "airflow" main
> > project.
> > > >> But
> > > >> > it does not have to be (or rather it should not be).
> > > >> >
> > > >> > We should have it in a separate sub-project - similarly as
> > providers/*
> > > >> and
> > > >> > task_sdk are now separate projects - and we should add dependency
> > to the
> > > >> > common test code distribution from within all the projects that
> use
> > it
> > > >> (via
> > > >> > workspace feature).
> > > >> >
> > > >> > My proposal is to name it "common_test_code" (but I am open to any
> > > >> > suggestions): It will look like this
> > > >> >
> > > >> > .
> > > >> > |- airflow
> > > >> > |- common_test_code
> > > >> > | pyproject.toml
> > > >> > | src
> > > >> > | tests_common
> > > >> >
> > > >> > The code in airflow and providers will not change, they will
> > continue to
> > > >> > use "from tests_common import ...".
> > > >> >
> > > >> > * uv will work as it used to work, no changes will be needed - uv
> > sync
> > > >> will
> > > >> > automatically install the common_test_code as editable project
> > > >> >
> > > >> > * additionally - (and that's a big plus) after this change you
> > should be
> > > >> > able to run `uv sync` in a provider folder and have
> > provider-specific
> > > >> > separate venv, and basically you should be able to run tests for a
> > > >> provider
> > > >> > using that .venv and generally treat provider project as a
> > "standalone"
> > > >> one
> > > >> > - this is what I hinted at in the previous mail.
> > > >> >
> > > >> > * people who do not use uv but pip for example will have to
> > manually add
> > > >> > the `common_test_code` as an editable install in their venv. For
> > > >> example:
> > > >> >
> > > >> > pip install -e . -e ./task_sdk -e. ./providers/google -e
> > > >> ./common_test_code
> > > >> >
> > > >> > This is the next step in standardizing the layout of the Airflow
> > project
> > > >> > using workspaces.
> > > >> >
> > > >> > J.
> > > >> >
> > > >>
> > > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Jarek Potiuk <ja...@potiuk.com>.
Main reason is that this might avoid duplication and remove ambiguity of
what is being imported. If we keep the same name, we will have to have
something like that:
a) folder where project is
b) python package we import
So ...if we do tests_common, we will have to do:
tests_common <- folder
\pyproject.toml
\src
\tests_common <- Python package
And when you do:
import tests_common
Depending on the tooling, PYTHONPATH, whether implicit python packages are
enabled, WEIRD things might happen. We've seen it when we had "smtp"
provider name and it clashed with built-in package and that's why we have
now "unit.smtp", "integrations.smtp" and "system.smtp" as "canonical
imports" to avoid this problem.
This is similar - importing tests_common in this case might behave
differently if we keep the same name. That was my main motivation to have a
"different" name - whether it's "common_test_code" or something else, does
not matter, it just has to be significantly different.
Another approach could be what we did in providers, add extra package
inside the project, and import "tests_common" with a package prefix (for
example `airflow_tests`):
tests_common <- folder
\pyproject.toml
\src
\airflow_tests <- Python package
\tests_common <- Python package
But then we would have to change all the tests that import tests_common to:
from `airflow_tests.tests_common`
Easy to do, but I wanted to avoid another 1000+ files PR for our poor
reviewers :)
So basically, I think we have two options:
a) different "folder" name where we keep the project
b) adding parent package to "tests_common"
We could do both, I am happy with either approach but maybe we should do a
small unscientific poll, and have others propose better names (you know,
naming is hard :D).
J
pon., 17 lut 2025, 16:43 użytkownik Vincent Beck <vi...@apache.org>
napisał:
> Overall +1 on this one. Regarding the naming, why not keeping
> "tests_common" instead of "common_test_code"? I am not a big fan of
> "common_test_code" but it is obviously just a personal opinion (as it is
> always with naming :))
>
> On 2025/02/16 13:30:09 Jarek Potiuk wrote:
> > > Just wondernig... would an optional dependency not be the right place
> to
> > describe that apache-airflow-providers-google[tests] would have an
> > dependency to the common_tests subproject?
> > > Would mean you would need to install via
> > > pip install -e . -e ./task_sdk[tests] -e. ./providers/google[tests]
> >
> > Something like that. This is more of a details of workspace but we are
> > going to use "dev" dependency group for that - rather than extra. Details
> > to be worked out how pip interaction will look like (pip does not have
> > support for dependency groups yet - they are coming in 25.1 - they are
> > already merged in main)
> >
> > With `uv` dependency groups can be used today and "dev" dependency group
> is
> > installed automatically when you run uv sync or `uv pip install -e.` ->
> so
> > we will follow this along
> >
> > See details about "development dependencies" in uv here
> >
> https://docs.astral.sh/uv/concepts/projects/dependencies/#development-dependencies
> >
> > Dependency groups are already approved via PEP-735
> > https://peps.python.org/pep-0735/ and as mentioned - we are a release
> away
> > from having them released in `pip`, so for now we will have to emulate it
> > with extras for pip case I think but we will be able to remove it when
> pip
> > 25.1 is released and matures a bit.
> >
> > J.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Sun, Feb 16, 2025 at 2:23 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > > I would love to see some airflow_testing package which will be
> useful for
> > > testing airflow-related projects and involve independently.
> > >
> > > > Certainly, it's not a good thing to have tests import something from
> > > tests.
> > > New packages as projects are cheap and provide more flexibility and are
> > > useful from outside of the project.
> > >
> > > I already explained in the PR, but let me repeat my opinion here:
> > >
> > > IMHO It's extremely unlikely we are going to release and publish the
> > > common test code / fixtures in any way. They will continue to be in
> > > development-only-distribution and they will be treated as "internal
> detail".
> > >
> > > If we decide to release and publish them, we will have to maintain
> > > backwards compatibility and account for our users (like you) using
> them for
> > > their own purpose. That would block us or make it very difficult to
> make
> > > breaking changes in them.
> > >
> > > So while you will be free to continue copying the whole distribution
> and
> > > use it in your tests as you want (our licence allows that) - I
> seriously
> > > doubt we will ever release and publish it in "reusable form" with
> > > "compatibility guarantees". It's far more efficient if people like you
> just
> > > copy them and are aware that they can change any time.
> > >
> > > J.
> > >
> > > On Sun, Feb 16, 2025 at 2:21 PM Alexander Shorin <kx...@apache.org>
> > > wrote:
> > >
> > >> I would love to see some airflow_testing package which will be useful
> for
> > >> testing airflow-related projects and involve independently.
> > >>
> > >> Certainly, it's not a good thing to have tests import something from
> > >> tests.
> > >> New packages as projects are cheap and provide more flexibility and
> are
> > >> useful from outside of the project.
> > >>
> > >> Also this project could have a future for testing, compatibility,
> quality
> > >> and rest of measuring.
> > >>
> > >> --
> > >> ,,,^..^,,,
> > >>
> > >>
> > >> On Sun, Feb 16, 2025 at 4:12 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > >>
> > >> > Hello here,
> > >> >
> > >> > Next phase of the cleanup - it's been sped up by the comment from
> > >> @kxepal
> > >> > -
> https://github.com/apache/airflow/pull/46801#issuecomment-2661415731
> > >> -
> > >> > I
> > >> > have planned to do it a bit later this week, but maybe indeed it's a
> > >> good
> > >> > idea to start a discussion now so that people are not confused.
> > >> >
> > >> > Currently we are using "from tests_common" to reuse code in various
> > >> > providers and airflow, and this is fine, with the exception that
> > >> > "tests_common" is currently just a package in "airflow" main
> project.
> > >> But
> > >> > it does not have to be (or rather it should not be).
> > >> >
> > >> > We should have it in a separate sub-project - similarly as
> providers/*
> > >> and
> > >> > task_sdk are now separate projects - and we should add dependency
> to the
> > >> > common test code distribution from within all the projects that use
> it
> > >> (via
> > >> > workspace feature).
> > >> >
> > >> > My proposal is to name it "common_test_code" (but I am open to any
> > >> > suggestions): It will look like this
> > >> >
> > >> > .
> > >> > |- airflow
> > >> > |- common_test_code
> > >> > | pyproject.toml
> > >> > | src
> > >> > | tests_common
> > >> >
> > >> > The code in airflow and providers will not change, they will
> continue to
> > >> > use "from tests_common import ...".
> > >> >
> > >> > * uv will work as it used to work, no changes will be needed - uv
> sync
> > >> will
> > >> > automatically install the common_test_code as editable project
> > >> >
> > >> > * additionally - (and that's a big plus) after this change you
> should be
> > >> > able to run `uv sync` in a provider folder and have
> provider-specific
> > >> > separate venv, and basically you should be able to run tests for a
> > >> provider
> > >> > using that .venv and generally treat provider project as a
> "standalone"
> > >> one
> > >> > - this is what I hinted at in the previous mail.
> > >> >
> > >> > * people who do not use uv but pip for example will have to
> manually add
> > >> > the `common_test_code` as an editable install in their venv. For
> > >> example:
> > >> >
> > >> > pip install -e . -e ./task_sdk -e. ./providers/google -e
> > >> ./common_test_code
> > >> >
> > >> > This is the next step in standardizing the layout of the Airflow
> project
> > >> > using workspaces.
> > >> >
> > >> > J.
> > >> >
> > >>
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Vincent Beck <vi...@apache.org>.
Overall +1 on this one. Regarding the naming, why not keeping "tests_common" instead of "common_test_code"? I am not a big fan of "common_test_code" but it is obviously just a personal opinion (as it is always with naming :))
On 2025/02/16 13:30:09 Jarek Potiuk wrote:
> > Just wondernig... would an optional dependency not be the right place to
> describe that apache-airflow-providers-google[tests] would have an
> dependency to the common_tests subproject?
> > Would mean you would need to install via
> > pip install -e . -e ./task_sdk[tests] -e. ./providers/google[tests]
>
> Something like that. This is more of a details of workspace but we are
> going to use "dev" dependency group for that - rather than extra. Details
> to be worked out how pip interaction will look like (pip does not have
> support for dependency groups yet - they are coming in 25.1 - they are
> already merged in main)
>
> With `uv` dependency groups can be used today and "dev" dependency group is
> installed automatically when you run uv sync or `uv pip install -e.` -> so
> we will follow this along
>
> See details about "development dependencies" in uv here
> https://docs.astral.sh/uv/concepts/projects/dependencies/#development-dependencies
>
> Dependency groups are already approved via PEP-735
> https://peps.python.org/pep-0735/ and as mentioned - we are a release away
> from having them released in `pip`, so for now we will have to emulate it
> with extras for pip case I think but we will be able to remove it when pip
> 25.1 is released and matures a bit.
>
> J.
>
>
>
>
>
>
>
>
>
> On Sun, Feb 16, 2025 at 2:23 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > > I would love to see some airflow_testing package which will be useful for
> > testing airflow-related projects and involve independently.
> >
> > > Certainly, it's not a good thing to have tests import something from
> > tests.
> > New packages as projects are cheap and provide more flexibility and are
> > useful from outside of the project.
> >
> > I already explained in the PR, but let me repeat my opinion here:
> >
> > IMHO It's extremely unlikely we are going to release and publish the
> > common test code / fixtures in any way. They will continue to be in
> > development-only-distribution and they will be treated as "internal detail".
> >
> > If we decide to release and publish them, we will have to maintain
> > backwards compatibility and account for our users (like you) using them for
> > their own purpose. That would block us or make it very difficult to make
> > breaking changes in them.
> >
> > So while you will be free to continue copying the whole distribution and
> > use it in your tests as you want (our licence allows that) - I seriously
> > doubt we will ever release and publish it in "reusable form" with
> > "compatibility guarantees". It's far more efficient if people like you just
> > copy them and are aware that they can change any time.
> >
> > J.
> >
> > On Sun, Feb 16, 2025 at 2:21 PM Alexander Shorin <kx...@apache.org>
> > wrote:
> >
> >> I would love to see some airflow_testing package which will be useful for
> >> testing airflow-related projects and involve independently.
> >>
> >> Certainly, it's not a good thing to have tests import something from
> >> tests.
> >> New packages as projects are cheap and provide more flexibility and are
> >> useful from outside of the project.
> >>
> >> Also this project could have a future for testing, compatibility, quality
> >> and rest of measuring.
> >>
> >> --
> >> ,,,^..^,,,
> >>
> >>
> >> On Sun, Feb 16, 2025 at 4:12 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>
> >> > Hello here,
> >> >
> >> > Next phase of the cleanup - it's been sped up by the comment from
> >> @kxepal
> >> > - https://github.com/apache/airflow/pull/46801#issuecomment-2661415731
> >> -
> >> > I
> >> > have planned to do it a bit later this week, but maybe indeed it's a
> >> good
> >> > idea to start a discussion now so that people are not confused.
> >> >
> >> > Currently we are using "from tests_common" to reuse code in various
> >> > providers and airflow, and this is fine, with the exception that
> >> > "tests_common" is currently just a package in "airflow" main project.
> >> But
> >> > it does not have to be (or rather it should not be).
> >> >
> >> > We should have it in a separate sub-project - similarly as providers/*
> >> and
> >> > task_sdk are now separate projects - and we should add dependency to the
> >> > common test code distribution from within all the projects that use it
> >> (via
> >> > workspace feature).
> >> >
> >> > My proposal is to name it "common_test_code" (but I am open to any
> >> > suggestions): It will look like this
> >> >
> >> > .
> >> > |- airflow
> >> > |- common_test_code
> >> > | pyproject.toml
> >> > | src
> >> > | tests_common
> >> >
> >> > The code in airflow and providers will not change, they will continue to
> >> > use "from tests_common import ...".
> >> >
> >> > * uv will work as it used to work, no changes will be needed - uv sync
> >> will
> >> > automatically install the common_test_code as editable project
> >> >
> >> > * additionally - (and that's a big plus) after this change you should be
> >> > able to run `uv sync` in a provider folder and have provider-specific
> >> > separate venv, and basically you should be able to run tests for a
> >> provider
> >> > using that .venv and generally treat provider project as a "standalone"
> >> one
> >> > - this is what I hinted at in the previous mail.
> >> >
> >> > * people who do not use uv but pip for example will have to manually add
> >> > the `common_test_code` as an editable install in their venv. For
> >> example:
> >> >
> >> > pip install -e . -e ./task_sdk -e. ./providers/google -e
> >> ./common_test_code
> >> >
> >> > This is the next step in standardizing the layout of the Airflow project
> >> > using workspaces.
> >> >
> >> > J.
> >> >
> >>
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Jarek Potiuk <ja...@potiuk.com>.
> Just wondernig... would an optional dependency not be the right place to
describe that apache-airflow-providers-google[tests] would have an
dependency to the common_tests subproject?
> Would mean you would need to install via
> pip install -e . -e ./task_sdk[tests] -e. ./providers/google[tests]
Something like that. This is more of a details of workspace but we are
going to use "dev" dependency group for that - rather than extra. Details
to be worked out how pip interaction will look like (pip does not have
support for dependency groups yet - they are coming in 25.1 - they are
already merged in main)
With `uv` dependency groups can be used today and "dev" dependency group is
installed automatically when you run uv sync or `uv pip install -e.` -> so
we will follow this along
See details about "development dependencies" in uv here
https://docs.astral.sh/uv/concepts/projects/dependencies/#development-dependencies
Dependency groups are already approved via PEP-735
https://peps.python.org/pep-0735/ and as mentioned - we are a release away
from having them released in `pip`, so for now we will have to emulate it
with extras for pip case I think but we will be able to remove it when pip
25.1 is released and matures a bit.
J.
On Sun, Feb 16, 2025 at 2:23 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > I would love to see some airflow_testing package which will be useful for
> testing airflow-related projects and involve independently.
>
> > Certainly, it's not a good thing to have tests import something from
> tests.
> New packages as projects are cheap and provide more flexibility and are
> useful from outside of the project.
>
> I already explained in the PR, but let me repeat my opinion here:
>
> IMHO It's extremely unlikely we are going to release and publish the
> common test code / fixtures in any way. They will continue to be in
> development-only-distribution and they will be treated as "internal detail".
>
> If we decide to release and publish them, we will have to maintain
> backwards compatibility and account for our users (like you) using them for
> their own purpose. That would block us or make it very difficult to make
> breaking changes in them.
>
> So while you will be free to continue copying the whole distribution and
> use it in your tests as you want (our licence allows that) - I seriously
> doubt we will ever release and publish it in "reusable form" with
> "compatibility guarantees". It's far more efficient if people like you just
> copy them and are aware that they can change any time.
>
> J.
>
> On Sun, Feb 16, 2025 at 2:21 PM Alexander Shorin <kx...@apache.org>
> wrote:
>
>> I would love to see some airflow_testing package which will be useful for
>> testing airflow-related projects and involve independently.
>>
>> Certainly, it's not a good thing to have tests import something from
>> tests.
>> New packages as projects are cheap and provide more flexibility and are
>> useful from outside of the project.
>>
>> Also this project could have a future for testing, compatibility, quality
>> and rest of measuring.
>>
>> --
>> ,,,^..^,,,
>>
>>
>> On Sun, Feb 16, 2025 at 4:12 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> > Hello here,
>> >
>> > Next phase of the cleanup - it's been sped up by the comment from
>> @kxepal
>> > - https://github.com/apache/airflow/pull/46801#issuecomment-2661415731
>> -
>> > I
>> > have planned to do it a bit later this week, but maybe indeed it's a
>> good
>> > idea to start a discussion now so that people are not confused.
>> >
>> > Currently we are using "from tests_common" to reuse code in various
>> > providers and airflow, and this is fine, with the exception that
>> > "tests_common" is currently just a package in "airflow" main project.
>> But
>> > it does not have to be (or rather it should not be).
>> >
>> > We should have it in a separate sub-project - similarly as providers/*
>> and
>> > task_sdk are now separate projects - and we should add dependency to the
>> > common test code distribution from within all the projects that use it
>> (via
>> > workspace feature).
>> >
>> > My proposal is to name it "common_test_code" (but I am open to any
>> > suggestions): It will look like this
>> >
>> > .
>> > |- airflow
>> > |- common_test_code
>> > | pyproject.toml
>> > | src
>> > | tests_common
>> >
>> > The code in airflow and providers will not change, they will continue to
>> > use "from tests_common import ...".
>> >
>> > * uv will work as it used to work, no changes will be needed - uv sync
>> will
>> > automatically install the common_test_code as editable project
>> >
>> > * additionally - (and that's a big plus) after this change you should be
>> > able to run `uv sync` in a provider folder and have provider-specific
>> > separate venv, and basically you should be able to run tests for a
>> provider
>> > using that .venv and generally treat provider project as a "standalone"
>> one
>> > - this is what I hinted at in the previous mail.
>> >
>> > * people who do not use uv but pip for example will have to manually add
>> > the `common_test_code` as an editable install in their venv. For
>> example:
>> >
>> > pip install -e . -e ./task_sdk -e. ./providers/google -e
>> ./common_test_code
>> >
>> > This is the next step in standardizing the layout of the Airflow project
>> > using workspaces.
>> >
>> > J.
>> >
>>
>
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Jarek Potiuk <ja...@potiuk.com>.
> I would love to see some airflow_testing package which will be useful for
testing airflow-related projects and involve independently.
> Certainly, it's not a good thing to have tests import something from
tests.
New packages as projects are cheap and provide more flexibility and are
useful from outside of the project.
I already explained in the PR, but let me repeat my opinion here:
IMHO It's extremely unlikely we are going to release and publish the common
test code / fixtures in any way. They will continue to be in
development-only-distribution and they will be treated as "internal detail".
If we decide to release and publish them, we will have to maintain
backwards compatibility and account for our users (like you) using them for
their own purpose. That would block us or make it very difficult to make
breaking changes in them.
So while you will be free to continue copying the whole distribution and
use it in your tests as you want (our licence allows that) - I seriously
doubt we will ever release and publish it in "reusable form" with
"compatibility guarantees". It's far more efficient if people like you just
copy them and are aware that they can change any time.
J.
On Sun, Feb 16, 2025 at 2:21 PM Alexander Shorin <kx...@apache.org> wrote:
> I would love to see some airflow_testing package which will be useful for
> testing airflow-related projects and involve independently.
>
> Certainly, it's not a good thing to have tests import something from tests.
> New packages as projects are cheap and provide more flexibility and are
> useful from outside of the project.
>
> Also this project could have a future for testing, compatibility, quality
> and rest of measuring.
>
> --
> ,,,^..^,,,
>
>
> On Sun, Feb 16, 2025 at 4:12 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Hello here,
> >
> > Next phase of the cleanup - it's been sped up by the comment from
> @kxepal
> > - https://github.com/apache/airflow/pull/46801#issuecomment-2661415731
> -
> > I
> > have planned to do it a bit later this week, but maybe indeed it's a good
> > idea to start a discussion now so that people are not confused.
> >
> > Currently we are using "from tests_common" to reuse code in various
> > providers and airflow, and this is fine, with the exception that
> > "tests_common" is currently just a package in "airflow" main project. But
> > it does not have to be (or rather it should not be).
> >
> > We should have it in a separate sub-project - similarly as providers/*
> and
> > task_sdk are now separate projects - and we should add dependency to the
> > common test code distribution from within all the projects that use it
> (via
> > workspace feature).
> >
> > My proposal is to name it "common_test_code" (but I am open to any
> > suggestions): It will look like this
> >
> > .
> > |- airflow
> > |- common_test_code
> > | pyproject.toml
> > | src
> > | tests_common
> >
> > The code in airflow and providers will not change, they will continue to
> > use "from tests_common import ...".
> >
> > * uv will work as it used to work, no changes will be needed - uv sync
> will
> > automatically install the common_test_code as editable project
> >
> > * additionally - (and that's a big plus) after this change you should be
> > able to run `uv sync` in a provider folder and have provider-specific
> > separate venv, and basically you should be able to run tests for a
> provider
> > using that .venv and generally treat provider project as a "standalone"
> one
> > - this is what I hinted at in the previous mail.
> >
> > * people who do not use uv but pip for example will have to manually add
> > the `common_test_code` as an editable install in their venv. For example:
> >
> > pip install -e . -e ./task_sdk -e. ./providers/google -e
> ./common_test_code
> >
> > This is the next step in standardizing the layout of the Airflow project
> > using workspaces.
> >
> > J.
> >
>
Re: [DISCUSS] Turn "tests_common" into separate distribution for development
Posted by Alexander Shorin <kx...@apache.org>.
I would love to see some airflow_testing package which will be useful for
testing airflow-related projects and involve independently.
Certainly, it's not a good thing to have tests import something from tests.
New packages as projects are cheap and provide more flexibility and are
useful from outside of the project.
Also this project could have a future for testing, compatibility, quality
and rest of measuring.
--
,,,^..^,,,
On Sun, Feb 16, 2025 at 4:12 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hello here,
>
> Next phase of the cleanup - it's been sped up by the comment from @kxepal
> - https://github.com/apache/airflow/pull/46801#issuecomment-2661415731 -
> I
> have planned to do it a bit later this week, but maybe indeed it's a good
> idea to start a discussion now so that people are not confused.
>
> Currently we are using "from tests_common" to reuse code in various
> providers and airflow, and this is fine, with the exception that
> "tests_common" is currently just a package in "airflow" main project. But
> it does not have to be (or rather it should not be).
>
> We should have it in a separate sub-project - similarly as providers/* and
> task_sdk are now separate projects - and we should add dependency to the
> common test code distribution from within all the projects that use it (via
> workspace feature).
>
> My proposal is to name it "common_test_code" (but I am open to any
> suggestions): It will look like this
>
> .
> |- airflow
> |- common_test_code
> | pyproject.toml
> | src
> | tests_common
>
> The code in airflow and providers will not change, they will continue to
> use "from tests_common import ...".
>
> * uv will work as it used to work, no changes will be needed - uv sync will
> automatically install the common_test_code as editable project
>
> * additionally - (and that's a big plus) after this change you should be
> able to run `uv sync` in a provider folder and have provider-specific
> separate venv, and basically you should be able to run tests for a provider
> using that .venv and generally treat provider project as a "standalone" one
> - this is what I hinted at in the previous mail.
>
> * people who do not use uv but pip for example will have to manually add
> the `common_test_code` as an editable install in their venv. For example:
>
> pip install -e . -e ./task_sdk -e. ./providers/google -e ./common_test_code
>
> This is the next step in standardizing the layout of the Airflow project
> using workspaces.
>
> J.
>