[ANNOUNCE] Test suites are now separated to core/providers/task_sdk/integration/system

[Jarek Potiuk <ja...@potiuk.com>]

Hello here,

*TL;DR; We have a change in how our CI runs providers/core/system tests and
some change in breeze commands to run them, following the provider folder
move.*

There will likely be teething problems that we will address when they arise
- if you see unusual problems in your CI - let us know in the*
#internal-airflow-ci-cd* channel in slack.

I just merged a change that I've been working on last week
https://github.com/apache/airflow/pull/43979  which follows up the move of
providers to separate folders and introducing task_sdk - separating CI
tests of ours to "core" / "providers"/ "integration" / "system" /
"task_sdk". Historically they were together, because we had one folder with
all tests - the recent changes split those tests and sources out - this
change adapts the test harness of ours to that split.

Each of the tests has now separate command in "testing" command group of
Breeze:

[image: image.png]

For example if you want to use breeze to run provider tests in a single
command, you should use "providers-tests" (with corresponding type of
tests) and when you want to run some airflow core tests you should run
"core-tests" command:

breeze testing providers-tests --test-type Providers[amazon]
breeze testing core-tests --test-type CLI

There is no change to the bare `breeze` command. If you use `breeze` to
enter the shell, you will still be able to run both "core" and "providers"
tests using pytest - as usual.

Additionally system tests got their dedicated command, and the way to run
and define system tests have been cleaned up (we had a mixture of old and
new way of running tests introduced by AIP-47 - now both defining and
running system tests is not only consistent but also tested in our CI so
that it won't break accidentally (as it broke when we moved providers):

breeze testing system-tests providers/tests/system/example_empty.py

or with pytest

pytest --system  providers/tests/system/example_empty.py

Note - for the system tests you should not put "--system amazon" as it used
to be before AIP-47 - the `--system` is just a boolean flag now that allows
separating running system tests and regular tests.

This is the first (of four) stage of much more complete refactor  that is
planned here
https://github.com/apache/airflow/issues/42632#issuecomment-2449671014  -
and we will follow it step-by-step pausing after each step and fixing
teething problems before moving to next stage, to not disrupt the regular
contributions.

J.

Re: [ANNOUNCE] Test suites are now separated to core/providers/task_sdk/integration/system

[Amogh Desai <am...@gmail.com>]

👏👏👏

Thanks & Regards,
Amogh Desai


On Fri, Nov 15, 2024 at 3:22 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> Hello here,
>
> *TL;DR; We have a change in how our CI runs providers/core/system tests
> and some change in breeze commands to run them, following the provider
> folder move.*
>
> There will likely be teething problems that we will address when they
> arise - if you see unusual problems in your CI - let us know in the*
> #internal-airflow-ci-cd* channel in slack.
>
> I just merged a change that I've been working on last week
> https://github.com/apache/airflow/pull/43979  which follows up the move
> of providers to separate folders and introducing task_sdk - separating CI
> tests of ours to "core" / "providers"/ "integration" / "system" /
> "task_sdk". Historically they were together, because we had one folder with
> all tests - the recent changes split those tests and sources out - this
> change adapts the test harness of ours to that split.
>
> Each of the tests has now separate command in "testing" command group of
> Breeze:
>
> [image: image.png]
>
> For example if you want to use breeze to run provider tests in a single
> command, you should use "providers-tests" (with corresponding type of
> tests) and when you want to run some airflow core tests you should run
> "core-tests" command:
>
> breeze testing providers-tests --test-type Providers[amazon]
> breeze testing core-tests --test-type CLI
>
> There is no change to the bare `breeze` command. If you use `breeze` to
> enter the shell, you will still be able to run both "core" and "providers"
> tests using pytest - as usual.
>
> Additionally system tests got their dedicated command, and the way to run
> and define system tests have been cleaned up (we had a mixture of old and
> new way of running tests introduced by AIP-47 - now both defining and
> running system tests is not only consistent but also tested in our CI so
> that it won't break accidentally (as it broke when we moved providers):
>
> breeze testing system-tests providers/tests/system/example_empty.py
>
> or with pytest
>
> pytest --system  providers/tests/system/example_empty.py
>
> Note - for the system tests you should not put "--system amazon" as it
> used to be before AIP-47 - the `--system` is just a boolean flag now that
> allows separating running system tests and regular tests.
>
> This is the first (of four) stage of much more complete refactor  that is
> planned here
> https://github.com/apache/airflow/issues/42632#issuecomment-2449671014  -
> and we will follow it step-by-step pausing after each step and fixing
> teething problems before moving to next stage, to not disrupt the regular
> contributions.
>
> J.
>
>