Re: Grouping tests using pytest markers

[Jarek Potiuk <Ja...@polidea.com>]

Since we need to tackle CI stability I would like to bump this one if
anyone wants to say something :) I am full speed ahead with implementing
those.

*Proposed Breeze changes:*
>
>    - `./breeze` by default will start only the main 'airflow-testing'
>    image. This way no huge resource usage will be needed when breeze is
>    started by default
>    - './breeze --all-integrations` will start all dependent images (so we
>    will be able to run all tests)
>    - './breeze --integrations [kubernetes,cassandra,mongo,
>    rabbitmq,redis,openldap,kerberos] - you will be able to choose which
>    integrations you want to start
>    - When you run `breeze --backend postgres` it will only start postgres
>    not mysql and the other way round.
>
> I have PR in progress: https://github.com/apache/airflow/pull/7091
(depends on few others)

After this is merged ./breeze will only start 'airflow-testing' image. You
will be able to launch other docker images (mongo/cassandra and others)
with *--integration mongo --integration cassandra* etc. (or --integration
all to launch all of them). This will be great for local testing (resource
usage!). This will also work in CI ( I will split the test jobs into
separate ones).


> *Proposed Pytest marks:*
>
>    -
>    pytest.mark.integrations('kubernetes'),pytest.mark.integrations('cassandra'),.....
>    - pytest,mark.backends("postgres"), pytest,mark.backends("mysql"),
>    pytest.mark.backends("sqlite")
>
> During tests I will identify the tests that need particular integrations
and will mark/skip them appropriately and work out the right pytest
behaviour.

*Proposed Pytest behaviour:*
>
>    - `pytest` -> in Breeze will run all tests that are applicable within
>    the current environment:
>       - it will only run non-marked tests by default, applicable with
>       current selected backend
>       - when (for example) you stared cassandra is added it will
>       additionally run pytest.mark.integrations('cassandra')
>    - `pytest` in local environment by default will only run non-marked
>    tests
>    - `pytest --integrations [kubernetes, ....]` will only run the
>    integration tests selected (will convert the switch into the corresponding
>    markers (as explained in the example above)
>    - `pytest --backends [postgres| mysql | sqlite] will only run the
>    specific tests that use postgres/mysql/sqlite specific tests
>
> More details when I get to this one. Ideally all should be autodetected -
i.e. when you have no integration enabled, the corresponding tests should
be skipped, we should also be able to run tests for particular integration
or selected integrations with one command.

J.