Automated system tests - AIP4

[Tobiasz Kędzierski <to...@gmail.com>]

Hello everyone,

I worked intensively on the new operators and also made some improvements
and reviews of PRs to existing ones.
I noticed that sometimes a code which looks valid and passing all tests
finally didn't work with the service. It could be small mistake which
passed uncatched or some external service made with time some breaking
changes to its API.
The most reliable way to verify is operator really working is to run system
test for it.

There is a great Breeze environment which allows to do this is easy way.
However creating  Google Cloud or AWS project, configuring it to work it
with local Breeze environment may be time consuming and error prone.
Running system tests also costs money.
Due to this reasons system tests that are not being run frequently may
become problematic to run. In my opinion all of that makes system tests
little bit abandoned however I see a lot of potential benefits of running
system tests in scheduled manner.
The most important one is that it allow to catch problems with the
operators before they will reported by the users.

Long time ago Jarek created AIP-4 [1][2] which describes automation of
system tests.
I wanted to refresh it and due to this
https://github.com/PolideaInternal/airflow-system-tests repository was
published.
It describes how to setup Google Cloud Platform project and consists of
configuration which allows to run tests in Breeze within Google Cloud Build.
Additional scripts allows to trigger bunch of system tests with one command
(they could be triggered periodically by scheduler).

It is not perfect, still need some work, but by using it I was able to run
several tests related to GCP once a week and I am looking for a way to do
it within community.
I would appreciate hearing your opinion on this.

Best regards
Tobiasz

https://github.com/PolideaInternal/airflow-system-tests
[1] https://github.com/apache/airflow/issues/10835
[2]
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+Automation+of+System+Tests+for+external+systems