[GitHub] [airflow] potiuk commented on a change in pull request #7774: [AIRFLOW-XXXX] document system tests mechanism better

[GitBox <gi...@apache.org>]

potiuk commented on a change in pull request #7774: [AIRFLOW-XXXX] document system tests mechanism better
URL: https://github.com/apache/airflow/pull/7774#discussion_r395534761
 
 

 ##########
 File path: TESTING.rst
 ##########
 @@ -543,31 +587,197 @@ tests whenever an operator/hook/sensor is added/modified in a given system.
   the system class will take care about running the DAG. Note that the DAG_FOLDER should be
   a subdirectory of the ``tests.test_utils.AIRFLOW_MAIN_FOLDER`` + ``providers/<SYSTEM_NAME>/example_dags``.
 
-An example of a system test is available in:
 
-``airflow.tests.providers.google.operators.test_natunal_language_system.CloudNaturalLanguageExampleDagsTest``.
+A simple example of a system test is available in:
+
+``tests/providers/google/cloud/operators/test_compute_system.py``.
+
+It runs two DAGs defined in ``airflow.providers.google.cloud.example_dags.example_compute.py`` and
+``airflow.providers.google.cloud.example_dags.example_compute_igm.py``.
+
+Preparing backport packages for System Tests for Airflow 1.10.* series
+----------------------------------------------------------------------
+
+In order to run system tests with old Airflow version you need to prepare backport packages. This
+can be done by running ``./scripts/ci/ci_prepare_bacport_packages.sh <PACKAGES TO BUILD>``. For
+example the below command will build google postgres and mysql packages:
+
+.. code-block:: bash
+
+  ./scripts/ci/ci_prepare_bacport_packages.sh google postgres mysql
+
+Those packages will be prepared in ./dist folder. This folder is mapped to /dist folder
+when you enter Breeze, so it is easy to automate installing of those packages for testing.
 
-It runs the DAG defined in ``airflow.providers.google.cloud.example_dags.example_natural_language.py``.
 
-Running Tests for Older Airflow Versions
-----------------------------------------
+Installing backported for Airflow 1.10.* series
+-----------------------------------------------
 
 The tests can be executed against the master version of Airflow but they also work
 with older versions. This is especially useful to test back-ported operators
 from Airflow 2.0 to 1.10.* versions.
 
 To run the tests for Airflow 1.10.* series, you need to run Breeze with
-``--install-airflow-version==<VERSION>`` to install a different version of Airflow.
+``--install-airflow-version=<VERSION>`` to install a different version of Airflow.
 If ``current`` is specified (default), then the current version of Airflow is used.
 Otherwise, the released version of Airflow is installed.
 
-The commands make sure that the source version of master Airflow is removed and the released version of
-Airflow from ``Pypi`` is installed. Note that tests sources are not removed and they can be used
-to run tests (unit tests and system tests) against the freshly installed version.
+The ``-install-airflow-version=<VERSION>`` command make sure that the current (from sources) version of
+Airflow is removed and the released version of Airflow from ``Pypi`` is installed. Note that tests sources
+are not removed and they can be used to run tests (unit tests and system tests) against the
+freshly installed version.
+
+You should automate installing of the backport packages in your own
+``./files/airflow-breeze-config/variables.env`` file. You should make it depend on
+``RUN_AIRFLOW_1_10`` variable value equals to "true" so that
+installation of backport packages is only performed when you install airflow 1.10.*.
+The backport packages are available in ``/dist`` directory if they were prepared as described
+in the previous chapter.
+
+Typically the command in you variables.env file will be similar to:
+
+.. code-block:: bash
+
+  # install any packages from dist folder if they are available
+  if [[ ${RUN_AIRFLOW_1_10:=} == "true" ]]; then
+      pip install /dist/apache_airflow_providers_{google,postgres,mysql}*.whl || true
+  fi
+
+The command above will automatically install backported google, postgres amd mysql packages if they
+were prepared before entering breeze.
+
+
+Running system tests for backported packages in Airflow 1.10.* series
+---------------------------------------------------------------------
+
+Once you installed 1.10.* Airflow version with ``--install-airflow-version`` and prepared and
+installed the required packages via ``variables.env`` it should be as easy as running
+``pytest --systems=<SYSTEM_NAME> TEST_NAME``. Note that we have default timeout for runnning
+system tests set to 8 minutes and some system tests might take much longer to run and you might
+want to add ``-o faulthandler_timeout=2400`` (2400s = 40 minutes for example) to your
+pytest command.
+
+Typical system test session
+---------------------------
+
+Here is the typical session that you need to do to run system tests:
+
+1. Prepare backport packages
+
+.. code-block:: bash
+
+  ./scripts/ci/ci_prepare_bacport_packages.sh google postgres mysql
+
+2. Enter breeze with installing Airflow 1.10.*, forwarding credentials and installing
+   backported packages (you need appropriate line in ``./files/airflow-breeze-config/variables.env``)
+
+.. code-block:: bash
+
+   ./breeze --install-airflow-version 1.10.9 --python 3.6 --db-reset --forward-credentials restart
+
+This will:
+
+* install Airflow 1.10.9
+* restarts the whole environment (i.e. recreates metadata database from the scratch)
+* run Breeze with python 3.6 version
+* reset the Airflow database
+* forward your local credentials to Breeze
+
+3. Run the tests:
+
+.. code-block:: bash
+
+   pytest -o faulthandler_timeout=2400 --systems=google tests/providers/google/cloud/operators/test_compute_system.py
+
+
+Iteration with System Tests if your resources are slow to create
+----------------------------------------------------------------
+
+When you want to iterate on system tests, you might want to create slow resources first
+
+If you need to setup some external resources for your tests (for example compute instances in Google Cloud)
+you should set them up and teardown in the setUp/tearDown methods of your tests.
+Since those resources might be slow to create you might want to add some helpers that
+set them up and tear them down separately via manual operations so that you can iterate on
+the tests without waiting for setUp and tearDown with every test.
+
+In this case you should build in a mechanism to skip setUp and tearDown in case you manually
+created the resources. Somewhat complex example of that can be found in
+``tests.providers.google.cloud.operators.test_cloud_sql_system.py`` and the helper is
+available in ``tests.providers.google.cloud.operators.test_cloud_sql_system_helper.py``.
+
+When the helper is run with ``--action create`` to create cloud sql instances which are very slow
+to create and set-up so that you can iterate on running the system tests without
+loosing the time for creating theme every time. A temporary file is created to prevent from
+setting up and tearing down the instances when running the test.
+
+This example also shows how you can use the random number generated at the entry of Breeze if you
+have it in your variables.env (see previous chapter). In case of Cloud SQL you cannot reuse the
+same instance name for a week so we generate random number that is used across the whole session
+and store it in ``/random.txt`` file so that the names are unique during tests.
+
+NOTE! Do not forget to delete manually created resources before leaving the Breeze session. They
 
 Review comment:
   Highlighted!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services