You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2024/01/28 20:33:46 UTC
(airflow) branch main updated: Update documentation about System Tests (#37062)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 6f41010269 Update documentation about System Tests (#37062)
6f41010269 is described below
commit 6f41010269e4a2c052b564f8f0f977abbaf5dfc7
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Jan 28 21:33:39 2024 +0100
Update documentation about System Tests (#37062)
The documentation was a little out-dated.
---
contributing-docs/testing/system_tests.rst | 115 +++++++++--------------------
1 file changed, 35 insertions(+), 80 deletions(-)
diff --git a/contributing-docs/testing/system_tests.rst b/contributing-docs/testing/system_tests.rst
index a8a1a9fc85..f9add8b377 100644
--- a/contributing-docs/testing/system_tests.rst
+++ b/contributing-docs/testing/system_tests.rst
@@ -20,16 +20,40 @@ Airflow System Tests
System tests need to communicate with external services/systems that are available
if you have appropriate credentials configured for your tests.
-The system tests derive from the ``tests.test_utils.system_test_class.SystemTests`` class. They should also
-be marked with ``@pytest.marker.system(SYSTEM)`` where ``system`` designates the system
-to be tested (for example, ``google.cloud``). These tests are skipped by default.
-You can execute the system tests by providing the ``--system SYSTEM`` flag to ``pytest``. You can
-specify several --system flags if you want to execute tests for several systems.
+.. contents:: :local:
+
+Purpose of System Tests
+-----------------------
+
+Airflow system tests are pretty special because they serve three purposes:
+
+* they are runnable tests that communicate with external services
+* they are example_dags that users can just copy&paste and use as starting points for their own DAGs
+* the excerpts from these system tests are used to generate documentation
+
+Old System Tests
+----------------
+
+The system tests derive from the ``tests.test_utils.system_test_class.SystemTests`` class.
+
+Old versions of System tests should also be marked with ``@pytest.marker.system(SYSTEM)`` where ``system``
+designates the system to be tested (for example, ``google.cloud``). These tests are skipped by default.
+
+For older version of the tests, you can execute the system tests by providing the
+``--system SYSTEM`` flag to ``pytest``. You can specify several --system flags if you want to execute
+tests for several systems.
The system tests execute a specified example DAG file that runs the DAG end-to-end.
-.. contents:: :local:
+New System Tests
+----------------
+
+The new system tests follows
+[AIP-47 New Design of System Tests](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-47+New+design+of+Airflow+System+Tests)
+and those system tests do not require separate ``pytest`` flag to be executed, they also don't need a separate
+class to run - all the code is kept in the system test class that is also an executable DAG, it is discoverable
+by ``pytest`` and it is executable as Python script.
Environment for System Tests
----------------------------
@@ -74,13 +98,8 @@ run Google Cloud system tests.
RANDOM_POSTFIX=$(cat "${RANDOM_FILE}")
-
-To execute system tests, specify the ``--system SYSTEM``
-flag where ``SYSTEM`` is a system to run the system tests for. It can be repeated.
-
-
Forwarding Authentication from the Host
-----------------------------------------------------
+---------------------------------------
For system tests, you can also forward authentication from the host to your Breeze container. You can specify
the ``--forward-credentials`` flag when starting Breeze. Then, it will also forward the most commonly used
@@ -94,75 +113,11 @@ Currently forwarded credentials are:
* credentials stored in ``${HOME}/.docker`` for ``docker`` client
* credentials stored in ``${HOME}/.snowsql`` for ``snowsql`` - SnowSQL (Snowflake CLI client)
-Adding a New System Test
---------------------------
-
-We are working on automating system tests execution (AIP-4) but for now, system tests are skipped when
-tests are run in our CI system. But to enable the test automation, we encourage you to add system
-tests whenever an operator/hook/sensor is added/modified in a given system.
-
-* To add your own system tests, derive them from the
- ``tests.test_utils.system_tests_class.SystemTest`` class and mark with the
- ``@pytest.mark.system(SYSTEM_NAME)`` marker. The system name should follow the path defined in
- the ``providers`` package (for example, the system tests from ``tests.providers.google.cloud``
- package should be marked with ``@pytest.mark.system("google.cloud")``.
-
-* If your system tests need some credential files to be available for an
- authentication with external systems, make sure to keep these credentials in the
- ``files/airflow-breeze-config/keys`` directory. Mark your tests with
- ``@pytest.mark.credential_file(<FILE>)`` so that they are skipped if such a credential file is not there.
- The tests should read the right credentials and authenticate them on their own. The credentials are read
- in Breeze from the ``/files`` directory. The local "files" folder is mounted to the "/files" folder in Breeze.
-
-* If your system tests are long-running ones (i.e., require more than 20-30 minutes
- to complete), mark them with the ```@pytest.markers.long_running`` marker.
- Such tests are skipped by default unless you specify the ``--long-running`` flag to pytest.
-
-* The system test itself (python class) does not have any logic. Such a test runs
- the DAG specified by its ID. This DAG should contain the actual DAG logic
- to execute. Make sure to define the DAG in ``providers/<SYSTEM_NAME>/example_dags``. These example DAGs
- are also used to take some snippets of code out of them when documentation is generated. So, having these
- DAGs runnable is a great way to make sure the documentation is describing a working example. Inside
- your test class/test method, simply use ``self.run_dag(<DAG_ID>,<DAG_FOLDER>)`` to run the DAG. Then,
- the system class will take care about running the DAG. Note that the DAG_FOLDER should be
- a subdirectory of the ``tests.test_utils.AIRFLOW_MAIN_FOLDER`` + ``providers/<SYSTEM_NAME>/example_dags``.
-
-
-A simple example of a system test is available in:
-
-``tests/providers/google/cloud/operators/test_compute_system.py``.
-
-It runs two DAGs defined in ``airflow.providers.google.cloud.example_dags.example_compute.py``.
-
-
-The typical system test session
--------------------------------
-
-Here is the typical session that you need to do to run system tests:
-
-1. Enter breeze
-
-.. code-block:: bash
-
- breeze down
- breeze --python 3.8 --db-reset --forward-credentials
-
-This will:
-
-* stop the whole environment (i.e. recreates metadata database from the scratch)
-* run Breeze with:
-
- * python 3.8 version
- * resetting the Airflow database
- * forward your local credentials to Breeze
-
-3. Run the tests:
-
-.. code-block:: bash
+Running the System Tests
+------------------------
- pytest -o faulthandler_timeout=2400 \
- --system=google tests/providers/google/cloud/operators/test_compute_system.py
+Running the tests and developing tests in `Test documentation <../../tests/system/README.md>`__
------
+------
For other kinds of tests look at `Testing document <../09_testing.rst>`__