You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/03/20 08:18:05 UTC
[GitHub] [airflow] potiuk opened a new pull request #7774: [AIRFLOW-XXXX]
document system tests mechanism better
potiuk opened a new pull request #7774: [AIRFLOW-XXXX] document system tests mechanism better
URL: https://github.com/apache/airflow/pull/7774
---
Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7774: [AIRFLOW-XXXX] document system tests mechanism better
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7774: [AIRFLOW-XXXX] document system tests mechanism better
URL: https://github.com/apache/airflow/pull/7774#discussion_r395523833
##########
File path: TESTING.rst
##########
@@ -543,31 +587,197 @@ tests whenever an operator/hook/sensor is added/modified in a given system.
the system class will take care about running the DAG. Note that the DAG_FOLDER should be
a subdirectory of the ``tests.test_utils.AIRFLOW_MAIN_FOLDER`` + ``providers/<SYSTEM_NAME>/example_dags``.
-An example of a system test is available in:
-``airflow.tests.providers.google.operators.test_natunal_language_system.CloudNaturalLanguageExampleDagsTest``.
+A simple example of a system test is available in:
+
+``tests/providers/google/cloud/operators/test_compute_system.py``.
+
+It runs two DAGs defined in ``airflow.providers.google.cloud.example_dags.example_compute.py`` and
+``airflow.providers.google.cloud.example_dags.example_compute_igm.py``.
+
+Preparing backport packages for System Tests for Airflow 1.10.* series
+----------------------------------------------------------------------
+
+In order to run system tests with old Airflow version you need to prepare backport packages. This
+can be done by running ``./scripts/ci/ci_prepare_bacport_packages.sh <PACKAGES TO BUILD>``. For
+example the below command will build google postgres and mysql packages:
+
+.. code-block:: bash
+
+ ./scripts/ci/ci_prepare_bacport_packages.sh google postgres mysql
+
+Those packages will be prepared in ./dist folder. This folder is mapped to /dist folder
+when you enter Breeze, so it is easy to automate installing of those packages for testing.
-It runs the DAG defined in ``airflow.providers.google.cloud.example_dags.example_natural_language.py``.
-Running Tests for Older Airflow Versions
-----------------------------------------
+Installing backported for Airflow 1.10.* series
+-----------------------------------------------
The tests can be executed against the master version of Airflow but they also work
with older versions. This is especially useful to test back-ported operators
from Airflow 2.0 to 1.10.* versions.
To run the tests for Airflow 1.10.* series, you need to run Breeze with
-``--install-airflow-version==<VERSION>`` to install a different version of Airflow.
+``--install-airflow-version=<VERSION>`` to install a different version of Airflow.
If ``current`` is specified (default), then the current version of Airflow is used.
Otherwise, the released version of Airflow is installed.
-The commands make sure that the source version of master Airflow is removed and the released version of
-Airflow from ``Pypi`` is installed. Note that tests sources are not removed and they can be used
-to run tests (unit tests and system tests) against the freshly installed version.
+The ``-install-airflow-version=<VERSION>`` command make sure that the current (from sources) version of
+Airflow is removed and the released version of Airflow from ``Pypi`` is installed. Note that tests sources
+are not removed and they can be used to run tests (unit tests and system tests) against the
+freshly installed version.
+
+You should automate installing of the backport packages in your own
+``./files/airflow-breeze-config/variables.env`` file. You should make it depend on
+``RUN_AIRFLOW_1_10`` variable value equals to "true" so that
+installation of backport packages is only performed when you install airflow 1.10.*.
+The backport packages are available in ``/dist`` directory if they were prepared as described
+in the previous chapter.
+
+Typically the command in you variables.env file will be similar to:
+
+.. code-block:: bash
+
+ # install any packages from dist folder if they are available
+ if [[ ${RUN_AIRFLOW_1_10:=} == "true" ]]; then
+ pip install /dist/apache_airflow_providers_{google,postgres,mysql}*.whl || true
+ fi
+
+The command above will automatically install backported google, postgres amd mysql packages if they
+were prepared before entering breeze.
+
+
+Running system tests for backported packages in Airflow 1.10.* series
+---------------------------------------------------------------------
+
+Once you installed 1.10.* Airflow version with ``--install-airflow-version`` and prepared and
+installed the required packages via ``variables.env`` it should be as easy as running
+``pytest --systems=<SYSTEM_NAME> TEST_NAME``. Note that we have default timeout for runnning
+system tests set to 8 minutes and some system tests might take much longer to run and you might
+want to add ``-o faulthandler_timeout=2400`` (2400s = 40 minutes for example) to your
+pytest command.
+
+Typical system test session
+---------------------------
+
+Here is the typical session that you need to do to run system tests:
+
+1. Prepare backport packages
+
+.. code-block:: bash
+
+ ./scripts/ci/ci_prepare_bacport_packages.sh google postgres mysql
+
+2. Enter breeze with installing Airflow 1.10.*, forwarding credentials and installing
+ backported packages (you need appropriate line in ``./files/airflow-breeze-config/variables.env``)
+
+.. code-block:: bash
+
+ ./breeze --install-airflow-version 1.10.9 --python 3.6 --db-reset --forward-credentials restart
+
+This will:
+
+* install Airflow 1.10.9
+* restarts the whole environment (i.e. recreates metadata database from the scratch)
+* run Breeze with python 3.6 version
+* reset the Airflow database
+* forward your local credentials to Breeze
+
+3. Run the tests:
+
+.. code-block:: bash
+
+ pytest -o faulthandler_timeout=2400 --systems=google tests/providers/google/cloud/operators/test_compute_system.py
+
+
+Iteration with System Tests if your resources are slow to create
+----------------------------------------------------------------
+
+When you want to iterate on system tests, you might want to create slow resources first
Review comment:
When you want to iterate on system tests, you might want to create slow resources first
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7774: [AIRFLOW-XXXX] document
system tests mechanism better
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7774: [AIRFLOW-XXXX] document system tests mechanism better
URL: https://github.com/apache/airflow/pull/7774#issuecomment-601617976
Thanks fixed all points :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7774: [AIRFLOW-XXXX] document system tests mechanism better
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7774: [AIRFLOW-XXXX] document system tests mechanism better
URL: https://github.com/apache/airflow/pull/7774#discussion_r395521490
##########
File path: TESTING.rst
##########
@@ -486,16 +486,57 @@ The system tests execute a specified example DAG file that runs the DAG end-to-e
See more details about adding new system tests below.
-Running System Tests
---------------------
+Environment for System Tests
+----------------------------
+
**Prerequisites:** You may need to set some variables to run system tests. If you need to
-add some intialization of environment variables to Breeze, you can always add a
+add some initialization of environment variables to Breeze, you can add a
``variables.env`` file in the ``files/airflow-breeze-config/variables.env`` file. It will be automatically
-sourced when entering the Breeze environment.
+sourced when entering the Breeze environment. You can also add some additional initialization commands
Review comment:
Where those additional commands can be added? In `files/airflow-breeze-config/variables.env`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7774: [AIRFLOW-XXXX] document system tests mechanism better
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7774: [AIRFLOW-XXXX] document system tests mechanism better
URL: https://github.com/apache/airflow/pull/7774#discussion_r395525570
##########
File path: TESTING.rst
##########
@@ -543,31 +587,197 @@ tests whenever an operator/hook/sensor is added/modified in a given system.
the system class will take care about running the DAG. Note that the DAG_FOLDER should be
a subdirectory of the ``tests.test_utils.AIRFLOW_MAIN_FOLDER`` + ``providers/<SYSTEM_NAME>/example_dags``.
-An example of a system test is available in:
-``airflow.tests.providers.google.operators.test_natunal_language_system.CloudNaturalLanguageExampleDagsTest``.
+A simple example of a system test is available in:
+
+``tests/providers/google/cloud/operators/test_compute_system.py``.
+
+It runs two DAGs defined in ``airflow.providers.google.cloud.example_dags.example_compute.py`` and
+``airflow.providers.google.cloud.example_dags.example_compute_igm.py``.
+
+Preparing backport packages for System Tests for Airflow 1.10.* series
+----------------------------------------------------------------------
+
+In order to run system tests with old Airflow version you need to prepare backport packages. This
+can be done by running ``./scripts/ci/ci_prepare_bacport_packages.sh <PACKAGES TO BUILD>``. For
+example the below command will build google postgres and mysql packages:
+
+.. code-block:: bash
+
+ ./scripts/ci/ci_prepare_bacport_packages.sh google postgres mysql
+
+Those packages will be prepared in ./dist folder. This folder is mapped to /dist folder
+when you enter Breeze, so it is easy to automate installing of those packages for testing.
-It runs the DAG defined in ``airflow.providers.google.cloud.example_dags.example_natural_language.py``.
-Running Tests for Older Airflow Versions
-----------------------------------------
+Installing backported for Airflow 1.10.* series
+-----------------------------------------------
The tests can be executed against the master version of Airflow but they also work
with older versions. This is especially useful to test back-ported operators
from Airflow 2.0 to 1.10.* versions.
To run the tests for Airflow 1.10.* series, you need to run Breeze with
-``--install-airflow-version==<VERSION>`` to install a different version of Airflow.
+``--install-airflow-version=<VERSION>`` to install a different version of Airflow.
If ``current`` is specified (default), then the current version of Airflow is used.
Otherwise, the released version of Airflow is installed.
-The commands make sure that the source version of master Airflow is removed and the released version of
-Airflow from ``Pypi`` is installed. Note that tests sources are not removed and they can be used
-to run tests (unit tests and system tests) against the freshly installed version.
+The ``-install-airflow-version=<VERSION>`` command make sure that the current (from sources) version of
+Airflow is removed and the released version of Airflow from ``Pypi`` is installed. Note that tests sources
+are not removed and they can be used to run tests (unit tests and system tests) against the
+freshly installed version.
+
+You should automate installing of the backport packages in your own
+``./files/airflow-breeze-config/variables.env`` file. You should make it depend on
+``RUN_AIRFLOW_1_10`` variable value equals to "true" so that
+installation of backport packages is only performed when you install airflow 1.10.*.
+The backport packages are available in ``/dist`` directory if they were prepared as described
+in the previous chapter.
+
+Typically the command in you variables.env file will be similar to:
+
+.. code-block:: bash
+
+ # install any packages from dist folder if they are available
+ if [[ ${RUN_AIRFLOW_1_10:=} == "true" ]]; then
+ pip install /dist/apache_airflow_providers_{google,postgres,mysql}*.whl || true
+ fi
+
+The command above will automatically install backported google, postgres amd mysql packages if they
+were prepared before entering breeze.
+
+
+Running system tests for backported packages in Airflow 1.10.* series
+---------------------------------------------------------------------
+
+Once you installed 1.10.* Airflow version with ``--install-airflow-version`` and prepared and
+installed the required packages via ``variables.env`` it should be as easy as running
+``pytest --systems=<SYSTEM_NAME> TEST_NAME``. Note that we have default timeout for runnning
+system tests set to 8 minutes and some system tests might take much longer to run and you might
+want to add ``-o faulthandler_timeout=2400`` (2400s = 40 minutes for example) to your
+pytest command.
+
+Typical system test session
+---------------------------
+
+Here is the typical session that you need to do to run system tests:
+
+1. Prepare backport packages
+
+.. code-block:: bash
+
+ ./scripts/ci/ci_prepare_bacport_packages.sh google postgres mysql
+
+2. Enter breeze with installing Airflow 1.10.*, forwarding credentials and installing
+ backported packages (you need appropriate line in ``./files/airflow-breeze-config/variables.env``)
+
+.. code-block:: bash
+
+ ./breeze --install-airflow-version 1.10.9 --python 3.6 --db-reset --forward-credentials restart
+
+This will:
+
+* install Airflow 1.10.9
+* restarts the whole environment (i.e. recreates metadata database from the scratch)
+* run Breeze with python 3.6 version
+* reset the Airflow database
+* forward your local credentials to Breeze
+
+3. Run the tests:
+
+.. code-block:: bash
+
+ pytest -o faulthandler_timeout=2400 --systems=google tests/providers/google/cloud/operators/test_compute_system.py
+
+
+Iteration with System Tests if your resources are slow to create
+----------------------------------------------------------------
+
+When you want to iterate on system tests, you might want to create slow resources first
+
+If you need to setup some external resources for your tests (for example compute instances in Google Cloud)
+you should set them up and teardown in the setUp/tearDown methods of your tests.
+Since those resources might be slow to create you might want to add some helpers that
+set them up and tear them down separately via manual operations so that you can iterate on
+the tests without waiting for setUp and tearDown with every test.
+
+In this case you should build in a mechanism to skip setUp and tearDown in case you manually
+created the resources. Somewhat complex example of that can be found in
+``tests.providers.google.cloud.operators.test_cloud_sql_system.py`` and the helper is
+available in ``tests.providers.google.cloud.operators.test_cloud_sql_system_helper.py``.
+
+When the helper is run with ``--action create`` to create cloud sql instances which are very slow
+to create and set-up so that you can iterate on running the system tests without
+loosing the time for creating theme every time. A temporary file is created to prevent from
+setting up and tearing down the instances when running the test.
+
+This example also shows how you can use the random number generated at the entry of Breeze if you
+have it in your variables.env (see previous chapter). In case of Cloud SQL you cannot reuse the
+same instance name for a week so we generate random number that is used across the whole session
+and store it in ``/random.txt`` file so that the names are unique during tests.
+
+NOTE! Do not forget to delete manually created resources before leaving the Breeze session. They
Review comment:
Can you mark it red or highlighted to make it more visible. WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7774: [AIRFLOW-XXXX] document system tests mechanism better
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7774: [AIRFLOW-XXXX] document system tests mechanism better
URL: https://github.com/apache/airflow/pull/7774#discussion_r395526260
##########
File path: TESTING.rst
##########
@@ -543,31 +587,197 @@ tests whenever an operator/hook/sensor is added/modified in a given system.
the system class will take care about running the DAG. Note that the DAG_FOLDER should be
a subdirectory of the ``tests.test_utils.AIRFLOW_MAIN_FOLDER`` + ``providers/<SYSTEM_NAME>/example_dags``.
-An example of a system test is available in:
-``airflow.tests.providers.google.operators.test_natunal_language_system.CloudNaturalLanguageExampleDagsTest``.
+A simple example of a system test is available in:
+
+``tests/providers/google/cloud/operators/test_compute_system.py``.
+
+It runs two DAGs defined in ``airflow.providers.google.cloud.example_dags.example_compute.py`` and
+``airflow.providers.google.cloud.example_dags.example_compute_igm.py``.
+
+Preparing backport packages for System Tests for Airflow 1.10.* series
+----------------------------------------------------------------------
+
+In order to run system tests with old Airflow version you need to prepare backport packages. This
+can be done by running ``./scripts/ci/ci_prepare_bacport_packages.sh <PACKAGES TO BUILD>``. For
+example the below command will build google postgres and mysql packages:
+
+.. code-block:: bash
+
+ ./scripts/ci/ci_prepare_bacport_packages.sh google postgres mysql
+
+Those packages will be prepared in ./dist folder. This folder is mapped to /dist folder
+when you enter Breeze, so it is easy to automate installing of those packages for testing.
-It runs the DAG defined in ``airflow.providers.google.cloud.example_dags.example_natural_language.py``.
-Running Tests for Older Airflow Versions
-----------------------------------------
+Installing backported for Airflow 1.10.* series
+-----------------------------------------------
The tests can be executed against the master version of Airflow but they also work
with older versions. This is especially useful to test back-ported operators
from Airflow 2.0 to 1.10.* versions.
To run the tests for Airflow 1.10.* series, you need to run Breeze with
-``--install-airflow-version==<VERSION>`` to install a different version of Airflow.
+``--install-airflow-version=<VERSION>`` to install a different version of Airflow.
If ``current`` is specified (default), then the current version of Airflow is used.
Otherwise, the released version of Airflow is installed.
-The commands make sure that the source version of master Airflow is removed and the released version of
-Airflow from ``Pypi`` is installed. Note that tests sources are not removed and they can be used
-to run tests (unit tests and system tests) against the freshly installed version.
+The ``-install-airflow-version=<VERSION>`` command make sure that the current (from sources) version of
+Airflow is removed and the released version of Airflow from ``Pypi`` is installed. Note that tests sources
+are not removed and they can be used to run tests (unit tests and system tests) against the
+freshly installed version.
+
+You should automate installing of the backport packages in your own
+``./files/airflow-breeze-config/variables.env`` file. You should make it depend on
+``RUN_AIRFLOW_1_10`` variable value equals to "true" so that
+installation of backport packages is only performed when you install airflow 1.10.*.
+The backport packages are available in ``/dist`` directory if they were prepared as described
+in the previous chapter.
+
+Typically the command in you variables.env file will be similar to:
+
+.. code-block:: bash
+
+ # install any packages from dist folder if they are available
+ if [[ ${RUN_AIRFLOW_1_10:=} == "true" ]]; then
+ pip install /dist/apache_airflow_providers_{google,postgres,mysql}*.whl || true
+ fi
+
+The command above will automatically install backported google, postgres amd mysql packages if they
+were prepared before entering breeze.
+
+
+Running system tests for backported packages in Airflow 1.10.* series
+---------------------------------------------------------------------
+
+Once you installed 1.10.* Airflow version with ``--install-airflow-version`` and prepared and
+installed the required packages via ``variables.env`` it should be as easy as running
+``pytest --systems=<SYSTEM_NAME> TEST_NAME``. Note that we have default timeout for runnning
+system tests set to 8 minutes and some system tests might take much longer to run and you might
+want to add ``-o faulthandler_timeout=2400`` (2400s = 40 minutes for example) to your
+pytest command.
+
+Typical system test session
+---------------------------
+
+Here is the typical session that you need to do to run system tests:
+
+1. Prepare backport packages
+
+.. code-block:: bash
+
+ ./scripts/ci/ci_prepare_bacport_packages.sh google postgres mysql
+
+2. Enter breeze with installing Airflow 1.10.*, forwarding credentials and installing
+ backported packages (you need appropriate line in ``./files/airflow-breeze-config/variables.env``)
+
+.. code-block:: bash
+
+ ./breeze --install-airflow-version 1.10.9 --python 3.6 --db-reset --forward-credentials restart
+
+This will:
+
+* install Airflow 1.10.9
+* restarts the whole environment (i.e. recreates metadata database from the scratch)
+* run Breeze with python 3.6 version
+* reset the Airflow database
+* forward your local credentials to Breeze
+
+3. Run the tests:
+
+.. code-block:: bash
+
+ pytest -o faulthandler_timeout=2400 --systems=google tests/providers/google/cloud/operators/test_compute_system.py
+
+
+Iteration with System Tests if your resources are slow to create
+----------------------------------------------------------------
+
+When you want to iterate on system tests, you might want to create slow resources first
+
+If you need to setup some external resources for your tests (for example compute instances in Google Cloud)
+you should set them up and teardown in the setUp/tearDown methods of your tests.
+Since those resources might be slow to create you might want to add some helpers that
+set them up and tear them down separately via manual operations so that you can iterate on
+the tests without waiting for setUp and tearDown with every test.
+
+In this case you should build in a mechanism to skip setUp and tearDown in case you manually
+created the resources. Somewhat complex example of that can be found in
+``tests.providers.google.cloud.operators.test_cloud_sql_system.py`` and the helper is
+available in ``tests.providers.google.cloud.operators.test_cloud_sql_system_helper.py``.
+
+When the helper is run with ``--action create`` to create cloud sql instances which are very slow
+to create and set-up so that you can iterate on running the system tests without
+loosing the time for creating theme every time. A temporary file is created to prevent from
+setting up and tearing down the instances when running the test.
+
+This example also shows how you can use the random number generated at the entry of Breeze if you
+have it in your variables.env (see previous chapter). In case of Cloud SQL you cannot reuse the
+same instance name for a week so we generate random number that is used across the whole session
+and store it in ``/random.txt`` file so that the names are unique during tests.
+
+NOTE! Do not forget to delete manually created resources before leaving the Breeze session. They
+are usually expensive to run.
+
+Note that in case you have to update your backported operators or system tests (they are part of
+the backport packageS) you need to rebuild the packages outside of breeze and
+``pip remove/pip install`` those packages in order to get them installed. This is not needed
+if you run system tests with ``current`` airflow version, so it is better to iterate with the
+system tests with the ``current`` version and fix all problems there and only afterwards run
+the tests with Airflow 1.10.*
+
+Typical session then looks as follows:
+
+1. Prepare backport packages
+
+.. code-block:: bash
+
+ ./scripts/ci/ci_prepare_bacport_packages.sh google postgres mysql
+
+2. Enter breeze with installing Airflow 1.10.*, forwarding credentials and installing
+ backported packages (you need appropriate line in ``./files/airflow-breeze-config/variables.env``)
+
+.. code-block:: bash
+
+ ./breeze --install-airflow-version 1.10.9 --python 3.6 --db-reset --forward-credentials restart
+
+3. Run create action in helper (to create slowly created resources):
+
+.. code-block:: bash
+
+ python tests/providers/google/cloud/operators/test_cloud_sql_system_helper.py --action create
+
+4. Run the tests:
+
+.. code-block:: bash
+
+ pytest -o faulthandler_timeout=2400 --systems=google tests/providers/google/cloud/operators/test_compute_system.py
+
+5. In case you are running backport packages tests you need to rebuild and reinstall a package
+ every time you change the operators/hooks or example_dags. The example below shows reinstallation
+ of the google package:
+
+In the host:
+
+.. code-block:: bash
+
+ ./scripts/ci/ci_prepare_backport_packages.sh google
+
+In the container (:
Review comment:
Typo? `(:`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk merged pull request #7774: [AIRFLOW-XXXX] document
system tests mechanism better
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #7774: [AIRFLOW-XXXX] document system tests mechanism better
URL: https://github.com/apache/airflow/pull/7774
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7774: [AIRFLOW-XXXX] document system tests mechanism better
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7774: [AIRFLOW-XXXX] document system tests mechanism better
URL: https://github.com/apache/airflow/pull/7774#discussion_r395517338
##########
File path: TESTING.rst
##########
@@ -543,31 +587,197 @@ tests whenever an operator/hook/sensor is added/modified in a given system.
the system class will take care about running the DAG. Note that the DAG_FOLDER should be
a subdirectory of the ``tests.test_utils.AIRFLOW_MAIN_FOLDER`` + ``providers/<SYSTEM_NAME>/example_dags``.
-An example of a system test is available in:
-``airflow.tests.providers.google.operators.test_natunal_language_system.CloudNaturalLanguageExampleDagsTest``.
+A simple example of a system test is available in:
+
+``tests/providers/google/cloud/operators/test_compute_system.py``.
+
+It runs two DAGs defined in ``airflow.providers.google.cloud.example_dags.example_compute.py`` and
+``airflow.providers.google.cloud.example_dags.example_compute_igm.py``.
+
+Preparing backport packages for System Tests for Airflow 1.10.* series
+----------------------------------------------------------------------
+
+In order to run system tests with old Airflow version you need to prepare backport packages. This
+can be done by running ``./scripts/ci/ci_prepare_bacport_packages.sh <PACKAGES TO BUILD>``. For
+example the below command will build google postgres and mysql packages:
+
+.. code-block:: bash
+
+ ./scripts/ci/ci_prepare_bacport_packages.sh google postgres mysql
+
+Those packages will be prepared in ./dist folder. This folder is mapped to /dist folder
+when you enter Breeze, so it is easy to automate installing of those packages for testing.
-It runs the DAG defined in ``airflow.providers.google.cloud.example_dags.example_natural_language.py``.
-Running Tests for Older Airflow Versions
-----------------------------------------
+Installing backported for Airflow 1.10.* series
+-----------------------------------------------
The tests can be executed against the master version of Airflow but they also work
with older versions. This is especially useful to test back-ported operators
from Airflow 2.0 to 1.10.* versions.
To run the tests for Airflow 1.10.* series, you need to run Breeze with
-``--install-airflow-version==<VERSION>`` to install a different version of Airflow.
+``--install-airflow-version=<VERSION>`` to install a different version of Airflow.
If ``current`` is specified (default), then the current version of Airflow is used.
Otherwise, the released version of Airflow is installed.
-The commands make sure that the source version of master Airflow is removed and the released version of
-Airflow from ``Pypi`` is installed. Note that tests sources are not removed and they can be used
-to run tests (unit tests and system tests) against the freshly installed version.
+The ``-install-airflow-version=<VERSION>`` command make sure that the current (from sources) version of
+Airflow is removed and the released version of Airflow from ``Pypi`` is installed. Note that tests sources
+are not removed and they can be used to run tests (unit tests and system tests) against the
+freshly installed version.
+
+You should automate installing of the backport packages in your own
+``./files/airflow-breeze-config/variables.env`` file. You should make it depend on
+``RUN_AIRFLOW_1_10`` variable value equals to "true" so that
+installation of backport packages is only performed when you install airflow 1.10.*.
+The backport packages are available in ``/dist`` directory if they were prepared as described
+in the previous chapter.
+
+Typically the command in you variables.env file will be similar to:
+
+.. code-block:: bash
+
+ # install any packages from dist folder if they are available
+ if [[ ${RUN_AIRFLOW_1_10:=} == "true" ]]; then
+ pip install /dist/apache_airflow_providers_{google,postgres,mysql}*.whl || true
+ fi
+
+The command above will automatically install backported google, postgres amd mysql packages if they
+were prepared before entering breeze.
+
+
+Running system tests for backported packages in Airflow 1.10.* series
+---------------------------------------------------------------------
+
+Once you installed 1.10.* Airflow version with ``--install-airflow-version`` and prepared and
+installed the required packages via ``variables.env`` it should be as easy as running
+``pytest --systems=<SYSTEM_NAME> TEST_NAME``. Note that we have default timeout for runnning
+system tests set to 8 minutes and some system tests might take much longer to run and you might
+want to add ``-o faulthandler_timeout=2400`` (2400s = 40 minutes for example) to your
+pytest command.
+
+Typical system test session
+---------------------------
+
+Here is the typical session that you need to do to run system tests:
+
+1. Prepare backport packages
+
+.. code-block:: bash
+
+ ./scripts/ci/ci_prepare_bacport_packages.sh google postgres mysql
+
+2. Enter breeze with installing Airflow 1.10.*, forwarding credentials and installing
+ backported packages (you need appropriate line in ``./files/airflow-breeze-config/variables.env``)
+
+.. code-block:: bash
+
+ ./breeze --install-airflow-version 1.10.9 --python 3.6 --db-reset --forward-credentials restart
+
+This will:
+
+* install Airflow 1.10.9
+* restarts the whole environment (i.e. recreates metadata database from the scratch)
+* run Breeze with python 3.6 version
+* reset the Airflow database
+* forward your local credentials to Breeze
+
+3. Run the tests:
+
+.. code-block:: bash
+
+ pytest -o faulthandler_timeout=2400 --systems=google tests/providers/google/cloud/operators/test_compute_system.py
+
+
+Iteration with System Tests if your resources are slow to create
+----------------------------------------------------------------
+
+When you want to iterate on system tests, you might want to create slow resources first
+
+If you need to setup some external resources for your tests (for example compute instances in Google Cloud)
+you should set them up and teardown in the setUp/tearDown methods of your tests.
+Since those resources might be slow to create you might want to add some helpers that
Review comment:
Too long sentence, imo.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7774:
[AIRFLOW-XXXX] document system tests mechanism better
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7774: [AIRFLOW-XXXX] document system tests mechanism better
URL: https://github.com/apache/airflow/pull/7774#discussion_r395534761
##########
File path: TESTING.rst
##########
@@ -543,31 +587,197 @@ tests whenever an operator/hook/sensor is added/modified in a given system.
the system class will take care about running the DAG. Note that the DAG_FOLDER should be
a subdirectory of the ``tests.test_utils.AIRFLOW_MAIN_FOLDER`` + ``providers/<SYSTEM_NAME>/example_dags``.
-An example of a system test is available in:
-``airflow.tests.providers.google.operators.test_natunal_language_system.CloudNaturalLanguageExampleDagsTest``.
+A simple example of a system test is available in:
+
+``tests/providers/google/cloud/operators/test_compute_system.py``.
+
+It runs two DAGs defined in ``airflow.providers.google.cloud.example_dags.example_compute.py`` and
+``airflow.providers.google.cloud.example_dags.example_compute_igm.py``.
+
+Preparing backport packages for System Tests for Airflow 1.10.* series
+----------------------------------------------------------------------
+
+In order to run system tests with old Airflow version you need to prepare backport packages. This
+can be done by running ``./scripts/ci/ci_prepare_bacport_packages.sh <PACKAGES TO BUILD>``. For
+example the below command will build google postgres and mysql packages:
+
+.. code-block:: bash
+
+ ./scripts/ci/ci_prepare_bacport_packages.sh google postgres mysql
+
+Those packages will be prepared in ./dist folder. This folder is mapped to /dist folder
+when you enter Breeze, so it is easy to automate installing of those packages for testing.
-It runs the DAG defined in ``airflow.providers.google.cloud.example_dags.example_natural_language.py``.
-Running Tests for Older Airflow Versions
-----------------------------------------
+Installing backported for Airflow 1.10.* series
+-----------------------------------------------
The tests can be executed against the master version of Airflow but they also work
with older versions. This is especially useful to test back-ported operators
from Airflow 2.0 to 1.10.* versions.
To run the tests for Airflow 1.10.* series, you need to run Breeze with
-``--install-airflow-version==<VERSION>`` to install a different version of Airflow.
+``--install-airflow-version=<VERSION>`` to install a different version of Airflow.
If ``current`` is specified (default), then the current version of Airflow is used.
Otherwise, the released version of Airflow is installed.
-The commands make sure that the source version of master Airflow is removed and the released version of
-Airflow from ``Pypi`` is installed. Note that tests sources are not removed and they can be used
-to run tests (unit tests and system tests) against the freshly installed version.
+The ``-install-airflow-version=<VERSION>`` command make sure that the current (from sources) version of
+Airflow is removed and the released version of Airflow from ``Pypi`` is installed. Note that tests sources
+are not removed and they can be used to run tests (unit tests and system tests) against the
+freshly installed version.
+
+You should automate installing of the backport packages in your own
+``./files/airflow-breeze-config/variables.env`` file. You should make it depend on
+``RUN_AIRFLOW_1_10`` variable value equals to "true" so that
+installation of backport packages is only performed when you install airflow 1.10.*.
+The backport packages are available in ``/dist`` directory if they were prepared as described
+in the previous chapter.
+
+Typically the command in you variables.env file will be similar to:
+
+.. code-block:: bash
+
+ # install any packages from dist folder if they are available
+ if [[ ${RUN_AIRFLOW_1_10:=} == "true" ]]; then
+ pip install /dist/apache_airflow_providers_{google,postgres,mysql}*.whl || true
+ fi
+
+The command above will automatically install backported google, postgres amd mysql packages if they
+were prepared before entering breeze.
+
+
+Running system tests for backported packages in Airflow 1.10.* series
+---------------------------------------------------------------------
+
+Once you installed 1.10.* Airflow version with ``--install-airflow-version`` and prepared and
+installed the required packages via ``variables.env`` it should be as easy as running
+``pytest --systems=<SYSTEM_NAME> TEST_NAME``. Note that we have default timeout for runnning
+system tests set to 8 minutes and some system tests might take much longer to run and you might
+want to add ``-o faulthandler_timeout=2400`` (2400s = 40 minutes for example) to your
+pytest command.
+
+Typical system test session
+---------------------------
+
+Here is the typical session that you need to do to run system tests:
+
+1. Prepare backport packages
+
+.. code-block:: bash
+
+ ./scripts/ci/ci_prepare_bacport_packages.sh google postgres mysql
+
+2. Enter breeze with installing Airflow 1.10.*, forwarding credentials and installing
+ backported packages (you need appropriate line in ``./files/airflow-breeze-config/variables.env``)
+
+.. code-block:: bash
+
+ ./breeze --install-airflow-version 1.10.9 --python 3.6 --db-reset --forward-credentials restart
+
+This will:
+
+* install Airflow 1.10.9
+* restarts the whole environment (i.e. recreates metadata database from the scratch)
+* run Breeze with python 3.6 version
+* reset the Airflow database
+* forward your local credentials to Breeze
+
+3. Run the tests:
+
+.. code-block:: bash
+
+ pytest -o faulthandler_timeout=2400 --systems=google tests/providers/google/cloud/operators/test_compute_system.py
+
+
+Iteration with System Tests if your resources are slow to create
+----------------------------------------------------------------
+
+When you want to iterate on system tests, you might want to create slow resources first
+
+If you need to setup some external resources for your tests (for example compute instances in Google Cloud)
+you should set them up and teardown in the setUp/tearDown methods of your tests.
+Since those resources might be slow to create you might want to add some helpers that
+set them up and tear them down separately via manual operations so that you can iterate on
+the tests without waiting for setUp and tearDown with every test.
+
+In this case you should build in a mechanism to skip setUp and tearDown in case you manually
+created the resources. Somewhat complex example of that can be found in
+``tests.providers.google.cloud.operators.test_cloud_sql_system.py`` and the helper is
+available in ``tests.providers.google.cloud.operators.test_cloud_sql_system_helper.py``.
+
+When the helper is run with ``--action create`` to create cloud sql instances which are very slow
+to create and set-up so that you can iterate on running the system tests without
+loosing the time for creating theme every time. A temporary file is created to prevent from
+setting up and tearing down the instances when running the test.
+
+This example also shows how you can use the random number generated at the entry of Breeze if you
+have it in your variables.env (see previous chapter). In case of Cloud SQL you cannot reuse the
+same instance name for a week so we generate random number that is used across the whole session
+and store it in ``/random.txt`` file so that the names are unique during tests.
+
+NOTE! Do not forget to delete manually created resources before leaving the Breeze session. They
Review comment:
Highlighted!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7774: [AIRFLOW-XXXX] document system tests mechanism better
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7774: [AIRFLOW-XXXX] document system tests mechanism better
URL: https://github.com/apache/airflow/pull/7774#discussion_r395523833
##########
File path: TESTING.rst
##########
@@ -543,31 +587,197 @@ tests whenever an operator/hook/sensor is added/modified in a given system.
the system class will take care about running the DAG. Note that the DAG_FOLDER should be
a subdirectory of the ``tests.test_utils.AIRFLOW_MAIN_FOLDER`` + ``providers/<SYSTEM_NAME>/example_dags``.
-An example of a system test is available in:
-``airflow.tests.providers.google.operators.test_natunal_language_system.CloudNaturalLanguageExampleDagsTest``.
+A simple example of a system test is available in:
+
+``tests/providers/google/cloud/operators/test_compute_system.py``.
+
+It runs two DAGs defined in ``airflow.providers.google.cloud.example_dags.example_compute.py`` and
+``airflow.providers.google.cloud.example_dags.example_compute_igm.py``.
+
+Preparing backport packages for System Tests for Airflow 1.10.* series
+----------------------------------------------------------------------
+
+In order to run system tests with old Airflow version you need to prepare backport packages. This
+can be done by running ``./scripts/ci/ci_prepare_bacport_packages.sh <PACKAGES TO BUILD>``. For
+example the below command will build google postgres and mysql packages:
+
+.. code-block:: bash
+
+ ./scripts/ci/ci_prepare_bacport_packages.sh google postgres mysql
+
+Those packages will be prepared in ./dist folder. This folder is mapped to /dist folder
+when you enter Breeze, so it is easy to automate installing of those packages for testing.
-It runs the DAG defined in ``airflow.providers.google.cloud.example_dags.example_natural_language.py``.
-Running Tests for Older Airflow Versions
-----------------------------------------
+Installing backported for Airflow 1.10.* series
+-----------------------------------------------
The tests can be executed against the master version of Airflow but they also work
with older versions. This is especially useful to test back-ported operators
from Airflow 2.0 to 1.10.* versions.
To run the tests for Airflow 1.10.* series, you need to run Breeze with
-``--install-airflow-version==<VERSION>`` to install a different version of Airflow.
+``--install-airflow-version=<VERSION>`` to install a different version of Airflow.
If ``current`` is specified (default), then the current version of Airflow is used.
Otherwise, the released version of Airflow is installed.
-The commands make sure that the source version of master Airflow is removed and the released version of
-Airflow from ``Pypi`` is installed. Note that tests sources are not removed and they can be used
-to run tests (unit tests and system tests) against the freshly installed version.
+The ``-install-airflow-version=<VERSION>`` command make sure that the current (from sources) version of
+Airflow is removed and the released version of Airflow from ``Pypi`` is installed. Note that tests sources
+are not removed and they can be used to run tests (unit tests and system tests) against the
+freshly installed version.
+
+You should automate installing of the backport packages in your own
+``./files/airflow-breeze-config/variables.env`` file. You should make it depend on
+``RUN_AIRFLOW_1_10`` variable value equals to "true" so that
+installation of backport packages is only performed when you install airflow 1.10.*.
+The backport packages are available in ``/dist`` directory if they were prepared as described
+in the previous chapter.
+
+Typically the command in you variables.env file will be similar to:
+
+.. code-block:: bash
+
+ # install any packages from dist folder if they are available
+ if [[ ${RUN_AIRFLOW_1_10:=} == "true" ]]; then
+ pip install /dist/apache_airflow_providers_{google,postgres,mysql}*.whl || true
+ fi
+
+The command above will automatically install backported google, postgres amd mysql packages if they
+were prepared before entering breeze.
+
+
+Running system tests for backported packages in Airflow 1.10.* series
+---------------------------------------------------------------------
+
+Once you installed 1.10.* Airflow version with ``--install-airflow-version`` and prepared and
+installed the required packages via ``variables.env`` it should be as easy as running
+``pytest --systems=<SYSTEM_NAME> TEST_NAME``. Note that we have default timeout for runnning
+system tests set to 8 minutes and some system tests might take much longer to run and you might
+want to add ``-o faulthandler_timeout=2400`` (2400s = 40 minutes for example) to your
+pytest command.
+
+Typical system test session
+---------------------------
+
+Here is the typical session that you need to do to run system tests:
+
+1. Prepare backport packages
+
+.. code-block:: bash
+
+ ./scripts/ci/ci_prepare_bacport_packages.sh google postgres mysql
+
+2. Enter breeze with installing Airflow 1.10.*, forwarding credentials and installing
+ backported packages (you need appropriate line in ``./files/airflow-breeze-config/variables.env``)
+
+.. code-block:: bash
+
+ ./breeze --install-airflow-version 1.10.9 --python 3.6 --db-reset --forward-credentials restart
+
+This will:
+
+* install Airflow 1.10.9
+* restarts the whole environment (i.e. recreates metadata database from the scratch)
+* run Breeze with python 3.6 version
+* reset the Airflow database
+* forward your local credentials to Breeze
+
+3. Run the tests:
+
+.. code-block:: bash
+
+ pytest -o faulthandler_timeout=2400 --systems=google tests/providers/google/cloud/operators/test_compute_system.py
+
+
+Iteration with System Tests if your resources are slow to create
+----------------------------------------------------------------
+
+When you want to iterate on system tests, you might want to create slow resources first
Review comment:
When you want to iterate on system tests, you might want to create slow resources first.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7774: [AIRFLOW-XXXX] document system tests mechanism better
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7774: [AIRFLOW-XXXX] document system tests mechanism better
URL: https://github.com/apache/airflow/pull/7774#discussion_r395523833
##########
File path: TESTING.rst
##########
@@ -543,31 +587,197 @@ tests whenever an operator/hook/sensor is added/modified in a given system.
the system class will take care about running the DAG. Note that the DAG_FOLDER should be
a subdirectory of the ``tests.test_utils.AIRFLOW_MAIN_FOLDER`` + ``providers/<SYSTEM_NAME>/example_dags``.
-An example of a system test is available in:
-``airflow.tests.providers.google.operators.test_natunal_language_system.CloudNaturalLanguageExampleDagsTest``.
+A simple example of a system test is available in:
+
+``tests/providers/google/cloud/operators/test_compute_system.py``.
+
+It runs two DAGs defined in ``airflow.providers.google.cloud.example_dags.example_compute.py`` and
+``airflow.providers.google.cloud.example_dags.example_compute_igm.py``.
+
+Preparing backport packages for System Tests for Airflow 1.10.* series
+----------------------------------------------------------------------
+
+In order to run system tests with old Airflow version you need to prepare backport packages. This
+can be done by running ``./scripts/ci/ci_prepare_bacport_packages.sh <PACKAGES TO BUILD>``. For
+example the below command will build google postgres and mysql packages:
+
+.. code-block:: bash
+
+ ./scripts/ci/ci_prepare_bacport_packages.sh google postgres mysql
+
+Those packages will be prepared in ./dist folder. This folder is mapped to /dist folder
+when you enter Breeze, so it is easy to automate installing of those packages for testing.
-It runs the DAG defined in ``airflow.providers.google.cloud.example_dags.example_natural_language.py``.
-Running Tests for Older Airflow Versions
-----------------------------------------
+Installing backported for Airflow 1.10.* series
+-----------------------------------------------
The tests can be executed against the master version of Airflow but they also work
with older versions. This is especially useful to test back-ported operators
from Airflow 2.0 to 1.10.* versions.
To run the tests for Airflow 1.10.* series, you need to run Breeze with
-``--install-airflow-version==<VERSION>`` to install a different version of Airflow.
+``--install-airflow-version=<VERSION>`` to install a different version of Airflow.
If ``current`` is specified (default), then the current version of Airflow is used.
Otherwise, the released version of Airflow is installed.
-The commands make sure that the source version of master Airflow is removed and the released version of
-Airflow from ``Pypi`` is installed. Note that tests sources are not removed and they can be used
-to run tests (unit tests and system tests) against the freshly installed version.
+The ``-install-airflow-version=<VERSION>`` command make sure that the current (from sources) version of
+Airflow is removed and the released version of Airflow from ``Pypi`` is installed. Note that tests sources
+are not removed and they can be used to run tests (unit tests and system tests) against the
+freshly installed version.
+
+You should automate installing of the backport packages in your own
+``./files/airflow-breeze-config/variables.env`` file. You should make it depend on
+``RUN_AIRFLOW_1_10`` variable value equals to "true" so that
+installation of backport packages is only performed when you install airflow 1.10.*.
+The backport packages are available in ``/dist`` directory if they were prepared as described
+in the previous chapter.
+
+Typically the command in you variables.env file will be similar to:
+
+.. code-block:: bash
+
+ # install any packages from dist folder if they are available
+ if [[ ${RUN_AIRFLOW_1_10:=} == "true" ]]; then
+ pip install /dist/apache_airflow_providers_{google,postgres,mysql}*.whl || true
+ fi
+
+The command above will automatically install backported google, postgres amd mysql packages if they
+were prepared before entering breeze.
+
+
+Running system tests for backported packages in Airflow 1.10.* series
+---------------------------------------------------------------------
+
+Once you installed 1.10.* Airflow version with ``--install-airflow-version`` and prepared and
+installed the required packages via ``variables.env`` it should be as easy as running
+``pytest --systems=<SYSTEM_NAME> TEST_NAME``. Note that we have default timeout for runnning
+system tests set to 8 minutes and some system tests might take much longer to run and you might
+want to add ``-o faulthandler_timeout=2400`` (2400s = 40 minutes for example) to your
+pytest command.
+
+Typical system test session
+---------------------------
+
+Here is the typical session that you need to do to run system tests:
+
+1. Prepare backport packages
+
+.. code-block:: bash
+
+ ./scripts/ci/ci_prepare_bacport_packages.sh google postgres mysql
+
+2. Enter breeze with installing Airflow 1.10.*, forwarding credentials and installing
+ backported packages (you need appropriate line in ``./files/airflow-breeze-config/variables.env``)
+
+.. code-block:: bash
+
+ ./breeze --install-airflow-version 1.10.9 --python 3.6 --db-reset --forward-credentials restart
+
+This will:
+
+* install Airflow 1.10.9
+* restarts the whole environment (i.e. recreates metadata database from the scratch)
+* run Breeze with python 3.6 version
+* reset the Airflow database
+* forward your local credentials to Breeze
+
+3. Run the tests:
+
+.. code-block:: bash
+
+ pytest -o faulthandler_timeout=2400 --systems=google tests/providers/google/cloud/operators/test_compute_system.py
+
+
+Iteration with System Tests if your resources are slow to create
+----------------------------------------------------------------
+
+When you want to iterate on system tests, you might want to create slow resources first
Review comment:
When you want to iterate on system tests, you might want to create slow resources first.
Missing dot at the end.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services