You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/02/09 20:39:13 UTC

[GitHub] [airflow] potiuk opened a new pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

potiuk opened a new pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389
 
 
   We will run system test on back-ported operators for 1.10* series of airflow
   and for that we need to have support for running system tests using pytest's
   markers and reading environment variables passed from HOST machine (to pass
   credentials). 
   
   This is the first step to automate system tests execution.
   
   ---
   Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   <sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r379122655
 
 

 ##########
 File path: TESTING.rst
 ##########
 @@ -31,8 +31,7 @@ Airflow Test Infrastructure
 
 * **System tests** are automatic tests that use external systems like
   Google Cloud Platform. These tests are intended for an end-to-end DAG execution.
-  Note that automated execution of these tests is still
-  `work in progress <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems#app-switcher>`_.
+  The tests can be executed on both.
 
 Review comment:
   Resolved.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-585195708
 
 
   Hello @ashb @mik-laj @nuclearpinguin @kaxil @dimberman  -> I updated the implementation now and it works really nicely. 
   
   Breeze is now turned into "airflow_1.10.*" testing playground. It will be super-useful to automatically run system tests (currently only for google/* operators) using the latests tests and "example_dags" from "providers", but using the installed airflow version.
   
   You can run `./breeze --install-airflow-version <VERSION>` . If you run it with "current" version it works as previously (current version from sources). If you run it with '1.10.2' -> '1.10.9' it will remove the current sources, will uninstall installed airflow master and will install the released version from PIP.  The automated installation works out of the box for all versions: 1.10.2 -> 1.10.9.
   
   There is an easy way to test installation and run system tests of the released "backported" providers package. 
   
   If you have binary wheel "providers" package prepared locally or downloaded, you just need to copy the wheel package to "files" dir.
   
   Then inside the container with installed airflow version you should be able to run: `pip install /files/<NAME_OF_THE_FILE>.whl` to install the "providers" package. And then the example tests should run out of the box for most operators.
   
   The "/opt/airflow/tests" folders and "/providers" folders are mapped into the container (and system test class handles it properly) so that we can still modify the tests locally and run them with airflow 1.10.*.
   
   For example, right after installing the 'providers' package you should be able to run:
   
   `pytest tests/providers/google/cloud/operators/test_functions_system.py --systems google.cloud` and the system tests 
   
   It will work for both - 2.0 and 1.10.* series providing that you have proper GCP credentials in "files/airflow-breeze-config/variables.env"
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r376829019
 
 

 ##########
 File path: TESTING.rst
 ##########
 @@ -474,20 +473,58 @@ More information:
 Airflow System Tests
 ====================
 
-The System tests for Airflow are not yet fully implemented. They are Work In Progress of the
-`AIP-4 Support for System Tests for external systems <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems>`__.
 These tests need to communicate with external services/systems that are available
 if you have appropriate credentials configured for your tests.
-The tests derive from ``tests.system_test_class.SystemTests`` class.
+The tests derive from ``tests.system_test_class.SystemTests`` class. They should also
+be marked with ``@pytest.marker.system(SYSTEM)`` where system designates the system
+to be tested (for example ``google.cloud``). Those tests are skipped by default.
+You can execute the tests providing ``--systems SYSTEMS`` flag to pytest.
 
-The system tests execute a specified
-example DAG file that runs the DAG end-to-end.
+The system tests execute a specified example DAG file that runs the DAG end-to-end.
+
+Most of the system tests have also addtional requirements for example they need to have
 
 Review comment:
   How this is dangerous for your local environment where you control both host and container? 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r378811678
 
 

 ##########
 File path: TESTING.rst
 ##########
 @@ -474,21 +473,54 @@ More information:
 Airflow System Tests
 ====================
 
-The System tests for Airflow are not yet fully implemented. They are Work In Progress of the
-`AIP-4 Support for System Tests for external systems <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems>`__.
 These tests need to communicate with external services/systems that are available
 if you have appropriate credentials configured for your tests.
-The tests derive from ``tests.system_test_class.SystemTests`` class.
+The tests derive from ``tests.system_test_class.SystemTests`` class. They should also
+be marked with ``@pytest.marker.system(SYSTEM)`` where system designates the system
+to be tested (for example ``google.cloud``). Those tests are skipped by default.
+You can execute the tests providing ``--systems SYSTEMS`` flag to pytest.
 
-The system tests execute a specified
-example DAG file that runs the DAG end-to-end.
+The system tests execute a specified example DAG file that runs the DAG end-to-end.
+
+Most of the system tests have also addtional requirements for example they need to have
+credentials to connect to external systems. This is don with pytest marker
+``@pytest.markers.credential_file(file)`` where "file" is a file containing
+credentials. The credential file should be relative path from
+``/files/airflow-breeze-config/keys`` directory. In Breeze environment it is mapped
+from the ``files`` directory in airflow's sources.
+
+Some of the system tests are long lasting ones (they require more than 20-30 minutes
+to complete}. They are marked with ```@pytest.markers.long_lasting`` marker.
+They are skipped by default unless you specify ``--long-lasting`` flag to pytest.
 
 An example of such a system test is
 ``airflow.tests.providers.google.operators.test_natural_language_system.CloudNaturalLanguageExampleDagsTest``.
 
-For now you can execute the system tests and follow messages printed to get them running. Soon more information on
-running the tests will be available.
 
+Running system tests
+--------------------
+
+As mentioned above - the system tests are only executed when you specify ``--systems SYSTEMS``
+flag where SYSTEMS is coma-separated list of systems to run the system tests for.
+
+
+Running tests for older Airflow versionsIRFLOW
 
 Review comment:
   Yep.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

ashb commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-586189517
 
 
   Does this fall under AIP-4? That is still draft. Should we vote on that AIP?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r376815947
 
 

 ##########
 File path: TESTING.rst
 ##########
 @@ -474,20 +473,58 @@ More information:
 Airflow System Tests
 ====================
 
-The System tests for Airflow are not yet fully implemented. They are Work In Progress of the
-`AIP-4 Support for System Tests for external systems <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems>`__.
 These tests need to communicate with external services/systems that are available
 if you have appropriate credentials configured for your tests.
-The tests derive from ``tests.system_test_class.SystemTests`` class.
+The tests derive from ``tests.system_test_class.SystemTests`` class. They should also
+be marked with ``@pytest.marker.system(SYSTEM)`` where system designates the system
+to be tested (for example ``google.cloud``). Those tests are skipped by default.
+You can execute the tests providing ``--systems SYSTEMS`` flag to pytest.
 
-The system tests execute a specified
-example DAG file that runs the DAG end-to-end.
+The system tests execute a specified example DAG file that runs the DAG end-to-end.
+
+Most of the system tests have also addtional requirements for example they need to have
+credentials to connect to external systems. This is don with pytest marker
+``@pytest.markers.credential_file(file)`` where "file" is a file containing
+credentials. The credential file should be relative path from
+``/files/airflow-breeze-config/keys`` directory. In Breeze environment it is mapped
+from the ``files`` directory in airflow's sources.
+
+Some of the system tests are long lasting ones (they require more than 20-30 minutes
+to complete}. They are marked with ```@pytest.markers.long_lasting`` marker.
+They are skipped by default unless you specify ``--long-lasting`` flag to pytest.
 
 An example of such a system test is
 ``airflow.tests.providers.google.operators.test_natural_language_system.CloudNaturalLanguageExampleDagsTest``.
 
-For now you can execute the system tests and follow messages printed to get them running. Soon more information on
-running the tests will be available.
+
+Running system tests for a system
+---------------------------------
+
+As mentioned above - the system tests are only executed when you specify ``--systems SYSTEMS
 
 Review comment:
   Missing "``" at the end of the sentence.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

nuclearpinguin commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-587346274
 
 
   It's a big change but it looks good to me from what I see :)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-585978655
 
 
   Hey @ashb @mik-laj @nuclearpinguin  -> all green for the system tests ready for backporting. BTW, I upladed the  docs and added "How to write system tests" doc...

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r376815875
 
 

 ##########
 File path: TESTING.rst
 ##########
 @@ -31,8 +31,7 @@ Airflow Test Infrastructure
 
 * **System tests** are automatic tests that use external systems like
   Google Cloud Platform. These tests are intended for an end-to-end DAG execution.
-  Note that automated execution of these tests is still
-  `work in progress <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems#app-switcher>`_.
+  The tests can be executde on both.
 
 Review comment:
   "executde" Small typo

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

codecov-io edited a comment on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-583898179
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=h1) Report
   > Merging [#7389](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/9cbd7de6d115795aba8bfb8addb060bfdfbdf87b?src=pr&el=desc) will **increase** coverage by `0.43%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7389/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7389      +/-   ##
   ==========================================
   + Coverage   86.26%   86.69%   +0.43%     
   ==========================================
     Files         877      877              
     Lines       41221    41242      +21     
   ==========================================
   + Hits        35559    35755     +196     
   + Misses       5662     5487     -175
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...viders/docker/example\_dags/example\_docker\_swarm.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZG9ja2VyL2V4YW1wbGVfZGFncy9leGFtcGxlX2RvY2tlcl9zd2FybS5weQ==) | `100% <0%> (ø)` | :arrow_up: |
   | [...ow/providers/docker/example\_dags/example\_docker.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZG9ja2VyL2V4YW1wbGVfZGFncy9leGFtcGxlX2RvY2tlci5weQ==) | `100% <0%> (ø)` | :arrow_up: |
   | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `88.12% <0%> (+0.19%)` | :arrow_up: |
   | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `91.16% <0%> (+0.25%)` | :arrow_up: |
   | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.66% <0%> (+0.43%)` | :arrow_up: |
   | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `92.09% <0%> (+1.12%)` | :arrow_up: |
   | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `91.73% <0%> (+1.65%)` | :arrow_up: |
   | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `91.43% <0%> (+3.42%)` | :arrow_up: |
   | [airflow/providers/postgres/hooks/postgres.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvaG9va3MvcG9zdGdyZXMucHk=) | `94.36% <0%> (+16.9%)` | :arrow_up: |
   | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `74.5% <0%> (+23.52%)` | :arrow_up: |
   | ... and [6 more](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=footer). Last update [9cbd7de...9751b69](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-589362536
 
 
   rebased and hopefully all good now.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r376831691
 
 

 ##########
 File path: scripts/ci/docker-compose/local.yml
 ##########
 @@ -53,7 +53,9 @@ services:
       - ../../../setup.cfg:/opt/airflow/setup.cfg:cached
       - ../../../setup.py:/opt/airflow/setup.py:cached
       - ../../../tests:/opt/airflow/tests:cached
+      - ../../../tests:/opt/airflow-backport/tests:cached
       - ../../../tmp:/opt/airflow/tmp:cached
+      - ${HOME}/.config:/root/.config
 
 Review comment:
   I am not sure I am following this line of thinking.  Not sure what "hurt themselves" mean in this case  - but happy to discuss it tomorrow.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r376828903
 
 

 ##########
 File path: scripts/ci/docker-compose/local.yml
 ##########
 @@ -53,7 +53,9 @@ services:
       - ../../../setup.cfg:/opt/airflow/setup.cfg:cached
       - ../../../setup.py:/opt/airflow/setup.py:cached
       - ../../../tests:/opt/airflow/tests:cached
+      - ../../../tests:/opt/airflow-backport/tests:cached
       - ../../../tmp:/opt/airflow/tmp:cached
+      - ${HOME}/.config:/root/.config
 
 Review comment:
   Precisely for that reason. This way for example gcloud authentication is mapped to the container, so you can use your host gcloud authentication to run gcloud commands in docker
   
   This is only for development/local execution so there is no risk in passing it to inside the docker. Both host and docker are controlled by the developer.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r376816351
 
 

 ##########
 File path: TESTING.rst
 ##########
 @@ -474,20 +473,58 @@ More information:
 Airflow System Tests
 ====================
 
-The System tests for Airflow are not yet fully implemented. They are Work In Progress of the
-`AIP-4 Support for System Tests for external systems <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems>`__.
 These tests need to communicate with external services/systems that are available
 if you have appropriate credentials configured for your tests.
-The tests derive from ``tests.system_test_class.SystemTests`` class.
+The tests derive from ``tests.system_test_class.SystemTests`` class. They should also
+be marked with ``@pytest.marker.system(SYSTEM)`` where system designates the system
+to be tested (for example ``google.cloud``). Those tests are skipped by default.
+You can execute the tests providing ``--systems SYSTEMS`` flag to pytest.
 
-The system tests execute a specified
-example DAG file that runs the DAG end-to-end.
+The system tests execute a specified example DAG file that runs the DAG end-to-end.
+
+Most of the system tests have also addtional requirements for example they need to have
+credentials to connect to external systems. This is don with pytest marker
+``@pytest.markers.credential_file(file)`` where "file" is a file containing
+credentials. The credential file should be relative path from
+``/files/airflow-breeze-config/keys`` directory. In Breeze environment it is mapped
+from the ``files`` directory in airflow's sources.
+
+Some of the system tests are long lasting ones (they require more than 20-30 minutes
+to complete}. They are marked with ```@pytest.markers.long_lasting`` marker.
+They are skipped by default unless you specify ``--long-lasting`` flag to pytest.
 
 An example of such a system test is
 ``airflow.tests.providers.google.operators.test_natural_language_system.CloudNaturalLanguageExampleDagsTest``.
 
-For now you can execute the system tests and follow messages printed to get them running. Soon more information on
-running the tests will be available.
+
+Running system tests for a system
+---------------------------------
+
+As mentioned above - the system tests are only executed when you specify ``--systems SYSTEMS
+flag where SYSTEMS is coma-separated list of systems to run the system tests for.
+
+
+Running system tests for older Airlow versions
+----------------------------------------------
+
+The system tests can be executed against the master version of airflow but they also work
+with older versions of airflow. This is especially useful to tests backported operators
+from airflow 2.0 to 1.10.* versions.
+
+In order to run the tests for airflFow 1.10.* series you need to do the following steps after entering
+the Breeze environment:
+
+.. code-block:: bash
+
+  unset PYTHONPATH
+  ./scripts/ci/in_container/install_older_airflow.sh <AIRFLOW_VERSION>
+  cd /opt/airflow-backport/
+
+The commands make sure that older version of Airflow is installed and source version of master airflow
+is removed from the PYTHONPATH. Also the current system tests are mapped to
 
 Review comment:
   ```suggestion
   is removed from the `PYTHONPATH`. Also the current system tests are mapped to
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk edited a comment on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-586260298

Sure. I can start a discussion and vote on it - but I think we could merge that one independently (and show it as an example) - for now it's limited only to GCP but it can show others how it can be done in an easy way.

I believe AIP-4 is is much more about running the System Tests in automated way rather than separating them out (and it requires Github Actions move to be completed and some serious discussion on how to store credentials). We've already added some separation (skipIfs, _system.py in GCP) - now we have a generic mechanism but it does not mean we introduce it in "global way" yet.

I think it's best if we merge this one first and then I could start a discussion on how to make the AIP-4 automation working. Actually i wanted to start another discussion right after about involving the community to add system tests to all other provider packages - so that we can automate it with AIP-4 as well. So the discussion might be quite big - having a working, generic implementation in GCP that is generic enough to be reused and to show how it works manually is a great way for people to take part in the discussion.

WDYT @ashb ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [airflow] potiuk edited a comment on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-585195708
 
 
   Hello @ashb @mik-laj @nuclearpinguin @kaxil @dimberman  -> I updated the implementation now and it works really nicely. 
   
   Breeze is now turned into "airflow_1.10.*" testing playground. It will be super-useful to automatically run system tests (currently only for google/* operators) using the latests tests and "example_dags" from "providers", but using the installed airflow version.
   
   You can run `./breeze --install-airflow-version <VERSION>` . If you run it with "current" version it works as previously (current version from sources). If you run it with '1.10.1' -> '1.10.9' it will remove the current sources, will uninstall installed airflow master and will install the released version from PIP.  The automated installation works out of the box for all versions: 1.10.1 -> 1.10.9.
   
   There is an easy way to test installation and run system tests of the released "backported" providers package. 
   
   If you have binary wheel "providers" package prepared locally or downloaded, you just need to copy the wheel package to "files" dir.
   
   Then inside the container with installed airflow version you should be able to run: `pip install /files/<NAME_OF_THE_FILE>.whl` to install the "providers" package. And then the example tests should run out of the box for most operators.
   
   The "/opt/airflow/tests" folders and "/providers" folders are mapped into the container (and system test class handles it properly) so that we can still modify the tests locally and run them with airflow 1.10.*.
   
   For example, right after installing the 'providers' package you should be able to run:
   
   `pytest tests/providers/google/cloud/operators/test_functions_system.py --systems google.cloud` and the system tests 
   
   It will work for both - 2.0 and 1.10.* series providing that you have proper GCP credentials in "files/airflow-breeze-config/variables.env"
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r376816276
 
 

 ##########
 File path: TESTING.rst
 ##########
 @@ -474,20 +473,58 @@ More information:
 Airflow System Tests
 ====================
 
-The System tests for Airflow are not yet fully implemented. They are Work In Progress of the
-`AIP-4 Support for System Tests for external systems <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems>`__.
 These tests need to communicate with external services/systems that are available
 if you have appropriate credentials configured for your tests.
-The tests derive from ``tests.system_test_class.SystemTests`` class.
+The tests derive from ``tests.system_test_class.SystemTests`` class. They should also
+be marked with ``@pytest.marker.system(SYSTEM)`` where system designates the system
+to be tested (for example ``google.cloud``). Those tests are skipped by default.
+You can execute the tests providing ``--systems SYSTEMS`` flag to pytest.
 
-The system tests execute a specified
-example DAG file that runs the DAG end-to-end.
+The system tests execute a specified example DAG file that runs the DAG end-to-end.
+
+Most of the system tests have also addtional requirements for example they need to have
+credentials to connect to external systems. This is don with pytest marker
+``@pytest.markers.credential_file(file)`` where "file" is a file containing
+credentials. The credential file should be relative path from
+``/files/airflow-breeze-config/keys`` directory. In Breeze environment it is mapped
+from the ``files`` directory in airflow's sources.
+
+Some of the system tests are long lasting ones (they require more than 20-30 minutes
+to complete}. They are marked with ```@pytest.markers.long_lasting`` marker.
+They are skipped by default unless you specify ``--long-lasting`` flag to pytest.
 
 An example of such a system test is
 ``airflow.tests.providers.google.operators.test_natural_language_system.CloudNaturalLanguageExampleDagsTest``.
 
-For now you can execute the system tests and follow messages printed to get them running. Soon more information on
-running the tests will be available.
+
+Running system tests for a system
+---------------------------------
+
+As mentioned above - the system tests are only executed when you specify ``--systems SYSTEMS
+flag where SYSTEMS is coma-separated list of systems to run the system tests for.
+
+
+Running system tests for older Airlow versions
+----------------------------------------------
+
+The system tests can be executed against the master version of airflow but they also work
+with older versions of airflow. This is especially useful to tests backported operators
+from airflow 2.0 to 1.10.* versions.
 
 Review comment:
   ```suggestion
   from Airflow 2.0 to 1.10.* versions.
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r376828926
 
 

 ##########
 File path: TESTING.rst
 ##########
 @@ -474,20 +473,58 @@ More information:
 Airflow System Tests
 ====================
 
-The System tests for Airflow are not yet fully implemented. They are Work In Progress of the
-`AIP-4 Support for System Tests for external systems <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems>`__.
 These tests need to communicate with external services/systems that are available
 if you have appropriate credentials configured for your tests.
-The tests derive from ``tests.system_test_class.SystemTests`` class.
+The tests derive from ``tests.system_test_class.SystemTests`` class. They should also
+be marked with ``@pytest.marker.system(SYSTEM)`` where system designates the system
+to be tested (for example ``google.cloud``). Those tests are skipped by default.
+You can execute the tests providing ``--systems SYSTEMS`` flag to pytest.
 
-The system tests execute a specified
-example DAG file that runs the DAG end-to-end.
+The system tests execute a specified example DAG file that runs the DAG end-to-end.
+
+Most of the system tests have also addtional requirements for example they need to have
 
 Review comment:
   I think it is very dangerous that this key is available in the container, and certainly not acceptable if we want to finish AIP-4. I have an idea that we can discuss tomorrow in the office. This will remove the need for key files in the container and even on the developer's computer. 
   
   However, this is a separate topic that we can implement independently.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r376828926
 
 

 ##########
 File path: TESTING.rst
 ##########
 @@ -474,20 +473,58 @@ More information:
 Airflow System Tests
 ====================
 
-The System tests for Airflow are not yet fully implemented. They are Work In Progress of the
-`AIP-4 Support for System Tests for external systems <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems>`__.
 These tests need to communicate with external services/systems that are available
 if you have appropriate credentials configured for your tests.
-The tests derive from ``tests.system_test_class.SystemTests`` class.
+The tests derive from ``tests.system_test_class.SystemTests`` class. They should also
+be marked with ``@pytest.marker.system(SYSTEM)`` where system designates the system
+to be tested (for example ``google.cloud``). Those tests are skipped by default.
+You can execute the tests providing ``--systems SYSTEMS`` flag to pytest.
 
-The system tests execute a specified
-example DAG file that runs the DAG end-to-end.
+The system tests execute a specified example DAG file that runs the DAG end-to-end.
+
+Most of the system tests have also addtional requirements for example they need to have
 
 Review comment:
   I think it is very dangerous that this key is available in the container, and certainly not acceptable if we want to finish AIP-4. I have an idea that we can discuss tomorrow in the office. This will remove the need for key files in the container and even on the developer's computer.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-586260298

WDYT @ashb ?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

ashb commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r378795088
 
 

 ##########
 File path: BREEZE.rst
 ##########
 @@ -611,7 +617,13 @@ This is the current syntax for  `./breeze <./breeze>`_:
   -S, --static-check <STATIC_CHECK>
           Run selected static checks for currently changed files. You should specify static check that
           you would like to run or 'all' to run all checks. One of
-          [ all all-but-pylint bat-tests check-apache-license check-executables-have-shebangs check-hooks-apply check-merge-conflict check-xml debug-statements doctoc detect-private-key end-of-file-fixer flake8 forbid-tabs insert-license lint-dockerfile mixed-line-ending mypy pylint pylint-test setup-order shellcheck].
+
+  all all-but-pylint bat-tests check-apache-license
 
 Review comment:
   The formatting/indentation here looks off -- it should be `"            "` I think (to put this "under" the option it's for, otherwise it's too high up)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r376829097
 
 

 ##########
 File path: TESTING.rst
 ##########
 @@ -474,20 +473,58 @@ More information:
 Airflow System Tests
 ====================
 
-The System tests for Airflow are not yet fully implemented. They are Work In Progress of the
-`AIP-4 Support for System Tests for external systems <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems>`__.
 These tests need to communicate with external services/systems that are available
 if you have appropriate credentials configured for your tests.
-The tests derive from ``tests.system_test_class.SystemTests`` class.
+The tests derive from ``tests.system_test_class.SystemTests`` class. They should also
+be marked with ``@pytest.marker.system(SYSTEM)`` where system designates the system
+to be tested (for example ``google.cloud``). Those tests are skipped by default.
+You can execute the tests providing ``--systems SYSTEMS`` flag to pytest.
 
-The system tests execute a specified
-example DAG file that runs the DAG end-to-end.
+The system tests execute a specified example DAG file that runs the DAG end-to-end.
+
+Most of the system tests have also addtional requirements for example they need to have
 
 Review comment:
   Note that this is only for local testing for now and manual test execution. We do not yet plan execution in automated way in CI and I am open to discussion here for sure.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

ashb commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-583895848
 
 
   I've skimmed the changes and they look fine, but I don't see how this is connected to testing backported operators?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r376828903
 
 

 ##########
 File path: scripts/ci/docker-compose/local.yml
 ##########
 @@ -53,7 +53,9 @@ services:
       - ../../../setup.cfg:/opt/airflow/setup.cfg:cached
       - ../../../setup.py:/opt/airflow/setup.py:cached
       - ../../../tests:/opt/airflow/tests:cached
+      - ../../../tests:/opt/airflow-backport/tests:cached
       - ../../../tmp:/opt/airflow/tmp:cached
+      - ${HOME}/.config:/root/.config
 
 Review comment:
   Precisely for that reason. This way for example gcloud authentication is mapped to the container, so you can use your host gcloud authentication to run gcloud commands in docker
   
   This is only for CI/development/local execution so there is no risk in passing it to inside the docker. Both host and docker are controlled by the developer.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-589365981
 
 
   also applied the comments from @efedotova !

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r376830835

##########
File path: tests/test_utils/gcp_system_helpers.py
##########
@@ -32,43 +31,6 @@
)

-SKIP_TEST_WARNING = """
-The test is only run when the test is run in environment with GCP-system-tests enabled
-environment. You can enable it in one of two ways:
-
-* Set GCP_CONFIG_DIR environment variable to point to the GCP configuration
- directory which keeps the {} key.
-* Run this test within automated environment variable workspace where
- config directory is checked out next to the airflow one.
-
-"""
-
-SKIP_LONG_TEST_WARNING = """
-The test is only run when the test is run in with GCP-system-tests enabled
-environment. And environment variable GCP_ENABLE_LONG_TESTS is set to True.
-You can enable it in one of two ways:
-
-* Set GCP_CONFIG_DIR environment variable to point to the GCP configuration
- directory which keeps variables.env file with environment variables to set
- and keys directory which keeps service account keys in .json format and
- set GCP_ENABLE_LONG_TESTS to True
-* Run this test within automated environment variable workspace where
- config directory is checked out next to the airflow one.
-"""
-
-
-LOCAL_EXECUTOR_WARNING = """

Review comment:
Yes. I replaced it with @pytest.markers.backend("postgres", "mysql"). LocalExecutor is the default executor for both Postgres and Mysql and SequentialExecutor is default (and the only available) for sqlite. By constraining the tests to run only on Postgres/MySql we automatically set local executor for unit tests.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r376828458
 
 

 ##########
 File path: scripts/ci/docker-compose/local.yml
 ##########
 @@ -53,7 +53,9 @@ services:
       - ../../../setup.cfg:/opt/airflow/setup.cfg:cached
       - ../../../setup.py:/opt/airflow/setup.py:cached
       - ../../../tests:/opt/airflow/tests:cached
+      - ../../../tests:/opt/airflow-backport/tests:cached
       - ../../../tmp:/opt/airflow/tmp:cached
+      - ${HOME}/.config:/root/.config
 
 Review comment:
   Why do you want this directory to be available in the container? It contains very sensitive data, e.g. oAuth.2.0 for gcloud and hub.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk edited a comment on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-585195708
 
 
   Hello @ashb @mik-laj @nuclearpinguin @kaxil @dimberman  -> I updated the implementation now and it works really nicely. 
   
   Breeze is now turned into "airflow_1.10.*" testing playground. It will be super-useful to automatically run system tests (currently only for google/* operators) using the latests tests and "example_dags" from "providers", but using the installed airflow version.
   
   You can run `./breeze --install-airflow-version <VERSION>` . If you run it with "current" version it works as previously (current version from sources). If you run it with '1.10.1' -> '1.10.9' it will remove the current sources, will uninstall installed airflow master and will install the released version from PIP.  The automated installation works out of the box for all versions: 1.10.1 -> 1.10.9.
   
   There is an easy way to test installation and run system tests of the released "backported" providers package. 
   
   If you have binary wheel "providers" package prepared locally or downloaded, you just need to copy the wheel package to "files" dir. You can download it from the latest build of teh 
   
   Then inside the container with installed airflow version you should be able to run: `pip install /files/<NAME_OF_THE_FILE>.whl` to install the "providers" package. And then the example tests should run out of the box for most operators.
   
   The "/opt/airflow/tests" folders and "/providers" folders are mapped into the container (and system test class handles it properly) so that we can still modify the tests locally and run them with airflow 1.10.*.
   
   For example, right after installing the 'providers' package you should be able to run:
   
   `pytest tests/providers/google/cloud/operators/test_functions_system.py --systems google.cloud` and the system tests 
   
   It will work for both - 2.0 and 1.10.* series providing that you have proper GCP credentials in "files/airflow-breeze-config/variables.env"
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r378910666
 
 

 ##########
 File path: TESTING.rst
 ##########
 @@ -31,8 +31,7 @@ Airflow Test Infrastructure
 
 * **System tests** are automatic tests that use external systems like
   Google Cloud Platform. These tests are intended for an end-to-end DAG execution.
-  Note that automated execution of these tests is still
-  `work in progress <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems#app-switcher>`_.
+  The tests can be executed on both.
 
 Review comment:
   On both what?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

codecov-io commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-583898179
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=h1) Report
   > Merging [#7389](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1464f16eabab8f3d6ab936a2cd2a3127f1536863?src=pr&el=desc) will **increase** coverage by `53.9%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7389/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7389      +/-   ##
   ==========================================
   + Coverage   32.59%   86.49%   +53.9%     
   ==========================================
     Files         871      871              
     Lines       40660    40660              
   ==========================================
   + Hits        13253    35169   +21916     
   + Misses      27407     5491   -21916
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/bin/airflow](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vYWlyZmxvdw==) | `84.61% <0%> (ø)` | :arrow_up: |
   | [airflow/plugins\_manager.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `86.95% <0%> (+0.72%)` | :arrow_up: |
   | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <0%> (+2.1%)` | :arrow_up: |
   | [airflow/exceptions.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGNlcHRpb25zLnB5) | `100% <0%> (+3.57%)` | :arrow_up: |
   | [airflow/utils/decorators.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kZWNvcmF0b3JzLnB5) | `90.47% <0%> (+4.76%)` | :arrow_up: |
   | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `92.25% <0%> (+5.63%)` | :arrow_up: |
   | [...roviders/google/cloud/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9wb3N0Z3Jlc190b19nY3MucHk=) | `52.94% <0%> (+5.88%)` | :arrow_up: |
   | [airflow/www/forms.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvZm9ybXMucHk=) | `100% <0%> (+8%)` | :arrow_up: |
   | [airflow/utils/timeout.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy90aW1lb3V0LnB5) | `75% <0%> (+8.33%)` | :arrow_up: |
   | [airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5) | `85.71% <0%> (+8.57%)` | :arrow_up: |
   | ... and [736 more](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=footer). Last update [1464f16...befb432](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-587763532
 
 
   Applied changes from @ashb comments in #7439 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-584021242
 
 
   @ashb -> one more thing. With few more breeze's modifications, It will actually be possible (bear with me) to also run unit tests with the old version of airflow. I plan to make it super easy to remove "sources" of airflow but leave tests and install any old version of Airflow inside the breeze docker. That will - in theory make it possible to run the "unit" and "integration" tests against old version of Airflow. But I am afraid due to this mocking it will not really work in many cases.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk edited a comment on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-586260298

So for me this is more of an enabler for AIP-4 discussion to happen.

WDYT @ashb ?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

ashb commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r378795761
 
 

 ##########
 File path: TESTING.rst
 ##########
 @@ -474,21 +473,54 @@ More information:
 Airflow System Tests
 ====================
 
-The System tests for Airflow are not yet fully implemented. They are Work In Progress of the
-`AIP-4 Support for System Tests for external systems <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems>`__.
 These tests need to communicate with external services/systems that are available
 if you have appropriate credentials configured for your tests.
-The tests derive from ``tests.system_test_class.SystemTests`` class.
+The tests derive from ``tests.system_test_class.SystemTests`` class. They should also
+be marked with ``@pytest.marker.system(SYSTEM)`` where system designates the system
+to be tested (for example ``google.cloud``). Those tests are skipped by default.
+You can execute the tests providing ``--systems SYSTEMS`` flag to pytest.
 
-The system tests execute a specified
-example DAG file that runs the DAG end-to-end.
+The system tests execute a specified example DAG file that runs the DAG end-to-end.
+
+Most of the system tests have also addtional requirements for example they need to have
+credentials to connect to external systems. This is don with pytest marker
+``@pytest.markers.credential_file(file)`` where "file" is a file containing
+credentials. The credential file should be relative path from
+``/files/airflow-breeze-config/keys`` directory. In Breeze environment it is mapped
+from the ``files`` directory in airflow's sources.
+
+Some of the system tests are long lasting ones (they require more than 20-30 minutes
+to complete}. They are marked with ```@pytest.markers.long_lasting`` marker.
+They are skipped by default unless you specify ``--long-lasting`` flag to pytest.
 
 An example of such a system test is
 ``airflow.tests.providers.google.operators.test_natural_language_system.CloudNaturalLanguageExampleDagsTest``.
 
-For now you can execute the system tests and follow messages printed to get them running. Soon more information on
-running the tests will be available.
 
+Running system tests
+--------------------
+
+As mentioned above - the system tests are only executed when you specify ``--systems SYSTEMS``
+flag where SYSTEMS is coma-separated list of systems to run the system tests for.
+
+
+Running tests for older Airflow versionsIRFLOW
 
 Review comment:
   ```suggestion
   Running tests for older Airflow versions
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-583898055
 
 
   
   
   
   > I've skimmed the changes and they look fine, but I don't see how this is connected to testing backported operators?
   
   I am writing devlist message shortly about it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r379122368
 
 

 ##########
 File path: BREEZE.rst
 ##########
 @@ -611,7 +617,13 @@ This is the current syntax for  `./breeze <./breeze>`_:
   -S, --static-check <STATIC_CHECK>
           Run selected static checks for currently changed files. You should specify static check that
           you would like to run or 'all' to run all checks. One of
-          [ all all-but-pylint bat-tests check-apache-license check-executables-have-shebangs check-hooks-apply check-merge-conflict check-xml debug-statements doctoc detect-private-key end-of-file-fixer flake8 forbid-tabs insert-license lint-dockerfile mixed-line-ending mypy pylint pylint-test setup-order shellcheck].
+
+  all all-but-pylint bat-tests check-apache-license
 
 Review comment:
   Resolved,

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r376815961
 
 

 ##########
 File path: TESTING.rst
 ##########
 @@ -474,20 +473,58 @@ More information:
 Airflow System Tests
 ====================
 
-The System tests for Airflow are not yet fully implemented. They are Work In Progress of the
-`AIP-4 Support for System Tests for external systems <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems>`__.
 These tests need to communicate with external services/systems that are available
 if you have appropriate credentials configured for your tests.
-The tests derive from ``tests.system_test_class.SystemTests`` class.
+The tests derive from ``tests.system_test_class.SystemTests`` class. They should also
+be marked with ``@pytest.marker.system(SYSTEM)`` where system designates the system
+to be tested (for example ``google.cloud``). Those tests are skipped by default.
+You can execute the tests providing ``--systems SYSTEMS`` flag to pytest.
 
-The system tests execute a specified
-example DAG file that runs the DAG end-to-end.
+The system tests execute a specified example DAG file that runs the DAG end-to-end.
+
+Most of the system tests have also addtional requirements for example they need to have
+credentials to connect to external systems. This is don with pytest marker
+``@pytest.markers.credential_file(file)`` where "file" is a file containing
+credentials. The credential file should be relative path from
+``/files/airflow-breeze-config/keys`` directory. In Breeze environment it is mapped
+from the ``files`` directory in airflow's sources.
+
+Some of the system tests are long lasting ones (they require more than 20-30 minutes
+to complete}. They are marked with ```@pytest.markers.long_lasting`` marker.
+They are skipped by default unless you specify ``--long-lasting`` flag to pytest.
 
 An example of such a system test is
 ``airflow.tests.providers.google.operators.test_natural_language_system.CloudNaturalLanguageExampleDagsTest``.
 
-For now you can execute the system tests and follow messages printed to get them running. Soon more information on
-running the tests will be available.
+
+Running system tests for a system
+---------------------------------
+
+As mentioned above - the system tests are only executed when you specify ``--systems SYSTEMS
+flag where SYSTEMS is coma-separated list of systems to run the system tests for.
+
+
+Running system tests for older Airlow versions
+----------------------------------------------
+
+The system tests can be executed against the master version of airflow but they also work
 
 Review comment:
   ```suggestion
   The system tests can be executed against the master version of Airflow but they also work
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

codecov-io edited a comment on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-583898179
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=h1) Report
   > Merging [#7389](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/1464f16eabab8f3d6ab936a2cd2a3127f1536863?src=pr&el=desc) will **increase** coverage by `54%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7389/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7389      +/-   ##
   ==========================================
   + Coverage   32.59%   86.59%     +54%     
   ==========================================
     Files         871      871              
     Lines       40660    40660              
   ==========================================
   + Hits        13253    35210   +21957     
   + Misses      27407     5450   -21957
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/bin/airflow](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vYWlyZmxvdw==) | `84.61% <0%> (ø)` | :arrow_up: |
   | [airflow/plugins\_manager.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `86.95% <0%> (+0.72%)` | :arrow_up: |
   | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <0%> (+2.1%)` | :arrow_up: |
   | [airflow/exceptions.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGNlcHRpb25zLnB5) | `100% <0%> (+3.57%)` | :arrow_up: |
   | [airflow/utils/decorators.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kZWNvcmF0b3JzLnB5) | `90.47% <0%> (+4.76%)` | :arrow_up: |
   | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `92.25% <0%> (+5.63%)` | :arrow_up: |
   | [...roviders/google/cloud/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9wb3N0Z3Jlc190b19nY3MucHk=) | `85.29% <0%> (+38.23%)` | :arrow_up: |
   | [airflow/www/forms.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvZm9ybXMucHk=) | `100% <0%> (+8%)` | :arrow_up: |
   | [airflow/utils/timeout.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy90aW1lb3V0LnB5) | `75% <0%> (+8.33%)` | :arrow_up: |
   | [airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5) | `85.71% <0%> (+8.57%)` | :arrow_up: |
   | ... and [736 more](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=footer). Last update [1464f16...e6faaed](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-583912240

@ashb > (i.e. why the system bit, Vs unit test is needed to check the backport)

The backported packages are already unit-tested. They are copied straight from the master where the unit tests have been run and are green in CI.

There is basically no easy way - unless you want to rewrite manually all the tests - to run unit tests with back-ported operators/hooks/sensors from 2.0 in the context of Airlfow 1.10. I at least could not find any way to do it easily - but if you think you can do it easily, it would be great. I just think it is awfully lot of effort.

Many of the unit tests use mocking for internal structure of packages and it would not work to convert those.

And it would not make a lot of sense because we would just (again) test the same - behaviors of the operators - pretty much all the "operators/hooks/sensors" test are mocking out the communication to the external systems and test internal logic of the operators. We do not need to test it again - we know that this works.

What we do not know if the operators are working with airflow-1.10 framework and whether the framework changes did not spoil anything. So we really need to test integration of the operators from 2.0 to 1.10 and the best we can do here is to run those (already unit-tested) operators in the context of 1.10 talking to the real system and see if they are still working there. We have a lot of such tests for GCP and we can fairly easiy test those 2.0 operators to run on top of airflow 1.10.x. Which is really what we want to know.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

ashb commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-583895928
 
 
   (i.e. why the system bit, Vs unit test is needed to check the backport)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk merged pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk merged pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r376816372
 
 

 ##########
 File path: TESTING.rst
 ##########
 @@ -474,20 +473,58 @@ More information:
 Airflow System Tests
 ====================
 
-The System tests for Airflow are not yet fully implemented. They are Work In Progress of the
-`AIP-4 Support for System Tests for external systems <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems>`__.
 These tests need to communicate with external services/systems that are available
 if you have appropriate credentials configured for your tests.
-The tests derive from ``tests.system_test_class.SystemTests`` class.
+The tests derive from ``tests.system_test_class.SystemTests`` class. They should also
+be marked with ``@pytest.marker.system(SYSTEM)`` where system designates the system
+to be tested (for example ``google.cloud``). Those tests are skipped by default.
+You can execute the tests providing ``--systems SYSTEMS`` flag to pytest.
 
-The system tests execute a specified
-example DAG file that runs the DAG end-to-end.
+The system tests execute a specified example DAG file that runs the DAG end-to-end.
+
+Most of the system tests have also addtional requirements for example they need to have
+credentials to connect to external systems. This is don with pytest marker
+``@pytest.markers.credential_file(file)`` where "file" is a file containing
+credentials. The credential file should be relative path from
+``/files/airflow-breeze-config/keys`` directory. In Breeze environment it is mapped
+from the ``files`` directory in airflow's sources.
+
+Some of the system tests are long lasting ones (they require more than 20-30 minutes
+to complete}. They are marked with ```@pytest.markers.long_lasting`` marker.
+They are skipped by default unless you specify ``--long-lasting`` flag to pytest.
 
 An example of such a system test is
 ``airflow.tests.providers.google.operators.test_natural_language_system.CloudNaturalLanguageExampleDagsTest``.
 
-For now you can execute the system tests and follow messages printed to get them running. Soon more information on
-running the tests will be available.
+
+Running system tests for a system
+---------------------------------
+
+As mentioned above - the system tests are only executed when you specify ``--systems SYSTEMS
+flag where SYSTEMS is coma-separated list of systems to run the system tests for.
+
+
+Running system tests for older Airlow versions
+----------------------------------------------
+
+The system tests can be executed against the master version of airflow but they also work
+with older versions of airflow. This is especially useful to tests backported operators
+from airflow 2.0 to 1.10.* versions.
+
+In order to run the tests for airflFow 1.10.* series you need to do the following steps after entering
+the Breeze environment:
+
+.. code-block:: bash
+
+  unset PYTHONPATH
+  ./scripts/ci/in_container/install_older_airflow.sh <AIRFLOW_VERSION>
+  cd /opt/airflow-backport/
+
+The commands make sure that older version of Airflow is installed and source version of master airflow
+is removed from the PYTHONPATH. Also the current system tests are mapped to
+/opt/airflow-backport/test directory so you can run them directly from there.
+
+You can skip <AIRLFOW_VERSION> and the latest version of stable airflow is installed.
 
 Review comment:
   ```suggestion
   You can skip <AIRLFOW_VERSION> and the latest version of stable Airflow is installed.
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

ashb commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r378794431
 
 

 ##########
 File path: BREEZE.rst
 ##########
 @@ -206,6 +206,12 @@ run tests immediately.
 You can `set up autocomplete <#setting-up-autocomplete>`_ for commands and add the
 checked-out Airflow repository to your PATH to run Breeze without the ``./`` and from any directory.
 
+
+When you enter the Breeze environment, automatically an environment file is sourced from
+``files/airflow-breeze-config/variables.env``. The ``file`` folder from your local sources is
 
 Review comment:
   ```suggestion
   ``files/airflow-breeze-config/variables.env``. The ``files`` folder from your local sources is
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r376829947
 
 

 ##########
 File path: scripts/ci/docker-compose/local.yml
 ##########
 @@ -53,7 +53,9 @@ services:
       - ../../../setup.cfg:/opt/airflow/setup.cfg:cached
       - ../../../setup.py:/opt/airflow/setup.py:cached
       - ../../../tests:/opt/airflow/tests:cached
+      - ../../../tests:/opt/airflow-backport/tests:cached
       - ../../../tmp:/opt/airflow/tmp:cached
+      - ${HOME}/.config:/root/.config
 
 Review comment:
   Personally, I don't trust any code that runs in the container. In my opinion, there should be no sensitive data in the container, because users who are unaware that some sensitive data is available in the container may hurt themselves. Could you add it as an optional feature?
   
   There is a technical possibility to do this in a way that does not allow the user to hurt himself. We can talk about it tomorrow.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-583890988
 
 
   @nuclearpinguin @mik-laj  (and @ashb / @kaxil ) - this is a change that makes it very easy to run system tests using pytest's markers in the breeze/CI environment. We will use it to test all the backported 2.0 packages on older airlfow versions. For now for Google/GCP for which we have system tests integrated, but it's very generic and we can apply it for other systems as well as we will have a possibility to run the system tests automatically in the future when we move to github actions.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r376932574
 
 

 ##########
 File path: TESTING.rst
 ##########
 @@ -474,20 +473,58 @@ More information:
 Airflow System Tests
 ====================
 
-The System tests for Airflow are not yet fully implemented. They are Work In Progress of the
-`AIP-4 Support for System Tests for external systems <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems>`__.
 These tests need to communicate with external services/systems that are available
 if you have appropriate credentials configured for your tests.
-The tests derive from ``tests.system_test_class.SystemTests`` class.
+The tests derive from ``tests.system_test_class.SystemTests`` class. They should also
+be marked with ``@pytest.marker.system(SYSTEM)`` where system designates the system
+to be tested (for example ``google.cloud``). Those tests are skipped by default.
+You can execute the tests providing ``--systems SYSTEMS`` flag to pytest.
 
-The system tests execute a specified
-example DAG file that runs the DAG end-to-end.
+The system tests execute a specified example DAG file that runs the DAG end-to-end.
+
+Most of the system tests have also addtional requirements for example they need to have
 
 Review comment:
   I agree however it should not be default - i will make it optional . I am also planning to have some small modifications to make it even easier to use with older versions of airflow. Bear with me,

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r376818874
 
 

 ##########
 File path: TESTING.rst
 ##########
 @@ -31,8 +31,7 @@ Airflow Test Infrastructure
 
 * **System tests** are automatic tests that use external systems like
   Google Cloud Platform. These tests are intended for an end-to-end DAG execution.
-  Note that automated execution of these tests is still
-  `work in progress <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems#app-switcher>`_.
+  The tests can be executde on both.
 
 Review comment:
   ```suggestion
     The tests can be executed on both.
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

codecov-io edited a comment on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-583898179
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=h1) Report
   > Merging [#7389](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/6b19889c0151b66202c2b1b38d99cbb54529b255?src=pr&el=desc) will **increase** coverage by `0.21%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7389/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7389      +/-   ##
   ==========================================
   + Coverage   86.44%   86.66%   +0.21%     
   ==========================================
     Files         874      874              
     Lines       40868    40868              
   ==========================================
   + Hits        35330    35417      +87     
   + Misses       5538     5451      -87
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/www/api/experimental/endpoints.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvYXBpL2V4cGVyaW1lbnRhbC9lbmRwb2ludHMucHk=) | `89.81% <0%> (ø)` | :arrow_up: |
   | [...low/providers/google/cloud/operators/sql\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9zcWxfdG9fZ2NzLnB5) | `92.07% <0%> (ø)` | :arrow_up: |
   | [airflow/providers/amazon/aws/hooks/s3.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9zMy5weQ==) | `95.85% <0%> (+0.13%)` | :arrow_up: |
   | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.34% <0%> (+0.14%)` | :arrow_up: |
   | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `88.12% <0%> (+0.19%)` | :arrow_up: |
   | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `91.16% <0%> (+0.25%)` | :arrow_up: |
   | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `91.73% <0%> (+0.82%)` | :arrow_up: |
   | [airflow/providers/apache/hive/hooks/hive.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvaG9va3MvaGl2ZS5weQ==) | `77.55% <0%> (+1.53%)` | :arrow_up: |
   | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `91.43% <0%> (+3.42%)` | :arrow_up: |
   | [...roviders/amazon/aws/operators/s3\_delete\_objects.py](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9vcGVyYXRvcnMvczNfZGVsZXRlX29iamVjdHMucHk=) | `100% <0%> (+9.52%)` | :arrow_up: |
   | ... and [3 more](https://codecov.io/gh/apache/airflow/pull/7389/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=footer). Last update [6b19889...0c78373](https://codecov.io/gh/apache/airflow/pull/7389?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r376818935
 
 

 ##########
 File path: TESTING.rst
 ##########
 @@ -474,20 +473,58 @@ More information:
 Airflow System Tests
 ====================
 
-The System tests for Airflow are not yet fully implemented. They are Work In Progress of the
-`AIP-4 Support for System Tests for external systems <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems>`__.
 These tests need to communicate with external services/systems that are available
 if you have appropriate credentials configured for your tests.
-The tests derive from ``tests.system_test_class.SystemTests`` class.
+The tests derive from ``tests.system_test_class.SystemTests`` class. They should also
+be marked with ``@pytest.marker.system(SYSTEM)`` where system designates the system
+to be tested (for example ``google.cloud``). Those tests are skipped by default.
+You can execute the tests providing ``--systems SYSTEMS`` flag to pytest.
 
-The system tests execute a specified
-example DAG file that runs the DAG end-to-end.
+The system tests execute a specified example DAG file that runs the DAG end-to-end.
+
+Most of the system tests have also addtional requirements for example they need to have
+credentials to connect to external systems. This is don with pytest marker
+``@pytest.markers.credential_file(file)`` where "file" is a file containing
+credentials. The credential file should be relative path from
+``/files/airflow-breeze-config/keys`` directory. In Breeze environment it is mapped
+from the ``files`` directory in airflow's sources.
+
+Some of the system tests are long lasting ones (they require more than 20-30 minutes
+to complete}. They are marked with ```@pytest.markers.long_lasting`` marker.
+They are skipped by default unless you specify ``--long-lasting`` flag to pytest.
 
 An example of such a system test is
 ``airflow.tests.providers.google.operators.test_natural_language_system.CloudNaturalLanguageExampleDagsTest``.
 
-For now you can execute the system tests and follow messages printed to get them running. Soon more information on
-running the tests will be available.
+
+Running system tests for a system
+---------------------------------
+
+As mentioned above - the system tests are only executed when you specify ``--systems SYSTEMS
 
 Review comment:
   ```suggestion
   As mentioned above - the system tests are only executed when you specify ``--systems SYSTEMS``
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r376829163
 
 

 ##########
 File path: tests/test_utils/gcp_system_helpers.py
 ##########
 @@ -32,43 +31,6 @@
 )
 
 
-SKIP_TEST_WARNING = """
-The test is only run when the test is run in environment with GCP-system-tests enabled
-environment. You can enable it in one of two ways:
-
-* Set GCP_CONFIG_DIR environment variable to point to the GCP configuration
-  directory which keeps the {} key.
-* Run this test within automated environment variable workspace where
-  config directory is checked out next to the airflow one.
-
-"""
-
-SKIP_LONG_TEST_WARNING = """
-The test is only run when the test is run in with GCP-system-tests enabled
-environment. And environment variable GCP_ENABLE_LONG_TESTS is set to True.
-You can enable it in one of two ways:
-
-* Set GCP_CONFIG_DIR environment variable to point to the GCP configuration
-  directory which keeps variables.env file with environment variables to set
-  and keys directory which keeps service account keys in .json format and
-  set GCP_ENABLE_LONG_TESTS to True
-* Run this test within automated environment variable workspace where
-  config directory is checked out next to the airflow one.
-"""
-
-
-LOCAL_EXECUTOR_WARNING = """
 
 Review comment:
   Don't we need it anymore? Do we have another way to run tests with LocalExecutor?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

mik-laj commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r376816332
 
 

 ##########
 File path: TESTING.rst
 ##########
 @@ -474,20 +473,58 @@ More information:
 Airflow System Tests
 ====================
 
-The System tests for Airflow are not yet fully implemented. They are Work In Progress of the
-`AIP-4 Support for System Tests for external systems <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems>`__.
 These tests need to communicate with external services/systems that are available
 if you have appropriate credentials configured for your tests.
-The tests derive from ``tests.system_test_class.SystemTests`` class.
+The tests derive from ``tests.system_test_class.SystemTests`` class. They should also
+be marked with ``@pytest.marker.system(SYSTEM)`` where system designates the system
+to be tested (for example ``google.cloud``). Those tests are skipped by default.
+You can execute the tests providing ``--systems SYSTEMS`` flag to pytest.
 
-The system tests execute a specified
-example DAG file that runs the DAG end-to-end.
+The system tests execute a specified example DAG file that runs the DAG end-to-end.
+
+Most of the system tests have also addtional requirements for example they need to have
+credentials to connect to external systems. This is don with pytest marker
+``@pytest.markers.credential_file(file)`` where "file" is a file containing
+credentials. The credential file should be relative path from
+``/files/airflow-breeze-config/keys`` directory. In Breeze environment it is mapped
+from the ``files`` directory in airflow's sources.
+
+Some of the system tests are long lasting ones (they require more than 20-30 minutes
+to complete}. They are marked with ```@pytest.markers.long_lasting`` marker.
+They are skipped by default unless you specify ``--long-lasting`` flag to pytest.
 
 An example of such a system test is
 ``airflow.tests.providers.google.operators.test_natural_language_system.CloudNaturalLanguageExampleDagsTest``.
 
-For now you can execute the system tests and follow messages printed to get them running. Soon more information on
-running the tests will be available.
+
+Running system tests for a system
+---------------------------------
+
+As mentioned above - the system tests are only executed when you specify ``--systems SYSTEMS
+flag where SYSTEMS is coma-separated list of systems to run the system tests for.
+
+
+Running system tests for older Airlow versions
+----------------------------------------------
+
+The system tests can be executed against the master version of airflow but they also work
+with older versions of airflow. This is especially useful to tests backported operators
+from airflow 2.0 to 1.10.* versions.
+
+In order to run the tests for airflFow 1.10.* series you need to do the following steps after entering
 
 Review comment:
   ```suggestion
   In order to run the tests for Airflow 1.10.* series you need to do the following steps after entering
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

potiuk commented on issue #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#issuecomment-586177817
 
 
   Ping.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests

Posted by GitBox <gi...@apache.org>.

ashb commented on a change in pull request #7389: [AIRFLOW-6763] Make systems tests ready for backport tests
URL: https://github.com/apache/airflow/pull/7389#discussion_r378795243
 
 

 ##########
 File path: TESTING.rst
 ##########
 @@ -31,8 +31,7 @@ Airflow Test Infrastructure
 
 * **System tests** are automatic tests that use external systems like
   Google Cloud Platform. These tests are intended for an end-to-end DAG execution.
-  Note that automated execution of these tests is still
-  `work in progress <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems#app-switcher>`_.
+  The tests can be executed on both.
 
 Review comment:
   On both what?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services