You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2020/11/14 00:39:44 UTC
[airflow] branch v1-10-test updated (ef2cf4d -> 612c69e)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git.
from ef2cf4d Use snakebite-py3 for HDFS dependency for Python3 (#12340)
new c56d91c Add Airflow 1.10.12 to breeze-complete (#10592)
new 527e513 Nightly tag push is not skipped in scheduled builds (#10597)
new 5ac7fd2 Update node installation cmd (#10744)
new 8736b86 Cache for kubernetes tests is updateable (#10945)
new 6fffe6d Fix typos in Dockerfile.ci (#11187)
new 1021d05 Fix typo in command in CI.rst (#11233)
new 5f37463 Add Python version to Breeze cmd (#11228)
new bfbbcab The bats script for CI image is now placed in the docker folder (#11262)
new 11a4e22 Fix link to static checks in CONTRIBUTING.rst (#11271)
new 6f86f98 Add remaining community guidelines to CONTRIBUTING.rst (#11312)
new 4a12f17 Moves Commiter's guide to CONTRIBUTING.rst (#11314)
new a41c6ca Adds --install-wheels flag to breeze command line (#11317)
new a159625 Improve Committer's guide docs (#11338)
new fde0151 Fix command to run tmux with breeze in BREEZE.rst (#11340)
new 8878998 Split tests to more sub-types (#11402)
new 612c69e Fix constraints generation script (#11412)
The 16 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.dockerignore | 3 +
.github/workflows/ci.yml | 64 ++++++++-
BREEZE.rst | 58 ++++++--
CI.rst | 2 +-
CONTRIBUTING.rst | 38 ++++-
Dockerfile.ci | 6 +-
TESTING.rst | 57 +++++++-
breeze | 91 +++++++++---
breeze-complete | 14 +-
scripts/ci/docker-compose/base.yml | 10 +-
scripts/ci/libraries/_docker.env | 5 +
scripts/ci/libraries/_initialization.sh | 52 ++++---
scripts/ci/libraries/_parameters.sh | 28 ++--
scripts/ci/libraries/_push_pull_remove_images.sh | 14 ++
scripts/ci/testing/ci_run_airflow_testing.sh | 11 +-
scripts/in_container/entrypoint_ci.sh | 154 ++++++++++++++++-----
scripts/in_container/run_ci_tests.sh | 59 ++++++--
scripts/in_container/run_generate_constraints.sh | 5 +-
tests/bats/test_breeze_complete.bats | 8 ++
tests/{ => core}/test_config_templates.py | 0
tests/{ => core}/test_configuration.py | 0
tests/{ => core}/test_core.py | 0
tests/{ => core}/test_example_dags_system.py | 0
.../test_impersonation_tests.py} | 0
tests/{ => core}/test_logging_config.py | 0
tests/{ => core}/test_sentry.py | 0
tests/{ => core}/test_sqlalchemy_config.py | 3 +-
tests/jobs/test_local_task_job.py | 2 +-
.../task/task_runner/test_standard_task_runner.py | 6 +-
29 files changed, 545 insertions(+), 145 deletions(-)
rename tests/{ => core}/test_config_templates.py (100%)
rename tests/{ => core}/test_configuration.py (100%)
rename tests/{ => core}/test_core.py (100%)
rename tests/{ => core}/test_example_dags_system.py (100%)
rename tests/{test_impersonation.py => core/test_impersonation_tests.py} (100%)
rename tests/{ => core}/test_logging_config.py (100%)
rename tests/{ => core}/test_sentry.py (100%)
rename tests/{ => core}/test_sqlalchemy_config.py (96%)
[airflow] 09/16: Fix link to static checks in CONTRIBUTING.rst
(#11271)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 11a4e2240dbfcdba04afd24d2d0361491220ee68
Author: Kamil Breguła <mi...@users.noreply.github.com>
AuthorDate: Mon Oct 5 09:08:54 2020 +0200
Fix link to static checks in CONTRIBUTING.rst (#11271)
(cherry picked from commit 1180f1ba4284cfa167b8e4dcb21f90f251812d68)
---
CONTRIBUTING.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index 7af8813..85558f3 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -341,7 +341,7 @@ Step 4: Prepare PR
* Run the tests in `Breeze <TESTING.rst#running-unit-tests-inside-breeze>`__.
- * Run and fix all the `static checks <STATIC_CODE_CHECKS>`__. If you have
+ * Run and fix all the `static checks <STATIC_CODE_CHECKS.rst>`__. If you have
`pre-commits installed <STATIC_CODE_CHECKS.rst#pre-commit-hooks>`__,
this step is automatically run while you are committing your code. If not, you can do it manually
via ``git add`` and then ``pre-commit run``.
[airflow] 08/16: The bats script for CI image is now placed in the
docker folder (#11262)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit bfbbcabbddfe5da9702f1cdbdcdfa7f19f39718b
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sun Oct 4 08:30:11 2020 +0200
The bats script for CI image is now placed in the docker folder (#11262)
The script was previously placed in scripts/ci which caused
a bit of a problem in 1-10-test branch where PRs were using
scripts/ci from the v1-10-test HEAD but they were missing
the ci script from the PR.
The scripts "ci" are parts of the host scripts that are
always taken from master when the image is built, but
all the other stuff should be taken from "docker"
folder - which will be taken from the PR.
(cherry picked from commit e89d384688a389bf7bd6f5d390a43f322f4956ec)
---
.dockerignore | 3 +++
1 file changed, 3 insertions(+)
diff --git a/.dockerignore b/.dockerignore
index 7c5ad9b..13c8543 100644
--- a/.dockerignore
+++ b/.dockerignore
@@ -39,6 +39,9 @@
!scripts/in_container
!scripts/docker
+# Add backport packages to the context
+!backport_packages
+
# Add tests and kubernetes_tests to context.
!tests
!kubernetes_tests
[airflow] 13/16: Improve Committer's guide docs (#11338)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit a159625a57582bda7a69f393b1c1cd50f19491cc
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Thu Oct 8 10:24:07 2020 +0100
Improve Committer's guide docs (#11338)
(cherry picked from commit 625afa2af226a759299927ad64ca25fe99ff5a1e)
---
CONTRIBUTING.rst | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index 0ab4079..5330485 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -1276,10 +1276,9 @@ The following commit policy passed by a vote 8(binding FOR) to 0 against on May
and slightly modified and consensus reached in October 2020:
* Commits need a +1 vote from a committer who is not the author
-* Furthermore, do not merge a PR that regresses linting or does not pass CI tests (unless we have
+* Do not merge a PR that regresses linting or does not pass CI tests (unless we have
justification such as clearly transient error).
-* When we do AIP voting, both PMC and committer +1s are considered as binding vote based on internal
- PMC discussion.
+* When we do AIP voting, both PMC and committer +1s are considered as binding vote.
Resources & Links
=================
[airflow] 04/16: Cache for kubernetes tests is updateable (#10945)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 8736b86ac74eb910adeecd0f3fdd1c5fd40cdd50
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Tue Sep 15 02:24:15 2020 +0200
Cache for kubernetes tests is updateable (#10945)
The cache in Github Actions is immutable - once you create it
it cannot be modified. That's why cache keys should contain
hash of all files that are used to create the cache.
Kubernetes cache key did not contain it, and as a side effect
the cache from master kubernetes setup.py was used in the v1-10-test
after the breeze changes were cherry-picked.
(cherry picked from commit 83ed6bdb3f003f2f8c3c7eaeba338194064cecc9)
---
.github/workflows/ci.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index b57fdbd..5e14f56 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -564,7 +564,7 @@ jobs:
cache-name: cache-kubernetes-tests-virtualenv-v4
with:
path: .build/.kubernetes_venv
- key: "${{ env.cache-name }}-${{ github.job }}-${{ hashFiles('setup.py') }}"
+ key: "${{ env.cache-name }}-${{ github.job }}-${{ hashFiles('setup.py') }}-v1"
- name: "Kubernetes Tests"
run: ./scripts/ci/kubernetes/ci_run_kubernetes_tests.sh
- name: "Upload KinD logs"
[airflow] 12/16: Adds --install-wheels flag to breeze command line
(#11317)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit a41c6cae5b6e18a19570fc4df8a7dccfad58b02d
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Thu Oct 8 10:06:53 2020 +0200
Adds --install-wheels flag to breeze command line (#11317)
If this flag is specified it will look for wheel packages placed in dist
folder and it will install the wheels from there after installing
Airflow. This is useful for testing backport packages as well as in the
future for testing provider packages for 2.0.
(cherry picked from commit de07d135ae1bda3f71dd83951bcfafc2b3ad9f89)
---
BREEZE.rst | 37 ++++++++++++++++++---------
breeze | 44 ++++++++++++++++++++++-----------
breeze-complete | 1 +
scripts/ci/docker-compose/base.yml | 3 ++-
scripts/ci/libraries/_initialization.sh | 10 ++++++++
scripts/in_container/entrypoint_ci.sh | 3 +++
6 files changed, 70 insertions(+), 28 deletions(-)
diff --git a/BREEZE.rst b/BREEZE.rst
index cd87a73..3002f05 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -1184,6 +1184,14 @@ This is the current syntax for `./breeze <./breeze>`_:
If specified, installs Airflow directly from reference in GitHub. This happens at
image building time in production image and at container entering time for CI image.
+ --no-rbac-ui
+ Disables RBAC UI when Airflow 1.10.* is installed.
+
+ --install-wheels
+ If specified it will look for wheel packages placed in dist folder and it will install the
+ wheels from there after installing Airflow. This is useful for testing backport
+ packages as well as in the future for testing provider packages for 2.0.
+
-I, --production-image
Use production image for entering the environment and builds (not for tests).
@@ -1212,6 +1220,9 @@ This is the current syntax for `./breeze <./breeze>`_:
--image-tag TAG
Additional tag in the image.
+ --disable-pip-cache
+ Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
+
--additional-extras ADDITIONAL_EXTRAS
Additional extras to pass to build images The default is no additional extras.
@@ -1262,9 +1273,6 @@ This is the current syntax for `./breeze <./breeze>`_:
constraint file placed in 'docker-context-files' folder, in which case it has to be
in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
- --disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
-
--install-local-pip-wheels
This flag is only used in production image building. If it is used then instead of
installing Airflow from PyPI, the packages are installed from the .whl packages placed
@@ -1710,6 +1718,9 @@ This is the current syntax for `./breeze <./breeze>`_:
--image-tag TAG
Additional tag in the image.
+ --disable-pip-cache
+ Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
+
--additional-extras ADDITIONAL_EXTRAS
Additional extras to pass to build images The default is no additional extras.
@@ -1760,9 +1771,6 @@ This is the current syntax for `./breeze <./breeze>`_:
constraint file placed in 'docker-context-files' folder, in which case it has to be
in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
- --disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
-
--install-local-pip-wheels
This flag is only used in production image building. If it is used then instead of
installing Airflow from PyPI, the packages are installed from the .whl packages placed
@@ -1950,9 +1958,6 @@ This is the current syntax for `./breeze <./breeze>`_:
cassandra kerberos mongo openldap presto rabbitmq redis all
- --rbac-ui
- Enable RBAC UI.
-
--init-script INIT_SCRIPT_FILE
Initialization script name - Sourced from files/airflow-breeze-config. Default value
init.sh. It will be executed after the environment is configured and started.
@@ -2036,6 +2041,14 @@ This is the current syntax for `./breeze <./breeze>`_:
If specified, installs Airflow directly from reference in GitHub. This happens at
image building time in production image and at container entering time for CI image.
+ --no-rbac-ui
+ Disables RBAC UI when Airflow 1.10.* is installed.
+
+ --install-wheels
+ If specified it will look for wheel packages placed in dist folder and it will install the
+ wheels from there after installing Airflow. This is useful for testing backport
+ packages as well as in the future for testing provider packages for 2.0.
+
****************************************************************************************************
Credentials
@@ -2071,6 +2084,9 @@ This is the current syntax for `./breeze <./breeze>`_:
--image-tag TAG
Additional tag in the image.
+ --disable-pip-cache
+ Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
+
--additional-extras ADDITIONAL_EXTRAS
Additional extras to pass to build images The default is no additional extras.
@@ -2121,9 +2137,6 @@ This is the current syntax for `./breeze <./breeze>`_:
constraint file placed in 'docker-context-files' folder, in which case it has to be
in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
- --disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
-
--install-local-pip-wheels
This flag is only used in production image building. If it is used then instead of
installing Airflow from PyPI, the packages are installed from the .whl packages placed
diff --git a/breeze b/breeze
index 522378f..e7a2ca8 100755
--- a/breeze
+++ b/breeze
@@ -63,7 +63,6 @@ export EXTRA_STATIC_CHECK_OPTIONS
# FORWARD_CREDENTIALS
# DB_RESET
# START_AIRFLOW
-# RBAC_UI
# INSTALL_AIRFLOW_VERSION
# INSTALL_AIRFLOW_REFERENCE
# FORCE_BUILD_IMAGES
@@ -124,8 +123,8 @@ function breeze::setup_default_breeze_constants() {
# If set to true, the sample dags will be created
export LOAD_EXAMPLES="false"
- # If set to true, RBAC mode is enabled
- export RBAC_UI="false"
+ # If set to true, RBAC UI will not be used for 1.10 version
+ export DISABLE_RBAC="false"
# If it set is set to specified version, then the source version of Airflow
# is removed and the specified version of Airflow is installed from PyPi
@@ -135,6 +134,10 @@ function breeze::setup_default_breeze_constants() {
# of Airflow is removed and the specified version of Airflow is installed from GitHub
export INSTALL_AIRFLOW_REFERENCE=""
+ # if set to true, the ci image will look for wheel packages in dist folder and will install them
+ # during entering the container
+ export INSTALL_WHEELS="false"
+
# Determines whether to force build without checking if it is needed
# Can be overridden by '--force-build-images' flag.
export FORCE_BUILD_IMAGES="false"
@@ -1078,12 +1081,6 @@ function breeze::parse_arguments() {
echo
shift 2
;;
- --rbac-ui)
- echo "Enable RBAC UI"
- echo
- export RBAC_UI="true"
- shift
- ;;
--load-example-dags)
export LOAD_EXAMPLES="true"
echo "Include Airflow sample dags"
@@ -1096,6 +1093,18 @@ function breeze::parse_arguments() {
echo
shift
;;
+ --no-rbac-ui)
+ export DISABLE_RBAC="true"
+ echo "When installing Airflow 1.10, RBAC UI will be disabled."
+ echo
+ shift
+ ;;
+ --install-wheels)
+ export INSTALL_WHEELS="true"
+ echo "Install wheels found in dist folder"
+ echo
+ shift
+ ;;
--)
shift
break
@@ -1978,9 +1987,6 @@ function breeze::flag_breeze_actions() {
${FORMATTED_INTEGRATIONS}
---rbac-ui
- Enable RBAC UI.
-
--init-script INIT_SCRIPT_FILE
Initialization script name - Sourced from files/airflow-breeze-config. Default value
init.sh. It will be executed after the environment is configured and started.
@@ -2078,6 +2084,14 @@ ${FORMATTED_INSTALL_AIRFLOW_VERSIONS}
-t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
If specified, installs Airflow directly from reference in GitHub. This happens at
image building time in production image and at container entering time for CI image.
+
+--no-rbac-ui
+ Disables RBAC UI when Airflow 1.10.* is installed.
+
+--install-wheels
+ If specified it will look for wheel packages placed in dist folder and it will install the
+ wheels from there after installing Airflow. This is useful for testing backport
+ packages as well as in the future for testing provider packages for 2.0.
"
}
@@ -2186,6 +2200,9 @@ ${FORMATTED_DEFAULT_PROD_EXTRAS}
--image-tag TAG
Additional tag in the image.
+--disable-pip-cache
+ Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
+
--additional-extras ADDITIONAL_EXTRAS
Additional extras to pass to build images The default is no additional extras.
@@ -2236,9 +2253,6 @@ Build options:
constraint file placed in 'docker-context-files' folder, in which case it has to be
in the form of '/docker-context-files/<NAME_OF_THE_FILE>'
---disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if github is not reachable during build.
-
--install-local-pip-wheels
This flag is only used in production image building. If it is used then instead of
installing Airflow from PyPI, the packages are installed from the .whl packages placed
diff --git a/breeze-complete b/breeze-complete
index 8b94cef..2c57a79 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -140,6 +140,7 @@ additional-extras: additional-python-deps:
dev-apt-deps: additional-dev-apt-deps: dev-apt-command: additional-dev-apt-command: additional-dev-apt-env:
runtime-apt-deps: additional-runtime-apt-deps: runtime-apt-command: additional-runtime-apt-command: additional-runtime-apt-env:
load-default-connections load-example-dags
+install-wheels no-rbac-ui
"
_breeze_commands="
diff --git a/scripts/ci/docker-compose/base.yml b/scripts/ci/docker-compose/base.yml
index 706d90c..f89aefd 100644
--- a/scripts/ci/docker-compose/base.yml
+++ b/scripts/ci/docker-compose/base.yml
@@ -52,7 +52,8 @@ services:
- START_AIRFLOW
- LOAD_EXAMPLES
- LOAD_DEFAULT_CONNECTIONS
- - RBAC_UI
+ - INSTALL_WHEELS
+ - DISABLE_RBAC
- ENABLED_SYSTEMS
- RUN_SYSTEM_TESTS
- PYTHON_MAJOR_MINOR_VERSION
diff --git a/scripts/ci/libraries/_initialization.sh b/scripts/ci/libraries/_initialization.sh
index 4831f34..576e80c 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -114,6 +114,14 @@ function initialization::initialize_base_variables() {
export LOAD_EXAMPLES=${LOAD_EXAMPLES:="false"}
export LOAD_DEFAULT_CONNECTIONS=${LOAD_DEFAULT_CONNECTIONS:="false"}
+
+ # If set to true, RBAC UI will not be used for 1.10 version
+ export DISABLE_RBAC=${DISABLE_RBAC:="false"}
+
+ # if set to true, the ci image will look for wheel packages in dist folder and will install them
+ # during entering the container
+ export INSTALL_WHEELS=${INSTALL_WHEELS:="false"}
+
# If set the specified file will be used to initialized Airflow after the environment is created,
# otherwise it will use files/airflow-breeze-config/init.sh
export INIT_SCRIPT_FILE=${INIT_SCRIPT_FILE:=""}
@@ -551,6 +559,8 @@ Initialization variables:
INIT_SCRIPT_FILE: ${INIT_SCRIPT_FILE}
LOAD_DEFAULT_CONNECTIONS: ${LOAD_DEFAULT_CONNECTIONS}
LOAD_EXAMPLES: ${LOAD_EXAMPLES}
+ INSTALL_WHEELS: ${INSTALL_WHEELS}
+ DISABLE_RBAC: ${DISABLE_RBAC}
EOF
diff --git a/scripts/in_container/entrypoint_ci.sh b/scripts/in_container/entrypoint_ci.sh
index e74fe40..f59e58e 100755
--- a/scripts/in_container/entrypoint_ci.sh
+++ b/scripts/in_container/entrypoint_ci.sh
@@ -88,6 +88,9 @@ else
install_released_airflow_version "${INSTALL_AIRFLOW_VERSION}"
fi
+if [[ ${INSTALL_WHEELS} == "true" ]]; then
+ pip install /dist/*.whl || true
+fi
export RUN_AIRFLOW_1_10=${RUN_AIRFLOW_1_10:="false"}
[airflow] 16/16: Fix constraints generation script (#11412)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 612c69e9f701acd68ace438d1ebb89365463bae9
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sun Oct 11 17:49:19 2020 +0200
Fix constraints generation script (#11412)
Constraints generation script was broken by recent changes
in naming of constraints URL variables and moving generation
of the link to the Dockerfile
This change restores the script's behaviour.
(cherry picked from commit ce2f19d30db32313fcee229561e05165144b446a)
---
scripts/ci/libraries/_docker.env | 1 +
scripts/in_container/run_generate_constraints.sh | 5 ++++-
2 files changed, 5 insertions(+), 1 deletion(-)
diff --git a/scripts/ci/libraries/_docker.env b/scripts/ci/libraries/_docker.env
index 86461d0..3290f20 100644
--- a/scripts/ci/libraries/_docker.env
+++ b/scripts/ci/libraries/_docker.env
@@ -31,6 +31,7 @@ PRINT_INFO_FROM_SCRIPTS
CI
LOAD_DEFAULT_CONNECTIONS
LOAD_EXAMPLES
+DEFAULT_CONSTRAINTS_BRANCH
GITHUB_REGISTRY_PULL_IMAGE_TAG
POSTGRES_VERSION
MYSQL_VERSION
diff --git a/scripts/in_container/run_generate_constraints.sh b/scripts/in_container/run_generate_constraints.sh
index dabf698..2809c1f 100755
--- a/scripts/in_container/run_generate_constraints.sh
+++ b/scripts/in_container/run_generate_constraints.sh
@@ -27,7 +27,10 @@ CURRENT_CONSTRAINT_FILE="${CONSTRAINTS_DIR}/constraints-${PYTHON_MAJOR_MINOR_VER
mkdir -pv "${CONSTRAINTS_DIR}"
-curl "${AIRFLOW_CONSTRAINTS_LOCATION}" --output "${LATEST_CONSTRAINT_FILE}"
+CONSTRAINTS_LOCATION="https://raw.githubusercontent.com/apache/airflow/${DEFAULT_CONSTRAINTS_BRANCH}/constraints-${PYTHON_MAJOR_MINOR_VERSION}.txt"
+readonly CONSTRAINTS_LOCATION
+
+curl "${CONSTRAINTS_LOCATION}" --output "${LATEST_CONSTRAINT_FILE}"
echo
echo "Freezing constraints to ${CURRENT_CONSTRAINT_FILE}"
[airflow] 01/16: Add Airflow 1.10.12 to breeze-complete (#10592)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit c56d91cf76af773da220aa78678ec40492f5de3f
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Thu Aug 27 09:10:08 2020 +0100
Add Airflow 1.10.12 to breeze-complete (#10592)
(cherry picked from commit bb3e45c2509bba760f720d60b008f2f71e1024e2)
---
BREEZE.rst | 2 +-
breeze-complete | 1 -
2 files changed, 1 insertion(+), 2 deletions(-)
diff --git a/BREEZE.rst b/BREEZE.rst
index 6bf6f7e..cd87a73 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -1814,7 +1814,7 @@ This is the current syntax for `./breeze <./breeze>`_:
Run selected static checks for currently changed files. You should specify static check that
you would like to run or 'all' to run all checks. One of:
- all airflow-config-yaml base-operator bats-tests bats-in-container-tests black build
+ all airflow-config-yaml base-operator bats-tests bats-in-container-tests build
check-apache-license check-builtin-literals check-executables-have-shebangs
check-hooks-apply check-integrations check-merge-conflict check-xml debug-statements
detect-private-key doctoc dont-use-safe-filter end-of-file-fixer fix-encoding-pragma
diff --git a/breeze-complete b/breeze-complete
index 95675b4..8b94cef 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -69,7 +69,6 @@ airflow-config-yaml
base-operator
bats-tests
bats-in-container-tests
-black
build
check-apache-license
check-builtin-literals
[airflow] 11/16: Moves Commiter's guide to CONTRIBUTING.rst (#11314)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 4a12f175efcd752d184f21be76edb2927a98b38e
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Wed Oct 7 21:14:55 2020 +0200
Moves Commiter's guide to CONTRIBUTING.rst (#11314)
I decided to move it to CONTRIBUTING.rst as is it is an important
documentation on what policies we have agreed to as community and
also it is a great resource for the contributor to learn what are
the committer's responsibilities.
Fixes: #10179
(cherry picked from commit d404cb06dd52d391a9ef26ef700b61f04a771cd5)
---
CONTRIBUTING.rst | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index 69a4585..0ab4079 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -1253,6 +1253,34 @@ Here are a few rules that are important to keep in mind when you enter our commu
* It’s OK to express your own emotions while communicating - it helps other people to understand you
* Be considerate for feelings of others. Tell about how you feel not what you think of others
+Committer Responsibilities
+==========================
+
+Committers are more than contributors. While it's important for committers to maintain standing by
+committing code, their key role is to build and foster a healthy and active community.
+This means that committers should:
+
+* Review PRs in a timely and reliable fashion
+* They should also help to actively whittle down the PR backlog
+* Answer questions (i.e. on the dev list, in PRs, in Github Issues, slack, etc...)
+* Take on core changes/bugs/feature requests
+* Some changes are important enough that a committer needs to ensure it gets done. This is especially
+ the case if no one from the community is taking it on.
+* Improve processes and tooling
+* Refactoring code
+
+Commit Policy
+=============
+
+The following commit policy passed by a vote 8(binding FOR) to 0 against on May 27, 2016 on the dev list
+and slightly modified and consensus reached in October 2020:
+
+* Commits need a +1 vote from a committer who is not the author
+* Furthermore, do not merge a PR that regresses linting or does not pass CI tests (unless we have
+ justification such as clearly transient error).
+* When we do AIP voting, both PMC and committer +1s are considered as binding vote based on internal
+ PMC discussion.
+
Resources & Links
=================
- `Airflow’s official documentation <http://airflow.apache.org/>`__
[airflow] 02/16: Nightly tag push is not skipped in scheduled
builds (#10597)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 527e5139dbc9a2aeb527a35e624c5efaf8ba6444
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Thu Aug 27 15:40:07 2020 +0200
Nightly tag push is not skipped in scheduled builds (#10597)
With recent refactors, nightly tag was not pushed on
scheduled event because it was depending on pushing images
to github registry. Pushing images to github registry is
skipped on scheduled builds, so pushing tag was also skipped.
(cherry picked from commit e565368f2e988a06b0398b77e66859c138905ae7)
---
.github/workflows/ci.yml | 35 +++++++++++++++++++++++++++++++++++
1 file changed, 35 insertions(+)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 46cdea3..b57fdbd 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -725,3 +725,38 @@ jobs:
github_token: ${{ secrets.GITHUB_TOKEN }}
branch: ${{ steps.constraints-branch.outputs.branch }}
directory: "repo"
+
+ tag-repo-nightly:
+ timeout-minutes: 10
+ name: "Tag repo nightly"
+ runs-on: ubuntu-latest
+ needs:
+ - docs
+ - docs-spell-check
+ - helm-tests
+ - static-checks
+ - static-checks-pylint
+ - tests-sqlite
+ - tests-postgres
+ - tests-mysql
+ - tests-kubernetes
+ - constraints-push
+ - prepare-backport-packages
+ if: github.event_name == 'schedule' && github.repository == 'apache/airflow'
+ steps:
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ uses: actions/checkout@v2
+ - name: "Free space"
+ run: ./scripts/ci/tools/ci_free_space_on_ci.sh
+ - name: "Tag commit"
+ run: |
+ BRANCH_NAME=$(echo "${{ github.ref }}" | sed 's/refs\/heads\///')
+ echo "Tagging ${BRANCH_NAME}"
+ git tag -f nightly-${BRANCH_NAME} HEAD
+ - name: "Push tags"
+ uses: ad-m/github-push-action@master
+ with:
+ github_token: ${{ secrets.GITHUB_TOKEN }}
+ tags: true
+ force: true
+ branch: master
[airflow] 14/16: Fix command to run tmux with breeze in BREEZE.rst
(#11340)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit fde0151c1a99d5a41046ed507153553fcc29aac3
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Thu Oct 8 16:47:56 2020 +0100
Fix command to run tmux with breeze in BREEZE.rst (#11340)
`breeze --start-airflow` -> `breeze start-airflow`
(cherry picked from commit ba60836456795c144a784612ba7549bb13f8c92b)
---
BREEZE.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/BREEZE.rst b/BREEZE.rst
index 3002f05..5409792 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -337,7 +337,7 @@ You can optionally reset database if specified as extra ``--db-reset`` flag and
start integrations (separate Docker images) if specified as extra ``--integration`` flags. You can also
chose which backend database should be used with ``--backend`` flag and python version with ``--python`` flag.
-You can also have breeze launch Airflow automatically ``breeze --start-airflow``, this will drop you in a
+You can also have breeze launch Airflow automatically ``breeze start-airflow``, this will drop you in a
tmux session with three panes (one to monitor the scheduler, one for the webserver and one with a shell
for additional commands.
[airflow] 10/16: Add remaining community guidelines to
CONTRIBUTING.rst (#11312)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6f86f983904064133d9612986cd940b295bb8ea2
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Wed Oct 7 05:33:47 2020 +0200
Add remaining community guidelines to CONTRIBUTING.rst (#11312)
We are cleaning up the docs from CWiki and this is what's left of
community guidelines that were maintained there.
Fixes #10181
(cherry picked from commit 18dcac8a01d10beb0fce0a34b432e25d10489214)
---
CONTRIBUTING.rst | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index 85558f3..69a4585 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -359,6 +359,11 @@ Step 4: Prepare PR
If you are creating ``example_dags`` directory, you need to create ``example_dags/__init__.py`` with Apache license or copy another ``__init__.py`` file that contains the necessary license.
+You need to have review of at least one committer (if you are committer yourself, it has to be
+another committer). Ideally you should have 2 or more committers reviewing the code that touches
+the core of Airflow.
+
+
Pull Request Guidelines
=======================
[airflow] 15/16: Split tests to more sub-types (#11402)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 8878998cf6081e45e4af6954d3f8294d78642285
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sun Oct 11 16:40:31 2020 +0200
Split tests to more sub-types (#11402)
We seem to have a problem with running all tests at once - most
likely due to some resource problems in our CI, therefore it makes
sense to split the tests into more batches. This is not yet full
implementation of selective tests but it is going in this direction
by splitting to Core/Providers/API/CLI tests. The full selective
tests approach will be implemented as part of #10507 issue.
This split is possible thanks to #10422 which moved building image
to a separate workflow - this way each image is only built once
and it is uploaded to a shared registry, where it is quickly
downloaded from rather than built by all the jobs separately - this
way we can have many more jobs as there is very little per-job
overhead before the tests start runnning.
(cherry picked from commit 5bc5994c2c1a8f73a644e29e98d3d132c6097ab2)
---
.github/workflows/ci.yml | 27 +++-
BREEZE.rst | 17 ++-
TESTING.rst | 57 +++++++-
breeze | 47 ++++++-
breeze-complete | 12 +-
scripts/ci/docker-compose/base.yml | 7 +-
scripts/ci/libraries/_docker.env | 4 +
scripts/ci/libraries/_initialization.sh | 42 +++---
scripts/ci/libraries/_parameters.sh | 28 ++--
scripts/ci/libraries/_push_pull_remove_images.sh | 14 ++
scripts/ci/testing/ci_run_airflow_testing.sh | 11 +-
scripts/in_container/entrypoint_ci.sh | 151 ++++++++++++++++-----
scripts/in_container/run_ci_tests.sh | 59 ++++++--
tests/bats/test_breeze_complete.bats | 8 ++
tests/{ => core}/test_config_templates.py | 0
tests/{ => core}/test_configuration.py | 0
tests/{ => core}/test_core.py | 0
tests/{ => core}/test_example_dags_system.py | 0
.../test_impersonation_tests.py} | 0
tests/{ => core}/test_logging_config.py | 0
tests/{ => core}/test_sentry.py | 0
tests/{ => core}/test_sqlalchemy_config.py | 3 +-
tests/jobs/test_local_task_job.py | 2 +-
.../task/task_runner/test_standard_task_runner.py | 6 +-
24 files changed, 390 insertions(+), 105 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 5e14f56..8c42d33 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -43,7 +43,7 @@ env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ github.run_id }}"
GITHUB_REGISTRY_PUSH_IMAGE_TAG: "latest"
- TEST_TYPES: '["Core", "Integration", "Heisentests"]'
+ TEST_TYPES: '["Core", "Providers", "API", "CLI", "Integration", "Other", "WWW", "Heisentests"]'
# You can switch between building the image in "Build Images" workflow or building them in CI workflow
# Separately for each job.
@@ -79,7 +79,7 @@ jobs:
postgresVersions: ${{ steps.versions.outputs.postgres-versions }}
defaultPostgresVersion: ${{ steps.versions.outputs.default-postgres-version }}
mysqlVersions: ${{ steps.versions.outputs.mysql-versions }}
- defaultMysqlVersion: ${{ steps.versions.outputs.default-mysql-version }}
+ defaultMySQLVersion: ${{ steps.versions.outputs.default-mysql-version }}
helmVersions: ${{ steps.versions.outputs.helm-versions }}
defaultHelmVersion: ${{ steps.versions.outputs.default-helm-version }}
kindVersions: ${{ steps.versions.outputs.kind-versions }}
@@ -274,6 +274,12 @@ jobs:
postgres-version: ${{ fromJson(needs.build-info.outputs.postgresVersions) }}
test-type: ${{ fromJson(needs.build-info.outputs.testTypes) }}
exclude: ${{ fromJson(needs.build-info.outputs.postgresExclude) }}
+ # Additionally include all postgres-only tests but only for default versions
+ # in case the regular tests are excluded in the future
+ include:
+ - python_version: ${{ needs.build-info.outputs.defaultPythonVersion }}
+ postgres-version: ${{ needs.build-info.outputs.defaultPostgresVersion }}
+ test-type: "Postgres"
fail-fast: false
env:
BACKEND: postgres
@@ -321,6 +327,12 @@ jobs:
mysql-version: ${{ fromJson(needs.build-info.outputs.mysqlVersions) }}
test-type: ${{ fromJson(needs.build-info.outputs.testTypes) }}
exclude: ${{ fromJson(needs.build-info.outputs.mysqlExclude) }}
+ # Additionally include all mysql-only tests but only for default versions
+ # in case the regular tests are excluded in the future
+ include:
+ - python_version: ${{ needs.build-info.outputs.defaultPythonVersion }}
+ postgres-version: ${{ needs.build-info.outputs.defaultMySQLVersion }}
+ test-type: "MySQL"
fail-fast: false
env:
BACKEND: mysql
@@ -405,9 +417,16 @@ jobs:
runs-on: ubuntu-latest
continue-on-error: true
needs: [build-info, trigger-tests, ci-images]
+ strategy:
+ matrix:
+ include:
+ - backend: mysql
+ - backend: postgres
+ - backend: sqlite
env:
- PYTHON_MAJOR_MINOR_VERSION: ${{needs.build-info.outputs.defaultPythonVersion}}
- BACKEND: postgres
+ BACKEND: ${{ matrix.backend }}
+ PYTHON_MAJOR_MINOR_VERSION: ${{ needs.build-info.outputs.defaultPythonVersion }}
+ MYSQL_VERSION: ${{needs.build-info.outputs.defaultMySQLVersion}}
POSTGRES_VERSION: ${{needs.build-info.outputs.defaultPostgresVersion}}
RUN_TESTS: true
TEST_TYPE: Quarantined
diff --git a/BREEZE.rst b/BREEZE.rst
index 5409792..3f3aca1 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -1862,7 +1862,12 @@ This is the current syntax for `./breeze <./breeze>`_:
Flags:
- Run 'breeze flags' to see all applicable flags.
+ --test-type TEST_TYPE
+ Type of the test to run. One of:
+
+ All,Core,Providers,API,CLI,Integration,Other,WWW,Heisentests,Postgres,MySQL
+
+ Default: All
####################################################################################################
@@ -2208,6 +2213,16 @@ This is the current syntax for `./breeze <./breeze>`_:
Default: latest.
****************************************************************************************************
+ Flags for running tests
+
+ --test-type TEST_TYPE
+ Type of the test to run. One of:
+
+ All,Core,Providers,API,CLI,Integration,Other,WWW,Heisentests,Postgres,MySQL
+
+ Default: All
+
+ ****************************************************************************************************
Increase verbosity of the scripts
-v, --verbose
diff --git a/TESTING.rst b/TESTING.rst
index fd8a9f7..f6b9964 100644
--- a/TESTING.rst
+++ b/TESTING.rst
@@ -131,23 +131,70 @@ Running Tests for a Specified Target Using Breeze from the Host
---------------------------------------------------------------
If you wish to only run tests and not to drop into shell, apply the
-``tests`` command. You can add extra targets and pytest flags after the ``tests`` command.
+``tests`` command. You can add extra targets and pytest flags after the ``--`` command. Note that
+often you want to run the tests with a clean/reset db, so usually you want to add ``--db-reset`` flag
+to breeze.
.. code-block:: bash
- ./breeze tests tests/hooks/test_druid_hook.py tests/tests_core.py --logging-level=DEBUG
+ ./breeze tests tests/hooks/test_druid_hook.py tests/tests_core.py --db-reset -- --logging-level=DEBUG
-You can run the whole test suite with a 'tests' test target:
+You can run the whole test suite without adding the test target:
.. code-block:: bash
- ./breeze tests tests
+ ./breeze tests --db-reset
You can also specify individual tests or a group of tests:
.. code-block:: bash
- ./breeze tests tests/test_core.py::TestCore
+ ./breeze tests --db-reset tests/test_core.py::TestCore
+
+
+Running Tests of a specified type from the Host
+-----------------------------------------------
+
+You can also run tests for a specific test type. For the stability and performance point of view
+we separated tests to different test types so that they can be run separately.
+
+You can select the test type by adding ``--test-type TEST_TYPE`` before the test command. There are two
+kinds of test types:
+
+* Per-directories types are added to select subset of the tests based on sub-directories in ``tests`` folder.
+ Example test types there - Core, Providers, CLI. The only action that happens when you choose the right
+ test folders are pre-selected. For those types of tests it is only useful to choose the test type
+ when you do not specify test to run.
+
+Runs all core tests:
+
+.. code-block:: bash
+
+ ./breeze --test-type Core --db-reset tests
+
+Runs all provider tests:
+
+.. code-block:: bash
+
+ ./breeze --test-type Providers --db-reset tests
+
+* Special kinds of tests - Integration, Heisentests, Quarantined, Postgres, MySQL which are marked with pytest
+ marks and for those you need to select the type using test-type switch. If you want to run such tests
+ using breeze, you need to pass appropriate ``--test-type`` otherwise the test will be skipped.
+ Similarly to the per-directory tests if you do not specify the test or tests to run,
+ all tests of a given type are run
+
+Run quarantined test_task_command.py test:
+
+.. code-block:: bash
+
+ ./breeze --test-type Quarantined tests tests/cli/commands/test_task_command.py --db-reset
+
+Run all Quarantined tests:
+
+.. code-block:: bash
+
+ ./breeze --test-type Quarantined tests --db-reset
Airflow Integration Tests
diff --git a/breeze b/breeze
index e7a2ca8..b4e0996 100755
--- a/breeze
+++ b/breeze
@@ -120,7 +120,7 @@ function breeze::setup_default_breeze_constants() {
# If set to true, the test connections will be created
export LOAD_DEFAULT_CONNECTIONS="false"
- # If set to true, the sample dags will be created
+ # If set to true, the sample dags will be used
export LOAD_EXAMPLES="false"
# If set to true, RBAC UI will not be used for 1.10 version
@@ -1073,6 +1073,7 @@ function breeze::parse_arguments() {
export SKIP_BUILDING_PROD_IMAGE="true"
export MOUNT_LOCAL_SOURCES="false"
export SKIP_CHECK_REMOTE_IMAGE="true"
+ export FAIL_ON_GITHUB_DOCKER_PULL_ERROR="true"
shift 2
;;
--init-script)
@@ -1105,6 +1106,12 @@ function breeze::parse_arguments() {
echo
shift
;;
+ --test-type)
+ export TEST_TYPE="${2}"
+ echo "Selected test type: ${TEST_TYPE}"
+ echo
+ shift 2
+ ;;
--)
shift
break
@@ -1255,12 +1262,8 @@ function breeze::parse_arguments() {
;;
tests)
last_subcommand="${1}"
- if [[ $# -lt 2 ]]; then
- run_help="true"
- else
- shift
- fi
command_to_run="run_tests"
+ shift
;;
toggle-suppress-cheatsheet)
last_subcommand="${1}"
@@ -1404,6 +1407,11 @@ function breeze::prepare_formatted_versions() {
FORMATTED_DEFAULT_PROD_EXTRAS=$(echo "${DEFAULT_PROD_EXTRAS=}" |
tr ',' ' ' | fold -w "${indented_screen_width}" -s | sed "s/ /,/g; s/^/${list_prefix}/")
readonly FORMATTED_DEFAULT_PROD_EXTRAS
+
+ FORMATTED_TEST_TYPES=$(echo "${_breeze_allowed_test_types=""}" |
+ tr ',' ' ' | fold -w "${indented_screen_width}" -s | sed "s/ /,/g; s/^/${list_prefix}/")
+ readonly FORMATTED_TEST_TYPES
+
}
#######################################################################################################
@@ -1737,7 +1745,7 @@ ${CMDNAME} tests [FLAGS] [TEST_TARGET ..] [-- <EXTRA_ARGS>]
'${CMDNAME} tests tests
Flags:
-$(breeze::flag_footer)
+$(breeze::flag_tests)
"
readonly DETAILED_USAGE_TESTS
export DETAILED_USAGE_TOGGLE_SUPPRESS_CHEATSHEET="
@@ -2359,6 +2367,25 @@ function breeze::flag_start_airflow() {
"
}
+#####################################################################################################
+#
+# Prints flags that control tests
+#
+# Outputs:
+# Flag information.
+#######################################################################################################
+function breeze::flag_tests() {
+ echo "
+--test-type TEST_TYPE
+ Type of the test to run. One of:
+
+${FORMATTED_TEST_TYPES}
+
+ Default: ${_breeze_default_test_type:=}
+
+"
+}
+
#######################################################################################################
#
# Prints all flags
@@ -2421,6 +2448,10 @@ $(breeze::print_star_line)
$(breeze::flag_pull_push_docker_images)
$(breeze::print_star_line)
+ Flags for running tests
+$(breeze::flag_tests)
+
+$(breeze::print_star_line)
Increase verbosity of the scripts
$(breeze::flag_verbosity)
@@ -2561,6 +2592,8 @@ function breeze::check_and_save_all_params() {
parameters::check_and_save_allowed_param "POSTGRES_VERSION" "Postgres version" "--postgres-version"
parameters::check_and_save_allowed_param "MYSQL_VERSION" "Mysql version" "--mysql-version"
+ parameters::check_allowed_param TEST_TYPE "Type of tests" "--test-type"
+
# Can't verify those - they can be anything, so let's just save them
parameters::save_to_file DOCKERHUB_USER
parameters::save_to_file DOCKERHUB_REPO
diff --git a/breeze-complete b/breeze-complete
index 2c57a79..9449f5e 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -33,6 +33,7 @@ _breeze_allowed_kind_versions="v0.8.0"
_breeze_allowed_mysql_versions="5.6 5.7"
_breeze_allowed_postgres_versions="9.6 10"
_breeze_allowed_kind_operations="start stop restart status deploy test shell"
+_breeze_allowed_test_types="All Core Providers API CLI Integration Other WWW Heisentests Postgres MySQL"
# shellcheck disable=SC2034
{
@@ -44,6 +45,7 @@ _breeze_allowed_kind_operations="start stop restart status deploy test shell"
_breeze_default_kind_version=$(echo "${_breeze_allowed_kind_versions}" | awk '{print $1}')
_breeze_default_postgres_version=$(echo "${_breeze_allowed_postgres_versions}" | awk '{print $1}')
_breeze_default_mysql_version=$(echo "${_breeze_allowed_mysql_versions}" | awk '{print $1}')
+ _breeze_default_test_type=$(echo "${_breeze_allowed_test_types}" | awk '{print $1}')
}
_breeze_allowed_install_airflow_versions=$(cat <<-EOF
@@ -141,6 +143,7 @@ dev-apt-deps: additional-dev-apt-deps: dev-apt-command: additional-dev-apt-comma
runtime-apt-deps: additional-runtime-apt-deps: runtime-apt-command: additional-runtime-apt-command: additional-runtime-apt-env:
load-default-connections load-example-dags
install-wheels no-rbac-ui
+test-type:
"
_breeze_commands="
@@ -163,7 +166,8 @@ _breeze_extra_arg_commands="
docker-compose
kind-cluster
static-check
-tests"
+tests
+"
_breeze_help_commands="
flags
@@ -248,6 +252,12 @@ function breeze_complete::get_known_values_breeze() {
kind-cluster)
_breeze_known_values="${_breeze_allowed_kind_operations}"
;;
+ tests)
+ _breeze_known_values="$(find tests -name '*.py')"
+ ;;
+ --test-type)
+ _breeze_known_values="${_breeze_allowed_test_types}"
+ ;;
*)
_breeze_known_values=""
;;
diff --git a/scripts/ci/docker-compose/base.yml b/scripts/ci/docker-compose/base.yml
index f89aefd..8a0781e 100644
--- a/scripts/ci/docker-compose/base.yml
+++ b/scripts/ci/docker-compose/base.yml
@@ -39,9 +39,7 @@ services:
- ENABLE_KIND_CLUSTER
- ENABLED_INTEGRATIONS
- RUN_INTEGRATION_TESTS
- - ONLY_RUN_LONG_RUNNING_TESTS
- - ONLY_RUN_QUARANTINED_TESTS
- - ONLY_RUN_HEISEN_TESTS
+ - TEST_TYPE
- GITHUB_TOKEN
- GITHUB_REPOSITORY
- ISSUE_ID
@@ -64,6 +62,9 @@ services:
- HOST_OS
- PYTHONDONTWRITEBYTECODE
- INIT_SCRIPT_FILE
+ - GITHUB_REGISTRY_PULL_IMAGE_TAG
+ - POSTGRES_VERSION
+ - MYSQL_VERSION
volumes:
# Pass docker to inside of the container so that Kind and Moto tests can use it.
- /var/run/docker.sock:/var/run/docker.sock
diff --git a/scripts/ci/libraries/_docker.env b/scripts/ci/libraries/_docker.env
index 986b7b4..86461d0 100644
--- a/scripts/ci/libraries/_docker.env
+++ b/scripts/ci/libraries/_docker.env
@@ -24,9 +24,13 @@ HOST_OS
HOST_HOME
HOST_AIRFLOW_SOURCES
PYTHON_MAJOR_MINOR_VERSION
+BACKEND
VERSION_SUFFIX_FOR_PYPI
VERSION_SUFFIX_FOR_SVN
PRINT_INFO_FROM_SCRIPTS
CI
LOAD_DEFAULT_CONNECTIONS
LOAD_EXAMPLES
+GITHUB_REGISTRY_PULL_IMAGE_TAG
+POSTGRES_VERSION
+MYSQL_VERSION
diff --git a/scripts/ci/libraries/_initialization.sh b/scripts/ci/libraries/_initialization.sh
index 576e80c..dd40a76 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -34,6 +34,12 @@ function initialization::create_directories() {
export FILES_DIR="${AIRFLOW_SOURCES}/files"
readonly FILES_DIR
+ # Create an empty .pypirc file that you can customise. It is .gitignored so it will never
+ # land in the repository - it is only added to the "build image" of production image
+ # So you can keep your credentials safe as long as you do not push the build image.
+ # The final image does not contain it.
+ touch "${AIRFLOW_SOURCES}/.pypirc"
+
# Directory where all the build cache is stored - we keep there status of all the docker images
# As well as hashes of the important files, but also we generate build scripts there that are
# Used to execute the commands for breeze
@@ -95,7 +101,7 @@ function initialization::initialize_base_variables() {
CURRENT_POSTGRES_VERSIONS+=("9.6" "10")
export CURRENT_POSTGRES_VERSIONS
- # Currently supported versions of MySQL
+ # Currently supported versions of MySQL
CURRENT_MYSQL_VERSIONS+=("5.6" "5.7")
export CURRENT_MYSQL_VERSIONS
@@ -111,8 +117,10 @@ function initialization::initialize_base_variables() {
# If set to true, the database will be initialized, a user created and webserver and scheduler started
export START_AIRFLOW=${START_AIRFLOW:="false"}
+ # If set to true, the sample dags will be used
export LOAD_EXAMPLES=${LOAD_EXAMPLES:="false"}
+ # If set to true, the test connections will be created
export LOAD_DEFAULT_CONNECTIONS=${LOAD_DEFAULT_CONNECTIONS:="false"}
# If set to true, RBAC UI will not be used for 1.10 version
@@ -126,9 +134,6 @@ function initialization::initialize_base_variables() {
# otherwise it will use files/airflow-breeze-config/init.sh
export INIT_SCRIPT_FILE=${INIT_SCRIPT_FILE:=""}
- # If set to true, RBAC mode is enabled
- export RBAC_UI=${RBAC_UI:="false"}
-
# Read airflow version from the version.py
AIRFLOW_VERSION=$(grep version "${AIRFLOW_SOURCES}/airflow/version.py" | awk '{print $3}' | sed "s/['+]//g")
export AIRFLOW_VERSION
@@ -170,7 +175,6 @@ function initialization::initialize_available_integrations() {
export AVAILABLE_INTEGRATIONS="cassandra kerberos mongo openldap presto rabbitmq redis"
}
-
# Needs to be declared outside of function for MacOS
FILES_FOR_REBUILD_CHECK=()
@@ -188,7 +192,6 @@ function initialization::initialize_files_for_rebuild_check() {
)
}
-
# Needs to be declared outside of function for MacOS
# extra flags passed to docker run for PROD image
@@ -257,6 +260,8 @@ function initialization::initialize_force_variables() {
# Can be set to true to skip if the image is newer in registry
export SKIP_CHECK_REMOTE_IMAGE=${SKIP_CHECK_REMOTE_IMAGE:="false"}
+ # Should be set to true if you expect image frm GitHub to be present and downloaded
+ export FAIL_ON_GITHUB_DOCKER_PULL_ERROR=${FAIL_ON_GITHUB_DOCKER_PULL_ERROR:="false"}
}
# Determine information about the host
@@ -425,11 +430,9 @@ function initialization::initialize_git_variables() {
function initialization::initialize_github_variables() {
# Defaults for interacting with GitHub
- export GITHUB_REPOSITORY=${GITHUB_REPOSITORY:="apache/airflow"}
- GITHUB_REPOSITORY_LOWERCASE="$(echo "${GITHUB_REPOSITORY}" |tr '[:upper:]' '[:lower:]')"
- export GITHUB_REPOSITORY_LOWERCASE
- export GITHUB_REGISTRY=${GITHUB_REGISTRY:="docker.pkg.github.com"}
export USE_GITHUB_REGISTRY=${USE_GITHUB_REGISTRY:="false"}
+
+ export GITHUB_REGISTRY=${GITHUB_REGISTRY:="docker.pkg.github.com"}
export GITHUB_REGISTRY_WAIT_FOR_IMAGE=${GITHUB_REGISTRY_WAIT_FOR_IMAGE:="false"}
export GITHUB_REGISTRY_PULL_IMAGE_TAG=${GITHUB_REGISTRY_PULL_IMAGE_TAG:="latest"}
export GITHUB_REGISTRY_PUSH_IMAGE_TAG=${GITHUB_REGISTRY_PUSH_IMAGE_TAG:="latest"}
@@ -439,8 +442,10 @@ function initialization::initialize_github_variables() {
# Used only in CI environment
export GITHUB_TOKEN="${GITHUB_TOKEN=""}"
export GITHUB_USERNAME="${GITHUB_USERNAME=""}"
+}
-
+function initialization::initialize_test_variables() {
+ export TEST_TYPE=${TEST_TYPE:="All"}
}
# Common environment that is initialized by both Breeze and CI scripts
@@ -459,6 +464,7 @@ function initialization::initialize_common_environment() {
initialization::initialize_kubernetes_variables
initialization::initialize_git_variables
initialization::initialize_github_variables
+ initialization::initialize_test_variables
}
function initialization::set_default_python_version_if_empty() {
@@ -480,7 +486,6 @@ Basic variables:
PYTHON_MAJOR_MINOR_VERSION: ${PYTHON_MAJOR_MINOR_VERSION}
DB_RESET: ${DB_RESET}
START_AIRFLOW: ${START_AIRFLOW}
- RBAC_UI: ${RBAC_UI}
DockerHub variables:
@@ -498,6 +503,7 @@ Force variables:
FORCE_BUILD_IMAGES: ${FORCE_BUILD_IMAGES}
FORCE_ANSWER_TO_QUESTIONS: ${FORCE_ANSWER_TO_QUESTIONS}
SKIP_CHECK_REMOTE_IMAGE: ${SKIP_CHECK_REMOTE_IMAGE}
+ FAIL_ON_GITHUB_DOCKER_PULL_ERROR: ${FAIL_ON_GITHUB_DOCKER_PULL_ERROR}
Host variables:
@@ -562,6 +568,10 @@ Initialization variables:
INSTALL_WHEELS: ${INSTALL_WHEELS}
DISABLE_RBAC: ${DISABLE_RBAC}
+Test variables:
+
+ TEST_TYPE: ${TEST_TYPE}
+
EOF
}
@@ -633,7 +643,6 @@ function initialization::make_constants_read_only() {
readonly POSTGRES_HOST_PORT
readonly MYSQL_HOST_PORT
-
readonly HOST_USER_ID
readonly HOST_GROUP_ID
readonly HOST_AIRFLOW_SOURCES
@@ -660,7 +669,6 @@ function initialization::make_constants_read_only() {
readonly VERBOSE
readonly START_AIRFLOW
- readonly RBAC_UI
readonly PRODUCTION_IMAGE
@@ -716,7 +724,6 @@ function initialization::make_constants_read_only() {
readonly GITHUB_TOKEN
readonly GITHUB_USERNAME
-
readonly FORWARD_CREDENTIALS
readonly USE_GITHUB_REGISTRY
@@ -740,21 +747,18 @@ function initialization::make_constants_read_only() {
}
-
# converts parameters to json array
function initialization::parameters_to_json() {
echo -n "["
local separator=""
local var
- for var in "${@}"
- do
+ for var in "${@}"; do
echo -n "${separator}\"${var}\""
separator=","
done
echo "]"
}
-
# output parameter name and value - both to stdout and to be set by GitHub Actions
function initialization::ga_output() {
echo "::set-output name=${1}::${2}"
diff --git a/scripts/ci/libraries/_parameters.sh b/scripts/ci/libraries/_parameters.sh
index 3f2f02d..d655853 100644
--- a/scripts/ci/libraries/_parameters.sh
+++ b/scripts/ci/libraries/_parameters.sh
@@ -17,21 +17,22 @@
# under the License.
# Reads environment variable passed as first parameter from the .build cache file
-function parameters::read_from_file {
+function parameters::read_from_file() {
cat "${BUILD_CACHE_DIR}/.$1" 2>/dev/null || true
}
# Saves environment variable passed as first parameter to the .build cache file
-function parameters::save_to_file {
+function parameters::save_to_file() {
# shellcheck disable=SC2005
- echo "$(eval echo "\$$1")" > "${BUILD_CACHE_DIR}/.$1"
+ echo "$(eval echo "\$$1")" >"${BUILD_CACHE_DIR}/.$1"
}
# check if parameter set for the variable is allowed (should be on the _breeze_allowed list)
-# and if it is, it saves it to .build cache file. In case the parameter is wrong, the
-# saved variable is removed (so that bad value is not used again in case it comes from there)
-# and exits with an error
-function parameters::check_and_save_allowed_param {
+# parameters:
+# $1 - name of the variable
+# $2 - descriptive name of the parameter
+# $3 - flag used to set te parameter
+function parameters::check_allowed_param() {
_VARIABLE_NAME="${1}"
_VARIABLE_DESCRIPTIVE_NAME="${2}"
_FLAG="${3}"
@@ -44,9 +45,7 @@ function parameters::check_and_save_allowed_param {
echo >&2
echo >&2 "Switch to supported value with ${_FLAG} flag."
- if [[ -n ${!_VARIABLE_NAME} && \
- -f "${BUILD_CACHE_DIR}/.${_VARIABLE_NAME}" && \
- ${!_VARIABLE_NAME} == $(cat "${BUILD_CACHE_DIR}/.${_VARIABLE_NAME}" ) ]]; then
+ if [[ -n ${!_VARIABLE_NAME} && -f "${BUILD_CACHE_DIR}/.${_VARIABLE_NAME}" && ${!_VARIABLE_NAME} == $(cat "${BUILD_CACHE_DIR}/.${_VARIABLE_NAME}") ]]; then
echo >&2
echo >&2 "Removing ${BUILD_CACHE_DIR}/.${_VARIABLE_NAME}. Next time you run it, it should be OK."
echo >&2
@@ -54,5 +53,12 @@ function parameters::check_and_save_allowed_param {
fi
exit 1
fi
- parameters::save_to_file "${_VARIABLE_NAME}"
+}
+# check if parameter set for the variable is allowed (should be on the _breeze_allowed list)
+# and if it is, it saves it to .build cache file. In case the parameter is wrong, the
+# saved variable is removed (so that bad value is not used again in case it comes from there)
+# and exits with an error
+function parameters::check_and_save_allowed_param() {
+ parameters::check_allowed_param "${@}"
+ parameters::save_to_file "${1}"
}
diff --git a/scripts/ci/libraries/_push_pull_remove_images.sh b/scripts/ci/libraries/_push_pull_remove_images.sh
index 5810303..fae0807 100644
--- a/scripts/ci/libraries/_push_pull_remove_images.sh
+++ b/scripts/ci/libraries/_push_pull_remove_images.sh
@@ -63,6 +63,20 @@ function push_pull_remove_images::pull_image_if_not_present_or_forced() {
echo
docker pull "${IMAGE_TO_PULL}"
EXIT_VALUE="$?"
+ if [[ ${EXIT_VALUE} != "0" && ${FAIL_ON_GITHUB_DOCKER_PULL_ERROR} == "true" ]]; then
+ >&2 echo
+ >&2 echo "ERROR! Exiting on docker pull error"
+ >&2 echo
+ >&2 echo "If you have authorisation problems, you might want to run:"
+ >&2 echo
+ >&2 echo "docker login ${IMAGE_TO_PULL%%\/*}"
+ >&2 echo
+ >&2 echo "You need to use generate token as the password, not your personal password."
+ >&2 echo "You can generete one at https://github.com/settings/tokens"
+ >&2 echo "Make sure to choose 'read:packages' scope".
+ >&2 echo
+ exit ${EXIT_VALUE}
+ fi
echo
return ${EXIT_VALUE}
fi
diff --git a/scripts/ci/testing/ci_run_airflow_testing.sh b/scripts/ci/testing/ci_run_airflow_testing.sh
index 8e6f8f5..5c24e48 100755
--- a/scripts/ci/testing/ci_run_airflow_testing.sh
+++ b/scripts/ci/testing/ci_run_airflow_testing.sh
@@ -27,11 +27,6 @@ ENABLED_INTEGRATIONS=${ENABLED_INTEGRATIONS:=""}
if [[ ${TEST_TYPE:=} == "Integration" ]]; then
export ENABLED_INTEGRATIONS="${AVAILABLE_INTEGRATIONS}"
export RUN_INTEGRATION_TESTS="${AVAILABLE_INTEGRATIONS}"
-elif [[ ${TEST_TYPE:=} == "Long" ]]; then
- export ONLY_RUN_LONG_RUNNING_TESTS="true"
-elif [[ ${TEST_TYPE:=} == "Quarantined" ]]; then
- export ONLY_RUN_QUARANTINED_TESTS="true"
- # Do not fail in quarantined tests
fi
for _INT in ${ENABLED_INTEGRATIONS}
@@ -81,7 +76,7 @@ function run_airflow_testing_in_docker() {
break
fi
done
- if [[ ${ONLY_RUN_QUARANTINED_TESTS:=} == "true" ]]; then
+ if [[ ${TEST_TYPE:=} == "Quarantined" ]]; then
if [[ ${exit_code} == "1" ]]; then
echo
echo "Some Quarantined tests failed. but we recorded it in an issue"
@@ -137,7 +132,3 @@ readonly RUN_INTEGRATION_TESTS
run_airflow_testing_in_docker "${@}"
-
-if [[ ${TEST_TYPE:=} == "Quarantined" ]]; then
- export ONLY_RUN_QUARANTINED_TESTS="true"
-fi
diff --git a/scripts/in_container/entrypoint_ci.sh b/scripts/in_container/entrypoint_ci.sh
index f59e58e..02ecab2 100755
--- a/scripts/in_container/entrypoint_ci.sh
+++ b/scripts/in_container/entrypoint_ci.sh
@@ -19,6 +19,15 @@ if [[ ${VERBOSE_COMMANDS:="false"} == "true" ]]; then
set -x
fi
+function disable_rbac_if_requested() {
+ if [[ ${DISABLE_RBAC:="false"} == "true" ]]; then
+ export AIRFLOW__WEBSERVER__RBAC="False"
+ else
+ export AIRFLOW__WEBSERVER__RBAC="True"
+ fi
+}
+
+
# shellcheck source=scripts/in_container/_in_container_script_init.sh
. /opt/airflow/scripts/in_container/_in_container_script_init.sh
@@ -189,17 +198,102 @@ if [[ "${GITHUB_ACTIONS}" == "true" ]]; then
"--pythonwarnings=ignore::DeprecationWarning"
"--pythonwarnings=ignore::PendingDeprecationWarning"
"--junitxml=${RESULT_LOG_FILE}"
+ # timeouts in seconds for individual tests
+ "--setup-timeout=20"
+ "--execution-timeout=60"
+ "--teardown-timeout=20"
+ # Only display summary for non-expected case
+ # f - failed
+ # E - error
+ # X - xpessed (passed even if expected to fail)
+ # The following cases are not displayed:
+ # s - skipped
+ # x - xfailed (expected to fail and failed)
+ # p - passed
+ # P - passed with output
+ "-rfEX"
)
else
- EXTRA_PYTEST_ARGS=()
+ EXTRA_PYTEST_ARGS=(
+ "-rfEX"
+ )
fi
-declare -a TESTS_TO_RUN
-TESTS_TO_RUN=("tests")
+declare -a SELECTED_TESTS CLI_TESTS API_TESTS PROVIDERS_TESTS CORE_TESTS WWW_TESTS \
+ ALL_TESTS ALL_PRESELECTED_TESTS ALL_OTHER_TESTS
+
+# Finds all directories that are not on the list of tests
+# - so that we do not skip any in the future if new directories are added
+function find_all_other_tests() {
+ local all_tests_dirs
+ all_tests_dirs=$(find "tests" -type d)
+ all_tests_dirs=$(echo "${all_tests_dirs}" | sed "/tests$/d" )
+ all_tests_dirs=$(echo "${all_tests_dirs}" | sed "/tests\/dags/d" )
+ local path
+ for path in "${ALL_PRESELECTED_TESTS[@]}"
+ do
+ escaped_path="${path//\//\\\/}"
+ all_tests_dirs=$(echo "${all_tests_dirs}" | sed "/${escaped_path}/d" )
+ done
+ for path in ${all_tests_dirs}
+ do
+ ALL_OTHER_TESTS+=("${path}")
+ done
+}
if [[ ${#@} -gt 0 && -n "$1" ]]; then
- TESTS_TO_RUN=("${@}")
+ SELECTED_TESTS=("${@}")
+else
+ CLI_TESTS=("tests/cli")
+ API_TESTS=("tests/api" "tests/api_connexion")
+ PROVIDERS_TESTS=("tests/providers")
+ CORE_TESTS=(
+ "tests/core"
+ "tests/executors"
+ "tests/jobs"
+ "tests/models"
+ "tests/serialization"
+ "tests/ti_deps"
+ "tests/utils"
+ )
+ WWW_TESTS=("tests/www")
+ ALL_TESTS=("tests")
+ ALL_PRESELECTED_TESTS=(
+ "${CLI_TESTS[@]}"
+ "${API_TESTS[@]}"
+ "${PROVIDERS_TESTS[@]}"
+ "${CORE_TESTS[@]}"
+ "${WWW_TESTS[@]}"
+ )
+
+ if [[ ${TEST_TYPE:=""} == "CLI" ]]; then
+ SELECTED_TESTS=("${CLI_TESTS[@]}")
+ elif [[ ${TEST_TYPE:=""} == "API" ]]; then
+ SELECTED_TESTS=("${API_TESTS[@]}")
+ elif [[ ${TEST_TYPE:=""} == "Providers" ]]; then
+ SELECTED_TESTS=("${PROVIDERS_TESTS[@]}")
+ elif [[ ${TEST_TYPE:=""} == "Core" ]]; then
+ SELECTED_TESTS=("${CORE_TESTS[@]}")
+ elif [[ ${TEST_TYPE:=""} == "WWW" ]]; then
+ SELECTED_TESTS=("${WWW_TESTS[@]}")
+ elif [[ ${TEST_TYPE:=""} == "Other" ]]; then
+ find_all_other_tests
+ SELECTED_TESTS=("${ALL_OTHER_TESTS[@]}")
+ elif [[ ${TEST_TYPE:=""} == "All" || ${TEST_TYPE} == "Quarantined" || \
+ ${TEST_TYPE} == "Postgres" || ${TEST_TYPE} == "MySQL" || \
+ ${TEST_TYPE} == "Heisentests" || ${TEST_TYPE} == "Long" || \
+ ${TEST_TYPE} == "Integration" ]]; then
+ SELECTED_TESTS=("${ALL_TESTS[@]}")
+ else
+ >&2 echo
+ >&2 echo "Wrong test type ${TEST_TYPE}"
+ >&2 echo
+ exit 1
+ fi
+
fi
+readonly SELECTED_TESTS CLI_TESTS API_TESTS PROVIDERS_TESTS CORE_TESTS WWW_TESTS \
+ ALL_TESTS ALL_PRESELECTED_TESTS
if [[ -n ${RUN_INTEGRATION_TESTS=} ]]; then
# Integration tests
@@ -207,43 +301,38 @@ if [[ -n ${RUN_INTEGRATION_TESTS=} ]]; then
do
EXTRA_PYTEST_ARGS+=("--integration" "${INT}")
done
- EXTRA_PYTEST_ARGS+=(
- # timeouts in seconds for individual tests
- "--setup-timeout=20"
- "--execution-timeout=60"
- "--teardown-timeout=20"
- # Do not display skipped tests
- "-rfExFpP"
- )
-
-elif [[ ${ONLY_RUN_LONG_RUNNING_TESTS:=""} == "true" ]]; then
+elif [[ ${TEST_TYPE:=""} == "Long" ]]; then
EXTRA_PYTEST_ARGS+=(
"-m" "long_running"
"--include-long-running"
- "--verbosity=1"
- "--setup-timeout=30"
- "--execution-timeout=120"
- "--teardown-timeout=30"
)
-elif [[ ${ONLY_RUN_QUARANTINED_TESTS:=""} == "true" ]]; then
+elif [[ ${TEST_TYPE:=""} == "Heisentests" ]]; then
EXTRA_PYTEST_ARGS+=(
- "-m" "quarantined"
- "--include-quarantined"
- "--verbosity=1"
- "--setup-timeout=10"
- "--execution-timeout=50"
- "--teardown-timeout=10"
+ "-m" "heisentests"
+ "--include-heisentests"
)
-else
- # Core tests
+elif [[ ${TEST_TYPE:=""} == "Postgres" ]]; then
EXTRA_PYTEST_ARGS+=(
- "--setup-timeout=10"
- "--execution-timeout=30"
- "--teardown-timeout=10"
+ "--backend"
+ "postgres"
+ )
+elif [[ ${TEST_TYPE:=""} == "MySQL" ]]; then
+ EXTRA_PYTEST_ARGS+=(
+ "--backend"
+ "mysql"
+ )
+elif [[ ${TEST_TYPE:=""} == "Quarantined" ]]; then
+ EXTRA_PYTEST_ARGS+=(
+ "-m" "quarantined"
+ "--include-quarantined"
)
fi
-ARGS=("${EXTRA_PYTEST_ARGS[@]}" "${TESTS_TO_RUN[@]}")
+echo
+echo "Running tests ${SELECTED_TESTS[*]}"
+echo
+
+ARGS=("${EXTRA_PYTEST_ARGS[@]}" "${SELECTED_TESTS[@]}")
if [[ ${RUN_SYSTEM_TESTS:="false"} == "true" ]]; then
"${IN_CONTAINER_DIR}/run_system_tests.sh" "${ARGS[@]}"
diff --git a/scripts/in_container/run_ci_tests.sh b/scripts/in_container/run_ci_tests.sh
index b505979..7494eec 100755
--- a/scripts/in_container/run_ci_tests.sh
+++ b/scripts/in_container/run_ci_tests.sh
@@ -30,12 +30,59 @@ RES=$?
set +x
if [[ "${RES}" == "0" && ${CI:="false"} == "true" ]]; then
echo "All tests successful"
- bash <(curl -s https://codecov.io/bash)
+ cp .coverage /files
+elif [[ "${RES}" != "0" ]]; then
+ EXTRA_ARGS=""
+ if [[ ${BACKEND} == "postgres" ]]; then
+ EXTRA_ARGS="--postgres-version ${POSTGRES_VERSION} "
+ elif [[ ${BACKEND} == "mysql" ]]; then
+ EXTRA_ARGS="--mysql-version ${MYSQL_VERSION} "
+ fi
+
+ >&2 echo "***********************************************************************************************"
+ >&2 echo "*"
+ >&2 echo "* ERROR! Some tests failed, unfortunately. Those might be transient errors,"
+ >&2 echo "* but usually you have to fix something."
+ >&2 echo "* See the above log for details."
+ >&2 echo "*"
+ >&2 echo "***********************************************************************************************"
+ >&2 echo "* You can easily reproduce the failed tests on your dev machine/"
+ >&2 echo "*"
+ >&2 echo "* When you have the source branch checked out locally:"
+ >&2 echo "*"
+ >&2 echo "* Run all tests:"
+ >&2 echo "*"
+ >&2 echo "* ./breeze --backend ${BACKEND} ${EXTRA_ARGS}--python ${PYTHON_MAJOR_MINOR_VERSION} --db-reset --test-type ${TEST_TYPE} tests"
+ >&2 echo "*"
+ >&2 echo "* Enter docker shell:"
+ >&2 echo "*"
+ >&2 echo "* ./breeze --backend ${BACKEND} ${EXTRA_ARGS}--python ${PYTHON_MAJOR_MINOR_VERSION} --db-reset --test-type ${TEST_TYPE} shell"
+ >&2 echo "*"
+ if [[ ${GITHUB_REGISTRY_PULL_IMAGE_TAG=} != "" ]]; then
+ >&2 echo "* When you do not have sources:"
+ >&2 echo "*"
+ >&2 echo "* Run all tests:"
+ >&2 echo "*"
+ >&2 echo "* ./breeze --github-image-id ${GITHUB_REGISTRY_PULL_IMAGE_TAG} --backend ${BACKEND} ${EXTRA_ARGS}--python ${PYTHON_MAJOR_MINOR_VERSION} --db-reset --test-type ${TEST_TYPE} tests"
+ >&2 echo "*"
+ >&2 echo "* Enter docker shell:"
+ >&2 echo "*"
+ >&2 echo "* ./breeze --github-image-id ${GITHUB_REGISTRY_PULL_IMAGE_TAG} --backend ${BACKEND} ${EXTRA_ARGS}--python ${PYTHON_MAJOR_MINOR_VERSION} --db-reset --test-type ${TEST_TYPE} shell"
+ >&2 echo "*"
+ fi
+ >&2 echo "*"
+ >&2 echo "* NOTE! Once you are in the docker shell, you can run failed test with:"
+ >&2 echo "*"
+ >&2 echo "* pytest [TEST_NAME]"
+ >&2 echo "*"
+ >&2 echo "* You can copy the test name from the output above"
+ >&2 echo "*"
+ >&2 echo "***********************************************************************************************"
fi
MAIN_GITHUB_REPOSITORY="apache/airflow"
-if [[ ${ONLY_RUN_QUARANTINED_TESTS:=} = "true" ]]; then
+if [[ ${TEST_TYPE:=} == "Quarantined" ]]; then
if [[ ${GITHUB_REPOSITORY} == "${MAIN_GITHUB_REPOSITORY}" ]]; then
if [[ ${RES} == "1" || ${RES} == "0" ]]; then
echo
@@ -47,15 +94,7 @@ if [[ ${ONLY_RUN_QUARANTINED_TESTS:=} = "true" ]]; then
echo "Pytest exited with ${RES} result. NOT Updating Quarantine Issue!"
echo
fi
- else
- echo
- echo "GitHub repository '${GITHUB_REPOSITORY}'. NOT Updating Quarantine Issue!"
- echo
fi
-else
- echo
- echo "Regular tests. NOT Updating Quarantine Issue!"
- echo
fi
if [[ ${CI:=} == "true" ]]; then
diff --git a/tests/bats/test_breeze_complete.bats b/tests/bats/test_breeze_complete.bats
index 2fee270..163db61 100644
--- a/tests/bats/test_breeze_complete.bats
+++ b/tests/bats/test_breeze_complete.bats
@@ -254,3 +254,11 @@
assert_equal "${_breeze_default_postgres_version}" "${POSTGRES_VERSION}"
}
+
+@test "Test default test type same as TEST_TYPE" {
+ load bats_utils
+ #shellcheck source=breeze-complete
+ source "${AIRFLOW_SOURCES}/breeze-complete"
+
+ assert_equal "${_breeze_default_test_type}" "${TEST_TYPE}"
+}
diff --git a/tests/test_config_templates.py b/tests/core/test_config_templates.py
similarity index 100%
rename from tests/test_config_templates.py
rename to tests/core/test_config_templates.py
diff --git a/tests/test_configuration.py b/tests/core/test_configuration.py
similarity index 100%
rename from tests/test_configuration.py
rename to tests/core/test_configuration.py
diff --git a/tests/test_core.py b/tests/core/test_core.py
similarity index 100%
rename from tests/test_core.py
rename to tests/core/test_core.py
diff --git a/tests/test_example_dags_system.py b/tests/core/test_example_dags_system.py
similarity index 100%
rename from tests/test_example_dags_system.py
rename to tests/core/test_example_dags_system.py
diff --git a/tests/test_impersonation.py b/tests/core/test_impersonation_tests.py
similarity index 100%
rename from tests/test_impersonation.py
rename to tests/core/test_impersonation_tests.py
diff --git a/tests/test_logging_config.py b/tests/core/test_logging_config.py
similarity index 100%
rename from tests/test_logging_config.py
rename to tests/core/test_logging_config.py
diff --git a/tests/test_sentry.py b/tests/core/test_sentry.py
similarity index 100%
rename from tests/test_sentry.py
rename to tests/core/test_sentry.py
diff --git a/tests/test_sqlalchemy_config.py b/tests/core/test_sqlalchemy_config.py
similarity index 96%
rename from tests/test_sqlalchemy_config.py
rename to tests/core/test_sqlalchemy_config.py
index b908ce2..6fa7ac9 100644
--- a/tests/test_sqlalchemy_config.py
+++ b/tests/core/test_sqlalchemy_config.py
@@ -76,7 +76,8 @@ class TestSqlAlchemySettings(unittest.TestCase):
mock_scoped_session,
mock_setup_event_handlers):
config = {
- ('core', 'sql_alchemy_connect_args'): 'tests.test_sqlalchemy_config.SQL_ALCHEMY_CONNECT_ARGS',
+ ('core', 'sql_alchemy_connect_args'):
+ 'tests.core.test_sqlalchemy_config.SQL_ALCHEMY_CONNECT_ARGS',
('core', 'sql_alchemy_pool_enabled'): 'False'
}
with conf_vars(config):
diff --git a/tests/jobs/test_local_task_job.py b/tests/jobs/test_local_task_job.py
index 0955825..72e3628 100644
--- a/tests/jobs/test_local_task_job.py
+++ b/tests/jobs/test_local_task_job.py
@@ -165,7 +165,7 @@ class LocalTaskJobTest(unittest.TestCase):
delta = (time2 - time1).total_seconds()
self.assertAlmostEqual(delta, job.heartrate, delta=0.05)
- @pytest.mark.xfail(condition=True, reason="This test might be flaky in postgres/mysql")
+ @pytest.mark.quarantined
def test_mark_success_no_kill(self):
"""
Test that ensures that mark_success in the UI doesn't cause
diff --git a/tests/task/task_runner/test_standard_task_runner.py b/tests/task/task_runner/test_standard_task_runner.py
index 2350807..e3fceec 100644
--- a/tests/task/task_runner/test_standard_task_runner.py
+++ b/tests/task/task_runner/test_standard_task_runner.py
@@ -69,7 +69,11 @@ class TestStandardTaskRunner(unittest.TestCase):
@classmethod
def tearDownClass(cls):
- clear_db_runs()
+ try:
+ clear_db_runs()
+ except Exception: # noqa pylint: disable=broad-except
+ # It might happen that we lost connection to the server here so we need to ignore any errors here
+ pass
def test_start_and_terminate(self):
local_task_job = mock.Mock()
[airflow] 05/16: Fix typos in Dockerfile.ci (#11187)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6fffe6db50ab6e6c6219a5846c7a42928d4ef366
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Tue Sep 29 06:41:05 2020 +0100
Fix typos in Dockerfile.ci (#11187)
Fixed some spellings
(cherry picked from commit 2ec12474ff4a303fa74c055394cea3193003fbdf)
---
Dockerfile.ci | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 122c5e2..f6353ff 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -39,7 +39,7 @@ ENV DEBIAN_FRONTEND=noninteractive LANGUAGE=C.UTF-8 LANG=C.UTF-8 LC_ALL=C.UTF-8
# By increasing this number we can do force build of all dependencies
ARG DEPENDENCIES_EPOCH_NUMBER="4"
-# Increase the value below to force renstalling of all dependencies
+# Increase the value below to force reinstalling of all dependencies
ENV DEPENDENCIES_EPOCH_NUMBER=${DEPENDENCIES_EPOCH_NUMBER}
# Install curl and gnupg2 - needed to download nodejs in the next step
@@ -246,7 +246,7 @@ ARG AIRFLOW_CONSTRAINTS_REFERENCE="constraints-master"
ARG AIRFLOW_CONSTRAINTS_URL="https://raw.githubusercontent.com/apache/airflow/${AIRFLOW_CONSTRAINTS_REFERENCE}/constraints-${PYTHON_MAJOR_MINOR_VERSION}.txt"
ENV AIRFLOW_CONSTRAINTS_URL=${AIRFLOW_CONSTRAINTS_URL}
-# By changing the CI build epoch we can force reinstalling Arflow from the current master
+# By changing the CI build epoch we can force reinstalling Airflow from the current master
# It can also be overwritten manually by setting the AIRFLOW_CI_BUILD_EPOCH environment variable.
ARG AIRFLOW_CI_BUILD_EPOCH="1"
ENV AIRFLOW_CI_BUILD_EPOCH=${AIRFLOW_CI_BUILD_EPOCH}
@@ -326,7 +326,7 @@ COPY . ${AIRFLOW_SOURCES}/
# Install autocomplete for airflow
RUN register-python-argcomplete airflow >> ~/.bashrc
-# Install autocomplete for Kubeclt
+# Install autocomplete for Kubectl
RUN echo "source /etc/bash_completion" >> ~/.bashrc
WORKDIR ${AIRFLOW_SOURCES}
[airflow] 06/16: Fix typo in command in CI.rst (#11233)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 1021d05f71bc138a994162d97c9510871b41be12
Author: Tobiasz Kędzierski <to...@polidea.com>
AuthorDate: Fri Oct 2 10:50:28 2020 +0200
Fix typo in command in CI.rst (#11233)
(cherry picked from commit 18f6cf138b3db50f54dfeedf10b176fe94fe355b)
---
CI.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/CI.rst b/CI.rst
index 4aa82f8..ab726a8 100644
--- a/CI.rst
+++ b/CI.rst
@@ -678,7 +678,7 @@ you need to reproduce a MySQL environment with kerberos integration enabled for
.. code-block:: bash
- ./breeze --github-image-id:210056909 --python 3.8 --integration kerberos
+ ./breeze --github-image-id 210056909 --python 3.8 --integration kerberos
You will be dropped into a shell with the exact version that was used during the CI run and you will
be able to run pytest tests manually, easily reproducing the environment that was used in CI. Note that in
[airflow] 03/16: Update node installation cmd (#10744)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 5ac7fd251b61557a4c2ca8c98055ba6924e59849
Author: Ryan Yuan <ry...@outlook.com>
AuthorDate: Sat Sep 5 16:50:29 2020 +1000
Update node installation cmd (#10744)
(cherry picked from commit d84b62d7e17dd559041754634bf299274f54e83f)
---
CONTRIBUTING.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index 309022c..d1bc4ca 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -813,7 +813,7 @@ To install yarn on macOS:
.. code-block:: bash
- brew install node --without-npm
+ brew install node
brew install yarn
yarn config set prefix ~/.yarn
[airflow] 07/16: Add Python version to Breeze cmd (#11228)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 5f37463dced322b02d5b2a109acd26b468020574
Author: Patrick Cando <32...@users.noreply.github.com>
AuthorDate: Fri Oct 2 10:04:19 2020 +0100
Add Python version to Breeze cmd (#11228)
(cherry picked from commit e37dfc8588448c2fb1e9c77d67462f06cd10039a)
---
CONTRIBUTING.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index d1bc4ca..7af8813 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -280,7 +280,7 @@ To generate a local version:
.. code-block:: bash
- ./breeze initialize-local-virtualenv
+ ./breeze initialize-local-virtualenv --python 3.6
6. Open your IDE (for example, PyCharm) and select the virtualenv you created
as the project's default virtualenv in your IDE.