You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2020/06/09 09:04:13 UTC

[airflow] branch v1-10-test updated (74d6d1d -> 4b8fbf2)

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git.


    from 74d6d1d  Add note about using dag_run.conf in BashOperator (#9143)
     new 863bed7  Fix --forward-credentials flag in Breeze (#8554)
     new 0d8a238  Fixed optimistions of non-py-code builds (#8601)
     new ab10f48  Fix the process of requirements generations (#8648)
     new f8bf546  Fixed test-target command (#8795)
     new af6aed4  Add comments to breeze scripts (#8797)
     new ff5b33e  Useful help information in test-target and docker-compose commands (#8796)
     new ffbde53  The librabbitmq library stopped installing for python3.7 (#8853)
     new 8d8d03e  Use Debian's provided JRE from Buster (#8919)
     new 431b52a  Re-run all tests when Dockerfile or Github worflow change (#8924)
     new 519401b  Hive/Hadoop minicluster needs JDK8 and JAVA_HOME to work (#8938)
     new 40d650b  Python base images are stored in cache (#8943)
     new 51a215c  Add ADDITIONAL_PYTHON_DEPS (#9031)
     new ab5e2ea  Add ADDITIONAL_AIRFLOW_EXTRAS (#9032)
     new 62c97ed  Additional python extras and deps can be set in breeze (#9035)
     new daca507  detect incompatible docker server version in breeze (#9042)
     new 374a3e8  Adds hive as extra in pyhive (#9075)
     new 4c6a82a  Prevents failure on fixing permissions for files with space in it (#9076)
     new 64194ee  Enable configurable git sync depth  (#9094)
     new b66a706  Don't reuse MY_DIR in breeze to mean different folder from ci/_utils.sh (#9098)
     new eed7d33  You can push with Breeze as separate command and to cache (#8976)
     new fbe1fe3  Produce less verbose output when building docker mount options (#9103)
     new 060cced  Display docs errors summary (#8392)
     new d4a00b7  Remove Hive/Hadoop/Java dependency from unit tests (#9029)
     new 8dacc27  Kubernetes Cluster is started on host not in the container (#8265)
     new c34cd6d  Fixes a bug where `build-image` command did not calculate md5 (#9130)
     new 945b0e6  Fix INTEGRATIONS[*]: unbound variable error in breeze (#9135)
     new 7236616  Cope with multiple processes get_remote_image_info in parallel (#9105)
     new 4e7c48e  Remove remnant kubernetes stuff from breeze scripts (#9138)
     new 86eeb00  Restrict google-cloud-texttospeach to <v2 (#9137)
     new 410bcd4  Use static binary linked docker client in CI image (#9126)
     new f16cae3  Remove vendored nvd3 and slugify libraries (#9136)
     new c0534d8  Replaces cloud-provider CLIs in CI image with scripts running containers (#9129)
     new a4dfb98  Split utils sh (#9132)
     new ec12334  Add PR/issue note in Contribution Workflow Example (#9177)
     new e5162b0  Fixes failure of the build scripts when remote repo does not exist (#9188)
     new 4b8fbf2  Improved cloud tool available in the trimmed down CI container (#9167)

The 36 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .coveragerc                                        |    1 -
 .flake8                                            |    2 +-
 .github/workflows/ci.yml                           |   69 +-
 .gitignore                                         |    4 +-
 .pre-commit-config.yaml                            |   40 +-
 .rat-excludes                                      |    5 +-
 BREEZE.rst                                         |  445 ++++--
 CI.rst                                             |   33 +-
 CONTRIBUTING.rst                                   |   10 +-
 Dockerfile                                         |   10 +-
 Dockerfile.ci                                      |  120 +-
 IMAGES.rst                                         |   27 +-
 LICENSE                                            |    2 +-
 MANIFEST.in                                        |    1 -
 README.md                                          |    2 +
 TESTING.rst                                        |  241 +++-
 airflow/_vendor/README                             |   13 -
 airflow/_vendor/__init__.py                        |   18 -
 airflow/_vendor/nvd3/LICENSE                       |   24 -
 airflow/_vendor/nvd3/NVD3Chart.py                  |  506 -------
 airflow/_vendor/nvd3/__init__.py                   |   29 -
 airflow/_vendor/nvd3/cumulativeLineChart.py        |  104 --
 airflow/_vendor/nvd3/discreteBarChart.py           |   91 --
 airflow/_vendor/nvd3/ipynb.py                      |   91 --
 airflow/_vendor/nvd3/lineChart.py                  |  120 --
 airflow/_vendor/nvd3/linePlusBarChart.py           |  131 --
 airflow/_vendor/nvd3/lineWithFocusChart.py         |  105 --
 airflow/_vendor/nvd3/multiBarChart.py              |   95 --
 airflow/_vendor/nvd3/multiBarHorizontalChart.py    |  100 --
 airflow/_vendor/nvd3/pieChart.py                   |  101 --
 airflow/_vendor/nvd3/scatterChart.py               |  121 --
 airflow/_vendor/nvd3/stackedAreaChart.py           |   99 --
 airflow/_vendor/nvd3/templates/base.html           |   35 -
 airflow/_vendor/nvd3/templates/content.html        |  123 --
 .../nvd3/templates/cumulativelinechart.html        |   10 -
 .../_vendor/nvd3/templates/discretebarchart.html   |   31 -
 .../_vendor/nvd3/templates/linebarwfocuschart.html |   60 -
 airflow/_vendor/nvd3/templates/linechart.html      |   46 -
 .../_vendor/nvd3/templates/lineplusbarchart.html   |   43 -
 .../_vendor/nvd3/templates/linewfocuschart.html    |   10 -
 airflow/_vendor/nvd3/templates/multibarchart.html  |   10 -
 .../nvd3/templates/multibarcharthorizontal.html    |   10 -
 airflow/_vendor/nvd3/templates/page.html           |   12 -
 airflow/_vendor/nvd3/templates/piechart.html       |   80 -
 airflow/_vendor/nvd3/templates/scatterchart.html   |   52 -
 .../_vendor/nvd3/templates/stackedareachart.html   |    7 -
 airflow/_vendor/nvd3/translator.py                 |   71 -
 airflow/_vendor/slugify/LICENSE                    |   21 -
 airflow/_vendor/slugify/__init__.py                |    6 -
 airflow/_vendor/slugify/slugify.py                 |  185 ---
 airflow/config_templates/config.yml                |    8 +
 airflow/config_templates/default_airflow.cfg       |    4 +
 airflow/executors/kubernetes_executor.py           |    2 +
 airflow/hooks/hive_hooks.py                        |    4 +-
 airflow/kubernetes/refresh_config.py               |    1 -
 airflow/kubernetes/volume_mount.py                 |   36 +-
 airflow/kubernetes/worker_configuration.py         |    2 +-
 airflow/operators/hive_operator.py                 |   15 +
 airflow/operators/hive_to_samba_operator.py        |   14 +-
 airflow/www/views.py                               |    7 +-
 breeze                                             |  629 +++++---
 breeze-complete                                    |   61 +-
 docs/Makefile                                      |  194 ---
 docs/build                                         |  265 ++++
 docs/build.sh                                      |  108 --
 .../kubernetes => kubernetes_tests}/__init__.py    |    0
 .../test_kubernetes_executor.py                    |  101 +-
 .../test_kubernetes_pod_operator.py                |  432 ++++--
 requirements/requirements-python3.6.txt            |    4 +-
 requirements/requirements-python3.7.txt            |    4 +-
 scripts/ci/_all_libs.sh                            |   49 +
 scripts/ci/_script_init.sh                         |    9 +-
 scripts/ci/_utils.sh                               | 1525 --------------------
 scripts/ci/ci_build_dockerhub.sh                   |   33 +-
 scripts/ci/ci_count_changed_files.sh               |    6 +-
 ...ic_checks.sh => ci_deploy_app_to_kubernetes.sh} |   35 +-
 scripts/ci/ci_docs.sh                              |    7 +-
 scripts/ci/ci_fix_ownership.sh                     |   12 +
 scripts/ci/ci_flake8.sh                            |    6 +
 ...oduction_images.sh => ci_load_image_to_kind.sh} |   17 +-
 scripts/ci/ci_mypy.sh                              |    3 +
 ...ges.sh => ci_perform_kind_cluster_operation.sh} |    7 +-
 scripts/ci/ci_run_airflow_testing.sh               |   39 +-
 scripts/ci/ci_run_kubernetes_tests.sh              |  105 ++
 scripts/ci/docker-compose/backend-sqlite.yml       |    2 +-
 scripts/ci/docker-compose/base.yml                 |    1 -
 scripts/ci/docker-compose/forward-credentials.yml  |    9 +-
 scripts/ci/docker-compose/local-prod.yml           |    3 +
 scripts/ci/docker-compose/local.yml                |   10 +-
 scripts/ci/docker-compose/runtime-kubernetes.yml   |   25 -
 scripts/ci/in_container/_in_container_utils.sh     |  101 +-
 scripts/ci/in_container/check_environment.sh       |   61 +-
 scripts/ci/in_container/entrypoint_ci.sh           |   85 +-
 .../ci/in_container/kubernetes/app/deploy_app.sh   |  210 ---
 .../app/templates/init_git_sync.template.yaml      |   36 -
 .../kubernetes/docker/airflow-test-env-init.sh     |   23 -
 .../ci/in_container/kubernetes/docker/bootstrap.sh |   27 -
 .../kubernetes/docker/rebuild_airflow_image.sh     |   74 -
 .../in_container/kubernetes/setup_kind_cluster.sh  |  187 ---
 scripts/ci/in_container/run_ci_tests.sh            |    7 +-
 scripts/ci/in_container/run_docs_build.sh          |    2 +-
 .../ci/in_container/run_generate_requirements.sh   |   39 +-
 scripts/ci/in_container/run_system_tests.sh        |    1 -
 .../kubernetes/app/postgres.yaml                   |    0
 .../{in_container => }/kubernetes/app/secrets.yaml |    0
 .../kubernetes/app/templates/airflow.template.yaml |   66 +-
 .../app/templates/configmaps.template.yaml         |    1 +
 .../app/templates/init_git_sync.template.yaml      |   40 +-
 .../{in_container => }/kubernetes/app/volumes.yaml |    0
 .../docker/airflow-test-env-init-dags.sh}          |   19 +-
 .../kubernetes/docker/airflow-test-env-init-db.sh  |   26 +-
 scripts/ci/kubernetes/docker/bootstrap.sh          |   74 +
 .../kubernetes/kind-cluster-conf.yaml              |    0
 scripts/ci/libraries/_build_images.sh              |  771 ++++++++++
 scripts/ci/libraries/_initialization.sh            |  235 +++
 scripts/ci/libraries/_kind.sh                      |  450 ++++++
 scripts/ci/libraries/_local_mounts.sh              |   66 +
 scripts/ci/libraries/_md5sum.sh                    |  137 ++
 scripts/ci/libraries/_parameters.sh                |   58 +
 scripts/ci/libraries/_permissions.sh               |   59 +
 scripts/ci/libraries/_push_pull_remove_images.sh   |  183 +++
 scripts/ci/{ci_docs.sh => libraries/_runs.sh}      |   39 +-
 scripts/ci/libraries/_sanity_checks.sh             |  166 +++
 scripts/ci/libraries/_spinner.sh                   |   53 +
 scripts/ci/libraries/_start_end.sh                 |   79 +
 .../{ci_bat_tests.sh => libraries/_verbosity.sh}   |   29 +-
 scripts/ci/pre_commit_local_yml_mounts.sh          |   32 +-
 scripts/ci/prepare_tool_scripts.sh                 |   64 +
 setup.cfg                                          |   28 +-
 setup.py                                           |   11 +-
 tests/bats/bats_utils.bash                         |    4 +-
 tests/bats/test_local_mounts.bats                  |   79 +-
 tests/conftest.py                                  |   37 -
 tests/contrib/operators/test_hive_stats.py         |  340 +++++
 tests/contrib/operators/test_hive_to_mysql.py      |  154 ++
 tests/contrib/operators/test_hive_to_samba.py      |   88 ++
 .../sensors/test_hive_partition.py}                |   39 +-
 tests/contrib/sensors/test_metastore_partition.py  |   43 +
 tests/contrib/sensors/test_webhdfs.py              |   42 +
 .../test_snowflake_operator.py => hive.py}         |   50 +-
 tests/hooks/test_hive_hook.py                      |  613 +++++---
 tests/operators/test_hive_operator.py              |  411 +++---
 tests/operators/test_operators.py                  |  249 +++-
 .../kubernetes/operators/test_kubernetes_pod.py    |  134 ++
 tests/runtime/__init__.py                          |   18 -
 tests/sensors/test_named_hive_partition_sensor.py  |  107 +-
 tests/test_utils/mock_hooks.py                     |   96 ++
 tests/test_utils/mock_operators.py                 |    9 +
 tests/test_utils/mock_process.py                   |   87 ++
 149 files changed, 6662 insertions(+), 6843 deletions(-)
 delete mode 100644 airflow/_vendor/README
 delete mode 100644 airflow/_vendor/__init__.py
 delete mode 100644 airflow/_vendor/nvd3/LICENSE
 delete mode 100644 airflow/_vendor/nvd3/NVD3Chart.py
 delete mode 100755 airflow/_vendor/nvd3/__init__.py
 delete mode 100644 airflow/_vendor/nvd3/cumulativeLineChart.py
 delete mode 100644 airflow/_vendor/nvd3/discreteBarChart.py
 delete mode 100644 airflow/_vendor/nvd3/ipynb.py
 delete mode 100644 airflow/_vendor/nvd3/lineChart.py
 delete mode 100644 airflow/_vendor/nvd3/linePlusBarChart.py
 delete mode 100644 airflow/_vendor/nvd3/lineWithFocusChart.py
 delete mode 100644 airflow/_vendor/nvd3/multiBarChart.py
 delete mode 100644 airflow/_vendor/nvd3/multiBarHorizontalChart.py
 delete mode 100644 airflow/_vendor/nvd3/pieChart.py
 delete mode 100644 airflow/_vendor/nvd3/scatterChart.py
 delete mode 100644 airflow/_vendor/nvd3/stackedAreaChart.py
 delete mode 100644 airflow/_vendor/nvd3/templates/base.html
 delete mode 100644 airflow/_vendor/nvd3/templates/content.html
 delete mode 100644 airflow/_vendor/nvd3/templates/cumulativelinechart.html
 delete mode 100644 airflow/_vendor/nvd3/templates/discretebarchart.html
 delete mode 100644 airflow/_vendor/nvd3/templates/linebarwfocuschart.html
 delete mode 100644 airflow/_vendor/nvd3/templates/linechart.html
 delete mode 100644 airflow/_vendor/nvd3/templates/lineplusbarchart.html
 delete mode 100644 airflow/_vendor/nvd3/templates/linewfocuschart.html
 delete mode 100644 airflow/_vendor/nvd3/templates/multibarchart.html
 delete mode 100644 airflow/_vendor/nvd3/templates/multibarcharthorizontal.html
 delete mode 100644 airflow/_vendor/nvd3/templates/page.html
 delete mode 100644 airflow/_vendor/nvd3/templates/piechart.html
 delete mode 100644 airflow/_vendor/nvd3/templates/scatterchart.html
 delete mode 100644 airflow/_vendor/nvd3/templates/stackedareachart.html
 delete mode 100644 airflow/_vendor/nvd3/translator.py
 delete mode 100644 airflow/_vendor/slugify/LICENSE
 delete mode 100644 airflow/_vendor/slugify/__init__.py
 delete mode 100644 airflow/_vendor/slugify/slugify.py
 delete mode 100644 docs/Makefile
 create mode 100755 docs/build
 delete mode 100755 docs/build.sh
 rename {tests/runtime/kubernetes => kubernetes_tests}/__init__.py (100%)
 rename {tests/runtime/kubernetes => kubernetes_tests}/test_kubernetes_executor.py (71%)
 rename {tests/runtime/kubernetes => kubernetes_tests}/test_kubernetes_pod_operator.py (64%)
 create mode 100755 scripts/ci/_all_libs.sh
 delete mode 100644 scripts/ci/_utils.sh
 copy scripts/ci/{ci_run_static_checks.sh => ci_deploy_app_to_kubernetes.sh} (61%)
 copy scripts/ci/{ci_push_production_images.sh => ci_load_image_to_kind.sh} (61%)
 copy scripts/ci/{ci_build_production_images.sh => ci_perform_kind_cluster_operation.sh} (89%)
 create mode 100755 scripts/ci/ci_run_kubernetes_tests.sh
 delete mode 100644 scripts/ci/docker-compose/runtime-kubernetes.yml
 delete mode 100755 scripts/ci/in_container/kubernetes/app/deploy_app.sh
 delete mode 100644 scripts/ci/in_container/kubernetes/app/templates/init_git_sync.template.yaml
 delete mode 100755 scripts/ci/in_container/kubernetes/docker/airflow-test-env-init.sh
 delete mode 100755 scripts/ci/in_container/kubernetes/docker/bootstrap.sh
 delete mode 100755 scripts/ci/in_container/kubernetes/docker/rebuild_airflow_image.sh
 delete mode 100755 scripts/ci/in_container/kubernetes/setup_kind_cluster.sh
 rename scripts/ci/{in_container => }/kubernetes/app/postgres.yaml (100%)
 rename scripts/ci/{in_container => }/kubernetes/app/secrets.yaml (100%)
 rename scripts/ci/{in_container => }/kubernetes/app/templates/airflow.template.yaml (70%)
 rename scripts/ci/{in_container => }/kubernetes/app/templates/configmaps.template.yaml (99%)
 copy airflow/api/auth/backend/deny_all.py => scripts/ci/kubernetes/app/templates/init_git_sync.template.yaml (55%)
 rename scripts/ci/{in_container => }/kubernetes/app/volumes.yaml (100%)
 copy scripts/ci/{pre_commit_update_extras.sh => kubernetes/docker/airflow-test-env-init-dags.sh} (72%)
 copy hooks/push => scripts/ci/kubernetes/docker/airflow-test-env-init-db.sh (64%)
 create mode 100755 scripts/ci/kubernetes/docker/bootstrap.sh
 rename scripts/ci/{in_container => }/kubernetes/kind-cluster-conf.yaml (100%)
 create mode 100644 scripts/ci/libraries/_build_images.sh
 create mode 100644 scripts/ci/libraries/_initialization.sh
 create mode 100644 scripts/ci/libraries/_kind.sh
 create mode 100644 scripts/ci/libraries/_local_mounts.sh
 create mode 100644 scripts/ci/libraries/_md5sum.sh
 create mode 100644 scripts/ci/libraries/_parameters.sh
 create mode 100644 scripts/ci/libraries/_permissions.sh
 create mode 100644 scripts/ci/libraries/_push_pull_remove_images.sh
 copy scripts/ci/{ci_docs.sh => libraries/_runs.sh} (51%)
 mode change 100755 => 100644
 create mode 100644 scripts/ci/libraries/_sanity_checks.sh
 create mode 100644 scripts/ci/libraries/_spinner.sh
 create mode 100644 scripts/ci/libraries/_start_end.sh
 copy scripts/ci/{ci_bat_tests.sh => libraries/_verbosity.sh} (53%)
 mode change 100755 => 100644
 create mode 100755 scripts/ci/prepare_tool_scripts.sh
 create mode 100644 tests/contrib/operators/test_hive_stats.py
 create mode 100644 tests/contrib/operators/test_hive_to_mysql.py
 create mode 100644 tests/contrib/operators/test_hive_to_samba.py
 copy tests/{sensors/test_timedelta_sensor.py => contrib/sensors/test_hive_partition.py} (50%)
 create mode 100644 tests/contrib/sensors/test_metastore_partition.py
 create mode 100644 tests/contrib/sensors/test_webhdfs.py
 copy tests/{contrib/operators/test_snowflake_operator.py => hive.py} (50%)
 create mode 100644 tests/providers/cncf/kubernetes/operators/test_kubernetes_pod.py
 delete mode 100644 tests/runtime/__init__.py
 create mode 100644 tests/test_utils/mock_hooks.py
 create mode 100644 tests/test_utils/mock_process.py


[airflow] 36/36: Improved cloud tool available in the trimmed down CI container (#9167)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 4b8fbf2f608360ccc86cf539423e5212f5c5778c
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Tue Jun 9 09:33:16 2020 +0200

    Improved cloud tool available in the trimmed down CI container (#9167)
    
    * Improved cloud tool available in the trimmed down CI container
    
    The tools now have shebangs which make them available for
    python tools. Also /opt/airflow is now mounted from the
    host Airflow sources which makes it possible for the tools to
    copy files directly to/from the sources of Airflow.
    
    It also contains one small change for Linux users - the files
    created by docker gcloud are created with root user so in order to fix
    that the directories mounted from the host are fixed when you exit
    the tool - their ownership is changed to be owned by the host user
    
    (cherry picked from commit de9d3401f9668c703a8f0665769fd17268f607ed)
---
 BREEZE.rst                                        |  3 ++
 Dockerfile.ci                                     | 21 +-------
 breeze                                            |  1 +
 scripts/ci/ci_docs.sh                             |  3 ++
 scripts/ci/ci_fix_ownership.sh                    |  3 ++
 scripts/ci/ci_flake8.sh                           |  6 +++
 scripts/ci/ci_mypy.sh                             |  3 ++
 scripts/ci/docker-compose/forward-credentials.yml |  2 +
 scripts/ci/docker-compose/local-prod.yml          |  2 +
 scripts/ci/docker-compose/local.yml               |  2 +
 scripts/ci/in_container/_in_container_utils.sh    | 16 ++++--
 scripts/ci/libraries/_initialization.sh           |  7 +++
 scripts/ci/libraries/_runs.sh                     |  6 +++
 scripts/ci/prepare_tool_scripts.sh                | 64 +++++++++++++++++++++++
 14 files changed, 114 insertions(+), 25 deletions(-)

diff --git a/BREEZE.rst b/BREEZE.rst
index aec975c..3a1a6a0 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -243,6 +243,9 @@ it is downloaded, it will stay until you remove the downloaded images from your
 For each of those CLI credentials are taken (automatically) from the credentials you have defined in
 your ${HOME} directory on host.
 
+Those tools also have host Airflow source directory mounted in /opt/airflow path
+so you can directly transfer files to/from your airflow host sources.
+
 Those are currently installed CLIs (they are available as aliases to the docker commands):
 
 +-----------------------+----------+-------------------------------------------------+-------------------+
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 8035fa2..8051431 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -307,26 +307,7 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
         pip install ${ADDITIONAL_PYTHON_DEPS}; \
     fi
 
-RUN \
-    AWSCLI_IMAGE="amazon/aws-cli:latest" && \
-    AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest" && \
-    GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest" && \
-    echo "docker run --rm -it -v \${HOST_HOME}/.aws:/root/.aws ${AWSCLI_IMAGE} \"\$@\"" \
-        > /usr/bin/aws && \
-    echo "docker pull ${AWSCLI_IMAGE}" > /usr/bin/aws-update && \
-    echo "docker run --rm -it -v \${HOST_HOME}/.azure:/root/.azure ${AZURECLI_IMAGE} \"\$@\"" \
-        > /usr/bin/az && \
-    echo "docker pull ${AZURECLI_IMAGE}" > /usr/bin/az-update && \
-    echo "docker run --rm -it -v \${HOST_HOME}/.config:/root/.config ${GCLOUD_IMAGE} bq \"\$@\"" \
-        > /usr/bin/bq && \
-    echo "docker pull ${GCLOUD_IMAGE}" > /usr/bin/bq-update && \
-    echo "docker run --rm -it -v \${HOST_HOME}/.config:/root/.config ${GCLOUD_IMAGE} gcloud \"\$@\"" \
-        > /usr/bin/gcloud && \
-    echo "docker pull ${GCLOUD_IMAGE}" > /usr/bin/gcloud-update && \
-    echo "docker run --rm -it -v \${HOST_HOME}/.config:/root/.config ${GCLOUD_IMAGE} gsutil \"\$@\"" \
-        > /usr/bin/gsutil && \
-    echo "docker pull ${GCLOUD_IMAGE}" > /usr/bin/gsutil-update && \
-    chmod a+x /usr/bin/aws /usr/bin/az /usr/bin/bq /usr/bin/gcloud /usr/bin/gsutil
+RUN scripts/ci/prepare_tool_scripts.sh
 
 WORKDIR ${AIRFLOW_SOURCES}
 
diff --git a/breeze b/breeze
index d9cfd92..005f917 100755
--- a/breeze
+++ b/breeze
@@ -443,6 +443,7 @@ export POSTGRES_HOST_PORT="${POSTGRES_HOST_PORT}"
 export POSTGRES_VERSION="${POSTGRES_VERSION}"
 export MYSQL_HOST_PORT="${MYSQL_HOST_PORT}"
 export MYSQL_VERSION="${MYSQL_VERSION}"
+export AIRFLOW_SOURCES="${AIRFLOW_SOURCES}"
 export MYSQL_ENCODING="${MYSQL_ENCODING}"
 export AIRFLOW_CI_IMAGE="${AIRFLOW_CI_IMAGE}"
 export AIRFOW_PROD_IMAGE="${AIRFLOW_PROD_IMAGE}"
diff --git a/scripts/ci/ci_docs.sh b/scripts/ci/ci_docs.sh
index e84de9b..417f6a6 100755
--- a/scripts/ci/ci_docs.sh
+++ b/scripts/ci/ci_docs.sh
@@ -28,6 +28,9 @@ function run_docs() {
             --env VERBOSE_COMMANDS \
             --env HOST_USER_ID="$(id -ur)" \
             --env HOST_GROUP_ID="$(id -gr)" \
+            --env HOST_OS="$(uname -s)" \
+            --env HOST_HOME="${HOME}" \
+            --env HOST_AIRFLOW_SOURCES="${AIRFLOW_SOURCES}" \
             --rm \
             "${AIRFLOW_CI_IMAGE}" \
             "--" "/opt/airflow/docs/build" \
diff --git a/scripts/ci/ci_fix_ownership.sh b/scripts/ci/ci_fix_ownership.sh
index 775cc58..81f5ac1 100755
--- a/scripts/ci/ci_fix_ownership.sh
+++ b/scripts/ci/ci_fix_ownership.sh
@@ -33,11 +33,14 @@ export AIRFLOW_IMAGE=${AIRFLOW_CI_IMAGE}
 export WEBSERVER_HOST_PORT=28080
 HOST_USER_ID="$(id -ur)"
 HOST_GROUP_ID="$(id -gr)"
+HOST_OS="$(uname -s)"
 
 export HOST_USER_ID
 export HOST_GROUP_ID
+export HOST_OS
 
 docker-compose \
     -f "${MY_DIR}/docker-compose/base.yml" \
     -f "${MY_DIR}/docker-compose/local.yml" \
+    -f "${MY_DIR}/docker-compose/forward-credentials.yml" \
     run airflow /opt/airflow/scripts/ci/in_container/run_fix_ownership.sh
diff --git a/scripts/ci/ci_flake8.sh b/scripts/ci/ci_flake8.sh
index 5a2da56..fac9be4 100755
--- a/scripts/ci/ci_flake8.sh
+++ b/scripts/ci/ci_flake8.sh
@@ -31,6 +31,9 @@ function run_flake8() {
             --env VERBOSE_COMMANDS \
             --env HOST_USER_ID="$(id -ur)" \
             --env HOST_GROUP_ID="$(id -gr)" \
+            --env HOST_OS="$(uname -s)" \
+            --env HOST_HOME="${HOME}" \
+            --env HOST_AIRFLOW_SOURCES="${AIRFLOW_SOURCES}" \
             --rm \
             "${AIRFLOW_CI_IMAGE}" \
             "--" "/opt/airflow/scripts/ci/in_container/run_flake8.sh" \
@@ -43,6 +46,9 @@ function run_flake8() {
             --env VERBOSE_COMMANDS \
             --env HOST_USER_ID="$(id -ur)" \
             --env HOST_GROUP_ID="$(id -gr)" \
+            --env HOST_OS="$(uname -s)" \
+            --env HOST_HOME="${HOME}" \
+            --env HOST_AIRFLOW_SOURCES="${AIRFLOW_SOURCES}" \
             --rm \
             "${AIRFLOW_CI_IMAGE}" \
             "--" "/opt/airflow/scripts/ci/in_container/run_flake8.sh" "${FILES[@]}" \
diff --git a/scripts/ci/ci_mypy.sh b/scripts/ci/ci_mypy.sh
index 8898d30..962d2c0 100755
--- a/scripts/ci/ci_mypy.sh
+++ b/scripts/ci/ci_mypy.sh
@@ -33,6 +33,9 @@ function run_mypy() {
         --env VERBOSE_COMMANDS \
         --env HOST_USER_ID="$(id -ur)" \
         --env HOST_GROUP_ID="$(id -gr)" \
+        --env HOST_OS="$(uname -s)" \
+        --env HOST_HOME="${HOME}" \
+        --env HOST_AIRFLOW_SOURCES="${AIRFLOW_SOURCES}" \
         "-v" "${AIRFLOW_SOURCES}/.mypy_cache:/opt/airflow/.mypy_cache" \
         --rm \
         "${AIRFLOW_CI_IMAGE}" \
diff --git a/scripts/ci/docker-compose/forward-credentials.yml b/scripts/ci/docker-compose/forward-credentials.yml
index 875b1ce..0eb0dbb 100644
--- a/scripts/ci/docker-compose/forward-credentials.yml
+++ b/scripts/ci/docker-compose/forward-credentials.yml
@@ -22,6 +22,8 @@ services:
     # Useful for gcloud/aws/kubernetes etc. authorisation to be passed
     # To inside docker. Use with care - your credentials will be available to
     # Everything you install in Docker
+    # If you add it here - also add it to "in_container_fix_ownership" method in
+    # the _in_container_utils.sh file to make it friendly for Linux users
     volumes:
       - ${HOME}/.aws:/root/.aws:cached
       - ${HOME}/.azure:/root/.azure:cached
diff --git a/scripts/ci/docker-compose/local-prod.yml b/scripts/ci/docker-compose/local-prod.yml
index ae8317d..a82b4f8 100644
--- a/scripts/ci/docker-compose/local-prod.yml
+++ b/scripts/ci/docker-compose/local-prod.yml
@@ -40,4 +40,6 @@ services:
       - HOST_USER_ID
       - HOST_GROUP_ID
       - HOST_HOME=${HOME}
+      - HOST_AIRFLOW_SOURCES=${AIRFLOW_SOURCES}
+      - HOST_OS
       - PYTHONDONTWRITEBYTECODE
diff --git a/scripts/ci/docker-compose/local.yml b/scripts/ci/docker-compose/local.yml
index 9603417..3c9e40b 100644
--- a/scripts/ci/docker-compose/local.yml
+++ b/scripts/ci/docker-compose/local.yml
@@ -60,6 +60,8 @@ services:
       - HOST_USER_ID
       - HOST_GROUP_ID
       - HOST_HOME=${HOME}
+      - HOST_AIRFLOW_SOURCES=${AIRFLOW_SOURCES}
+      - HOST_OS
       - PYTHONDONTWRITEBYTECODE
     ports:
       - "${WEBSERVER_HOST_PORT}:8080"
diff --git a/scripts/ci/in_container/_in_container_utils.sh b/scripts/ci/in_container/_in_container_utils.sh
index 9197a04..5d9fbb6 100644
--- a/scripts/ci/in_container/_in_container_utils.sh
+++ b/scripts/ci/in_container/_in_container_utils.sh
@@ -89,13 +89,19 @@ function in_container_cleanup_pycache() {
 
 #
 # Fixes ownership of files generated in container - if they are owned by root, they will be owned by
-# The host user.
+# The host user. Only needed if the host is Linux - on Mac, ownership of files is automatically
+# changed to the Host user via osxfs filesystem
 #
 function in_container_fix_ownership() {
-    set +o pipefail
-    sudo find "${AIRFLOW_SOURCES}" -print0 -user root \
-    | sudo xargs --null chown -v "${HOST_USER_ID}.${HOST_GROUP_ID}" --no-dereference >/dev/null 2>&1
-    set -o pipefail
+    if [[ ${HOST_OS:=} == "Linux" ]]; then
+        set +o pipefail
+        echo "Fixing ownership of mounted files"
+        sudo find "${AIRFLOW_SOURCES}" -print0 -user root \
+        | sudo xargs --null chown "${HOST_USER_ID}.${HOST_GROUP_ID}" --no-dereference >/dev/null 2>&1
+        sudo find "/root/.aws" "/root/.azure" "/root/.config" "/root/.docker" -print0 -user root \
+        | sudo xargs --null chown "${HOST_USER_ID}.${HOST_GROUP_ID}" --no-dereference || true >/dev/null 2>&1
+        set -o pipefail
+    fi
 }
 
 function in_container_go_to_airflow_sources() {
diff --git a/scripts/ci/libraries/_initialization.sh b/scripts/ci/libraries/_initialization.sh
index 9f86ba4..c8cd9f7 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -114,6 +114,13 @@ function initialize_common_environment {
     HOST_GROUP_ID="$(id -gr)"
     export HOST_GROUP_ID
 
+    # Set host OS. This is used to set the ownership properly when exiting
+    # The container on Linux - all files created inside docker are created with root user
+    # but they should be restored back to the host user
+    HOST_OS="$(uname -s)"
+    export HOST_OS
+
+
     # Add the right volume mount for sources, depending which mount strategy is used
     if [[ ${MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS} == "true" ]]; then
         print_info
diff --git a/scripts/ci/libraries/_runs.sh b/scripts/ci/libraries/_runs.sh
index cd5a44a..76b674d 100644
--- a/scripts/ci/libraries/_runs.sh
+++ b/scripts/ci/libraries/_runs.sh
@@ -25,6 +25,9 @@ function run_docs() {
             --env VERBOSE_COMMANDS \
             --env HOST_USER_ID="$(id -ur)" \
             --env HOST_GROUP_ID="$(id -gr)" \
+            --env HOST_OS="$(uname -s)" \
+            --env HOST_HOME="${HOME}" \
+            --env HOST_AIRFLOW_SOURCES="${AIRFLOW_SOURCES}" \
             --rm \
             "${AIRFLOW_CI_IMAGE}" \
             "--" "/opt/airflow/docs/build" \
@@ -40,6 +43,9 @@ function run_generate_requirements() {
         --env VERBOSE_COMMANDS \
         --env HOST_USER_ID="$(id -ur)" \
         --env HOST_GROUP_ID="$(id -gr)" \
+        --env HOST_OS="$(uname -s)" \
+        --env HOST_HOME="${HOME}" \
+        --env HOST_AIRFLOW_SOURCES="${AIRFLOW_SOURCES}" \
         --env UPGRADE_WHILE_GENERATING_REQUIREMENTS \
         --env PYTHON_MAJOR_MINOR_VERSION \
         --env CHECK_REQUIREMENTS_ONLY \
diff --git a/scripts/ci/prepare_tool_scripts.sh b/scripts/ci/prepare_tool_scripts.sh
new file mode 100755
index 0000000..7a98c50
--- /dev/null
+++ b/scripts/ci/prepare_tool_scripts.sh
@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+set -euo pipefail
+
+function prepare_tool_script() {
+    IMAGE="${1}"
+    VOLUME="${2}"
+    TOOL="${3}"
+    COMMAND="${4:-}"
+
+    TARGET_TOOL_PATH="/usr/bin/${TOOL}"
+    TARGET_TOOL_UPDATE_PATH="/usr/bin/${TOOL}-update"
+
+    cat >"${TARGET_TOOL_PATH}" <<EOF
+#!/usr/bin/env bash
+docker run --rm -it \
+    -v "\${HOST_AIRFLOW_SOURCES}/tmp:/tmp" \
+    -v "\${HOST_AIRFLOW_SOURCES}/files:/files" \
+    -v "\${HOST_AIRFLOW_SOURCES}:/opt/airflow" \
+    -v "\${HOST_HOME}/${VOLUME}:/root/${VOLUME}" \
+    "${IMAGE}" ${COMMAND} "\$@"
+RES=\$?
+if [[ \${HOST_OS} == "Linux" ]]; then
+    docker run --rm \
+        -v "\${HOST_AIRFLOW_SOURCES}/tmp:/tmp" \
+        -v "\${HOST_AIRFLOW_SOURCES}/files:/files" \
+        -v "\${HOST_HOME}/${VOLUME}:/root/${VOLUME}" \
+        "\${AIRFLOW_CI_IMAGE}" bash -c \
+        "find '/tmp/' '/files/' '/root/${VOLUME}' -user root -print0 | xargs --null chown '\${HOST_USER_ID}.\${HOST_GROUP_ID}' --no-dereference" >/dev/null 2>&1
+fi
+exit \${RES}
+EOF
+
+    cat >"${TARGET_TOOL_UPDATE_PATH}" <<EOF
+#!/usr/bin/env bash
+docker pull "${IMAGE}"
+EOF
+
+    chmod a+x "${TARGET_TOOL_PATH}" "${TARGET_TOOL_UPDATE_PATH}"
+}
+
+GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest"
+
+prepare_tool_script "amazon/aws-cli:latest" ".aws" aws
+prepare_tool_script "mcr.microsoft.com/azure-cli:latest" ".azure" az az
+prepare_tool_script "${GCLOUD_IMAGE}" ".config/gcloud" bq bq
+prepare_tool_script "${GCLOUD_IMAGE}" ".config/gcloud" gcloud gcloud
+prepare_tool_script "${GCLOUD_IMAGE}" ".config/gcloud" gsutil gsutil


[airflow] 23/36: Remove Hive/Hadoop/Java dependency from unit tests (#9029)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit d4a00b7c69d5d1c4514bc70b2a0022f36b85a06b
Author: James Timmins <ja...@astronomer.io>
AuthorDate: Wed Jun 3 04:49:27 2020 -0700

    Remove Hive/Hadoop/Java dependency from unit tests (#9029)
    
    (cherry picked from commit 10796cb7ce52c8ac2f68024e531fdda779547bdf)
---
 Dockerfile.ci                                     |  50 +-
 airflow/hooks/hive_hooks.py                       |   4 +-
 airflow/operators/hive_operator.py                |  15 +
 airflow/operators/hive_to_samba_operator.py       |  14 +-
 tests/contrib/operators/test_hive_stats.py        | 340 ++++++++++++
 tests/contrib/operators/test_hive_to_mysql.py     | 154 ++++++
 tests/contrib/operators/test_hive_to_samba.py     |  88 ++++
 tests/contrib/sensors/test_hive_partition.py      |  41 ++
 tests/contrib/sensors/test_metastore_partition.py |  43 ++
 tests/contrib/sensors/test_webhdfs.py             |  42 ++
 tests/hive.py                                     |  47 ++
 tests/hooks/test_hive_hook.py                     | 613 ++++++++++++++++------
 tests/operators/test_hive_operator.py             | 411 ++++++---------
 tests/operators/test_operators.py                 | 249 ++++++---
 tests/sensors/test_named_hive_partition_sensor.py | 107 +++-
 tests/test_utils/mock_hooks.py                    |  96 ++++
 tests/test_utils/mock_operators.py                |   9 +
 tests/test_utils/mock_process.py                  |  87 +++
 18 files changed, 1853 insertions(+), 557 deletions(-)

diff --git a/Dockerfile.ci b/Dockerfile.ci
index 2d107aa..15038a5 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -41,7 +41,7 @@ ENV DEBIAN_FRONTEND=noninteractive LANGUAGE=C.UTF-8 LANG=C.UTF-8 LC_ALL=C.UTF-8
     LC_CTYPE=C.UTF-8 LC_MESSAGES=C.UTF-8
 
 # By increasing this number we can do force build of all dependencies
-ARG DEPENDENCIES_EPOCH_NUMBER="3"
+ARG DEPENDENCIES_EPOCH_NUMBER="4"
 # Increase the value below to force renstalling of all dependencies
 ENV DEPENDENCIES_EPOCH_NUMBER=${DEPENDENCIES_EPOCH_NUMBER}
 
@@ -119,14 +119,10 @@ RUN adduser airflow \
 # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199
 RUN mkdir -pv /usr/share/man/man1 \
     && mkdir -pv /usr/share/man/man7 \
-    && curl -fsSL https://adoptopenjdk.jfrog.io/adoptopenjdk/api/gpg/key/public | apt-key add - \
-    && echo 'deb https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/ buster main' > \
-        /etc/apt/sources.list.d/adoptopenjdk.list \
     && apt-get update \
     && apt-get install --no-install-recommends -y \
       gnupg \
       libgcc-8-dev \
-      adoptopenjdk-8-hotspot-jre \
       apt-transport-https \
       bash-completion \
       ca-certificates \
@@ -150,50 +146,6 @@ RUN mkdir -pv /usr/share/man/man1 \
     && apt-get clean \
     && rm -rf /var/lib/apt/lists/*
 
-ENV JAVA_HOME=/usr/lib/jvm/adoptopenjdk-8-hotspot-jre-amd64
-
-# Install Hadoop and Hive
-# It is done in one step to share variables.
-ENV HADOOP_HOME="/opt/hadoop-cdh" HIVE_HOME="/opt/hive"
-
-RUN HADOOP_DISTRO="cdh" \
-    && HADOOP_MAJOR="5" \
-    && HADOOP_DISTRO_VERSION="5.11.0" \
-    && HADOOP_VERSION="2.6.0" \
-    && HADOOP_URL="https://archive.cloudera.com/${HADOOP_DISTRO}${HADOOP_MAJOR}/${HADOOP_DISTRO}/${HADOOP_MAJOR}/"\
-    && HADOOP_DOWNLOAD_URL="${HADOOP_URL}hadoop-${HADOOP_VERSION}-${HADOOP_DISTRO}${HADOOP_DISTRO_VERSION}.tar.gz" \
-    && HADOOP_TMP_FILE="/tmp/hadoop.tar.gz" \
-    && mkdir -pv "${HADOOP_HOME}" \
-    && curl --fail --location "${HADOOP_DOWNLOAD_URL}" --output "${HADOOP_TMP_FILE}" \
-    && tar xzf "${HADOOP_TMP_FILE}" --absolute-names --strip-components 1 -C "${HADOOP_HOME}" \
-    && rm "${HADOOP_TMP_FILE}" \
-    && echo "Installing Hive" \
-    && HIVE_VERSION="1.1.0" \
-    && HIVE_URL="${HADOOP_URL}hive-${HIVE_VERSION}-${HADOOP_DISTRO}${HADOOP_DISTRO_VERSION}.tar.gz" \
-    && HIVE_VERSION="1.1.0" \
-    && HIVE_TMP_FILE="/tmp/hive.tar.gz" \
-    && mkdir -pv "${HIVE_HOME}" \
-    && mkdir -pv "/user/hive/warehouse" \
-    && chmod -R 777 "${HIVE_HOME}" \
-    && chmod -R 777 "/user/" \
-    && curl --fail --location  "${HIVE_URL}" --output "${HIVE_TMP_FILE}" \
-    && tar xzf "${HIVE_TMP_FILE}" --strip-components 1 -C "${HIVE_HOME}" \
-    && rm "${HIVE_TMP_FILE}"
-
-ENV PATH "${PATH}:/opt/hive/bin"
-
-# Install Minicluster
-ENV MINICLUSTER_HOME="/opt/minicluster"
-
-RUN MINICLUSTER_BASE="https://github.com/bolkedebruin/minicluster/releases/download/" \
-    && MINICLUSTER_VER="1.1" \
-    && MINICLUSTER_URL="${MINICLUSTER_BASE}${MINICLUSTER_VER}/minicluster-${MINICLUSTER_VER}-SNAPSHOT-bin.zip" \
-    && MINICLUSTER_TMP_FILE="/tmp/minicluster.zip" \
-    && mkdir -pv "${MINICLUSTER_HOME}" \
-    && curl --fail --location "${MINICLUSTER_URL}" --output "${MINICLUSTER_TMP_FILE}" \
-    && unzip "${MINICLUSTER_TMP_FILE}" -d "/opt" \
-    && rm "${MINICLUSTER_TMP_FILE}"
-
 # Install Docker
 RUN curl --fail --location https://download.docker.com/linux/debian/gpg | apt-key add - \
     && add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian stretch stable" \
diff --git a/airflow/hooks/hive_hooks.py b/airflow/hooks/hive_hooks.py
index ccab93d..4e41834 100644
--- a/airflow/hooks/hive_hooks.py
+++ b/airflow/hooks/hive_hooks.py
@@ -19,6 +19,7 @@
 
 from __future__ import print_function, unicode_literals
 
+import collections
 import contextlib
 import os
 import re
@@ -181,7 +182,7 @@ class HiveCliHook(BaseHook):
             return []
         return as_flattened_list(
             zip(["-hiveconf"] * len(d),
-                ["{}={}".format(k, v) for k, v in d.items()])
+                ["{}={}".format(k, v) for k, v in collections.OrderedDict(sorted(d.items())).items()])
         )
 
     def run_cli(self, hql, schema=None, verbose=True, hive_conf=None):
@@ -825,6 +826,7 @@ class HiveServer2Hook(BaseHook):
             auth=auth_mechanism,
             kerberos_service_name=kerberos_service_name,
             username=db.login or username,
+            password=db.password,
             database=schema or db.schema or 'default')
 
     def _get_results(self, hql, schema='default', fetch_size=None, hive_conf=None):
diff --git a/airflow/operators/hive_operator.py b/airflow/operators/hive_operator.py
index 9cebc38..0d8bca7 100644
--- a/airflow/operators/hive_operator.py
+++ b/airflow/operators/hive_operator.py
@@ -19,8 +19,11 @@
 
 from __future__ import unicode_literals
 
+import os
 import re
 
+from airflow.utils import operator_helpers
+
 from airflow.hooks.hive_hooks import HiveCliHook
 from airflow.models import BaseOperator
 from airflow.utils.decorators import apply_defaults
@@ -134,9 +137,21 @@ class HiveOperator(BaseOperator):
         self.hook.run_cli(hql=self.hql, schema=self.schema, hive_conf=self.hiveconfs)
 
     def dry_run(self):
+        # Reset airflow environment variables to prevent
+        # existing env vars from impacting behavior.
+        self.clear_airflow_vars()
+
         self.hook = self.get_hook()
         self.hook.test_hql(hql=self.hql)
 
     def on_kill(self):
         if self.hook:
             self.hook.kill()
+
+    def clear_airflow_vars(self):
+        """
+        Reset airflow environment variables to prevent existing ones from impacting behavior.
+        """
+        blank_env_vars = {value['env_var_format']: '' for value in
+                          operator_helpers.AIRFLOW_VAR_NAME_FORMAT_MAPPING.values()}
+        os.environ.update(blank_env_vars)
diff --git a/airflow/operators/hive_to_samba_operator.py b/airflow/operators/hive_to_samba_operator.py
index 7963524..e48f96f 100644
--- a/airflow/operators/hive_to_samba_operator.py
+++ b/airflow/operators/hive_to_samba_operator.py
@@ -17,7 +17,7 @@
 # specific language governing permissions and limitations
 # under the License.
 
-import tempfile
+from tempfile import NamedTemporaryFile
 
 from airflow.hooks.hive_hooks import HiveServer2Hook
 from airflow.hooks.samba_hook import SambaHook
@@ -59,9 +59,9 @@ class Hive2SambaOperator(BaseOperator):
     def execute(self, context):
         samba = SambaHook(samba_conn_id=self.samba_conn_id)
         hive = HiveServer2Hook(hiveserver2_conn_id=self.hiveserver2_conn_id)
-        tmpfile = tempfile.NamedTemporaryFile()
-        self.log.info("Fetching file from Hive")
-        hive.to_csv(hql=self.hql, csv_filepath=tmpfile.name,
-                    hive_conf=context_to_airflow_vars(context))
-        self.log.info("Pushing to samba")
-        samba.push_from_local(self.destination_filepath, tmpfile.name)
+        with NamedTemporaryFile() as tmp_file:
+            self.log.info("Fetching file from Hive")
+            hive.to_csv(hql=self.hql, csv_filepath=tmp_file.name,
+                        hive_conf=context_to_airflow_vars(context))
+            self.log.info("Pushing to samba")
+            samba.push_from_local(self.destination_filepath, tmp_file.name)
diff --git a/tests/contrib/operators/test_hive_stats.py b/tests/contrib/operators/test_hive_stats.py
new file mode 100644
index 0000000..7df8e35
--- /dev/null
+++ b/tests/contrib/operators/test_hive_stats.py
@@ -0,0 +1,340 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import re
+import unittest
+from collections import OrderedDict
+
+from airflow.operators.hive_stats_operator import HiveStatsCollectionOperator
+from tests.compat import patch
+
+from airflow.exceptions import AirflowException
+from tests.hive import TestHiveEnvironment, DEFAULT_DATE_DS, DEFAULT_DATE
+from tests.test_utils.mock_hooks import MockHiveMetastoreHook, MockMySqlHook, MockPrestoHook
+
+
+class _FakeCol:
+    def __init__(self, col_name, col_type):
+        self.name = col_name
+        self.type = col_type
+
+
+fake_col = _FakeCol('col', 'string')
+
+
+class TestHiveStatsCollectionOperator(TestHiveEnvironment):
+
+    def setUp(self):
+        self.kwargs = dict(
+            table='table',
+            partition=dict(col='col', value='value'),
+            metastore_conn_id='metastore_conn_id',
+            presto_conn_id='presto_conn_id',
+            mysql_conn_id='mysql_conn_id',
+            task_id='test_hive_stats_collection_operator',
+        )
+        super(TestHiveStatsCollectionOperator, self).setUp()
+
+    def test_get_default_exprs(self):
+        col = 'col'
+
+        default_exprs = HiveStatsCollectionOperator(**self.kwargs).get_default_exprs(col, None)
+
+        self.assertEqual(default_exprs, {
+            (col, 'non_null'): 'COUNT({})'.format(col)
+        })
+
+    def test_get_default_exprs_blacklist(self):
+        col = 'blacklisted_col'
+        self.kwargs.update(dict(col_blacklist=[col]))
+
+        default_exprs = HiveStatsCollectionOperator(**self.kwargs).get_default_exprs(col, None)
+
+        self.assertEqual(default_exprs, {})
+
+    def test_get_default_exprs_number(self):
+        col = 'col'
+        for col_type in ['double', 'int', 'bigint', 'float']:
+            default_exprs = HiveStatsCollectionOperator(**self.kwargs).get_default_exprs(col, col_type)
+
+            self.assertEqual(default_exprs, {
+                (col, 'avg'): 'AVG({})'.format(col),
+                (col, 'max'): 'MAX({})'.format(col),
+                (col, 'min'): 'MIN({})'.format(col),
+                (col, 'non_null'): 'COUNT({})'.format(col),
+                (col, 'sum'): 'SUM({})'.format(col)
+            })
+
+    def test_get_default_exprs_boolean(self):
+        col = 'col'
+        col_type = 'boolean'
+
+        default_exprs = HiveStatsCollectionOperator(**self.kwargs).get_default_exprs(col, col_type)
+
+        self.assertEqual(default_exprs, {
+            (col, 'false'): 'SUM(CASE WHEN NOT {} THEN 1 ELSE 0 END)'.format(col),
+            (col, 'non_null'): 'COUNT({})'.format(col),
+            (col, 'true'): 'SUM(CASE WHEN {} THEN 1 ELSE 0 END)'.format(col)
+        })
+
+    def test_get_default_exprs_string(self):
+        col = 'col'
+        col_type = 'string'
+
+        default_exprs = HiveStatsCollectionOperator(**self.kwargs).get_default_exprs(col, col_type)
+
+        self.assertEqual(default_exprs, {
+            (col, 'approx_distinct'): 'APPROX_DISTINCT({})'.format(col),
+            (col, 'len'): 'SUM(CAST(LENGTH({}) AS BIGINT))'.format(col),
+            (col, 'non_null'): 'COUNT({})'.format(col)
+        })
+
+    @patch('airflow.operators.hive_stats_operator.json.dumps')
+    @patch('airflow.operators.hive_stats_operator.MySqlHook')
+    @patch('airflow.operators.hive_stats_operator.PrestoHook')
+    @patch('airflow.operators.hive_stats_operator.HiveMetastoreHook')
+    def test_execute(self, mock_hive_metastore_hook, mock_presto_hook, mock_mysql_hook, mock_json_dumps):
+        mock_hive_metastore_hook.return_value.get_table.return_value.sd.cols = [fake_col]
+        mock_mysql_hook.return_value.get_records.return_value = False
+
+        hive_stats_collection_operator = HiveStatsCollectionOperator(**self.kwargs)
+        hive_stats_collection_operator.execute(context={})
+
+        mock_hive_metastore_hook.assert_called_once_with(
+            metastore_conn_id=hive_stats_collection_operator.metastore_conn_id)
+        mock_hive_metastore_hook.return_value.get_table.assert_called_once_with(
+            table_name=hive_stats_collection_operator.table)
+        mock_presto_hook.assert_called_once_with(presto_conn_id=hive_stats_collection_operator.presto_conn_id)
+        mock_mysql_hook.assert_called_once_with(hive_stats_collection_operator.mysql_conn_id)
+        mock_json_dumps.assert_called_once_with(hive_stats_collection_operator.partition, sort_keys=True)
+        field_types = {
+            col.name: col.type for col in mock_hive_metastore_hook.return_value.get_table.return_value.sd.cols
+        }
+        exprs = {
+            ('', 'count'): 'COUNT(*)'
+        }
+        for col, col_type in list(field_types.items()):
+            exprs.update(hive_stats_collection_operator.get_default_exprs(col, col_type))
+        exprs = OrderedDict(exprs)
+        rows = [(hive_stats_collection_operator.ds,
+                 hive_stats_collection_operator.dttm,
+                 hive_stats_collection_operator.table,
+                 mock_json_dumps.return_value) +
+                (r[0][0], r[0][1], r[1])
+                for r in zip(exprs, mock_presto_hook.return_value.get_first.return_value)]
+        mock_mysql_hook.return_value.insert_rows.assert_called_once_with(
+            table='hive_stats',
+            rows=rows,
+            target_fields=[
+                'ds',
+                'dttm',
+                'table_name',
+                'partition_repr',
+                'col',
+                'metric',
+                'value',
+            ]
+        )
+
+    @patch('airflow.operators.hive_stats_operator.json.dumps')
+    @patch('airflow.operators.hive_stats_operator.MySqlHook')
+    @patch('airflow.operators.hive_stats_operator.PrestoHook')
+    @patch('airflow.operators.hive_stats_operator.HiveMetastoreHook')
+    def test_execute_with_assignment_func(self,
+                                          mock_hive_metastore_hook,
+                                          mock_presto_hook,
+                                          mock_mysql_hook,
+                                          mock_json_dumps):
+        def assignment_func(col, _):
+            return {
+                (col, 'test'): 'TEST({})'.format(col)
+            }
+
+        self.kwargs.update(dict(assignment_func=assignment_func))
+        mock_hive_metastore_hook.return_value.get_table.return_value.sd.cols = [fake_col]
+        mock_mysql_hook.return_value.get_records.return_value = False
+
+        hive_stats_collection_operator = HiveStatsCollectionOperator(**self.kwargs)
+        hive_stats_collection_operator.execute(context={})
+
+        field_types = {
+            col.name: col.type for col in mock_hive_metastore_hook.return_value.get_table.return_value.sd.cols
+        }
+        exprs = {
+            ('', 'count'): 'COUNT(*)'
+        }
+        for col, col_type in list(field_types.items()):
+            exprs.update(hive_stats_collection_operator.assignment_func(col, col_type))
+        exprs = OrderedDict(exprs)
+        rows = [(hive_stats_collection_operator.ds,
+                 hive_stats_collection_operator.dttm,
+                 hive_stats_collection_operator.table,
+                 mock_json_dumps.return_value) +
+                (r[0][0], r[0][1], r[1])
+                for r in zip(exprs, mock_presto_hook.return_value.get_first.return_value)]
+        mock_mysql_hook.return_value.insert_rows.assert_called_once_with(
+            table='hive_stats',
+            rows=rows,
+            target_fields=[
+                'ds',
+                'dttm',
+                'table_name',
+                'partition_repr',
+                'col',
+                'metric',
+                'value',
+            ]
+        )
+
+    @patch('airflow.operators.hive_stats_operator.json.dumps')
+    @patch('airflow.operators.hive_stats_operator.MySqlHook')
+    @patch('airflow.operators.hive_stats_operator.PrestoHook')
+    @patch('airflow.operators.hive_stats_operator.HiveMetastoreHook')
+    def test_execute_with_assignment_func_no_return_value(self,
+                                                          mock_hive_metastore_hook,
+                                                          mock_presto_hook,
+                                                          mock_mysql_hook,
+                                                          mock_json_dumps):
+        def assignment_func(_, __):
+            pass
+
+        self.kwargs.update(dict(assignment_func=assignment_func))
+        mock_hive_metastore_hook.return_value.get_table.return_value.sd.cols = [fake_col]
+        mock_mysql_hook.return_value.get_records.return_value = False
+
+        hive_stats_collection_operator = HiveStatsCollectionOperator(**self.kwargs)
+        hive_stats_collection_operator.execute(context={})
+
+        field_types = {
+            col.name: col.type for col in mock_hive_metastore_hook.return_value.get_table.return_value.sd.cols
+        }
+        exprs = {
+            ('', 'count'): 'COUNT(*)'
+        }
+        for col, col_type in list(field_types.items()):
+            exprs.update(hive_stats_collection_operator.get_default_exprs(col, col_type))
+        exprs = OrderedDict(exprs)
+        rows = [(hive_stats_collection_operator.ds,
+                 hive_stats_collection_operator.dttm,
+                 hive_stats_collection_operator.table,
+                 mock_json_dumps.return_value) +
+                (r[0][0], r[0][1], r[1])
+                for r in zip(exprs, mock_presto_hook.return_value.get_first.return_value)]
+        mock_mysql_hook.return_value.insert_rows.assert_called_once_with(
+            table='hive_stats',
+            rows=rows,
+            target_fields=[
+                'ds',
+                'dttm',
+                'table_name',
+                'partition_repr',
+                'col',
+                'metric',
+                'value',
+            ]
+        )
+
+    @patch('airflow.operators.hive_stats_operator.MySqlHook')
+    @patch('airflow.operators.hive_stats_operator.PrestoHook')
+    @patch('airflow.operators.hive_stats_operator.HiveMetastoreHook')
+    def test_execute_no_query_results(self, mock_hive_metastore_hook, mock_presto_hook, mock_mysql_hook):
+        mock_hive_metastore_hook.return_value.get_table.return_value.sd.cols = [fake_col]
+        mock_mysql_hook.return_value.get_records.return_value = False
+        mock_presto_hook.return_value.get_first.return_value = None
+
+        self.assertRaises(AirflowException, HiveStatsCollectionOperator(**self.kwargs).execute, context={})
+
+    @patch('airflow.operators.hive_stats_operator.json.dumps')
+    @patch('airflow.operators.hive_stats_operator.MySqlHook')
+    @patch('airflow.operators.hive_stats_operator.PrestoHook')
+    @patch('airflow.operators.hive_stats_operator.HiveMetastoreHook')
+    def test_execute_delete_previous_runs_rows(self,
+                                               mock_hive_metastore_hook,
+                                               mock_presto_hook,
+                                               mock_mysql_hook,
+                                               mock_json_dumps):
+        mock_hive_metastore_hook.return_value.get_table.return_value.sd.cols = [fake_col]
+        mock_mysql_hook.return_value.get_records.return_value = True
+
+        hive_stats_collection_operator = HiveStatsCollectionOperator(**self.kwargs)
+        hive_stats_collection_operator.execute(context={})
+
+        sql = """
+            DELETE FROM hive_stats
+            WHERE
+                table_name='{}' AND
+                partition_repr='{}' AND
+                dttm='{}';
+            """.format(
+            hive_stats_collection_operator.table,
+            mock_json_dumps.return_value,
+            hive_stats_collection_operator.dttm
+        )
+        mock_mysql_hook.return_value.run.assert_called_once_with(sql)
+
+    @unittest.skipIf(
+        'AIRFLOW_RUNALL_TESTS' not in os.environ,
+        "Skipped because AIRFLOW_RUNALL_TESTS is not set")
+    @patch('airflow.operators.hive_stats_operator.HiveMetastoreHook',
+           side_effect=MockHiveMetastoreHook)
+    def test_runs_for_hive_stats(self, mock_hive_metastore_hook):
+        mock_mysql_hook = MockMySqlHook()
+        mock_presto_hook = MockPrestoHook()
+        with patch('airflow.operators.hive_stats_operator.PrestoHook',
+                   return_value=mock_presto_hook):
+            with patch('airflow.operators.hive_stats_operator.MySqlHook',
+                       return_value=mock_mysql_hook):
+                op = HiveStatsCollectionOperator(
+                    task_id='hive_stats_check',
+                    table="airflow.static_babynames_partitioned",
+                    partition={'ds': DEFAULT_DATE_DS},
+                    dag=self.dag)
+                op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
+                       ignore_ti_state=True)
+
+        select_count_query = "SELECT COUNT(*) AS __count FROM airflow." \
+            + "static_babynames_partitioned WHERE ds = '2015-01-01';"
+        mock_presto_hook.get_first.assert_called_with(hql=select_count_query)
+
+        expected_stats_select_query = "SELECT 1 FROM hive_stats WHERE table_name='airflow." \
+            + "static_babynames_partitioned' AND " \
+            + "partition_repr='{\"ds\": \"2015-01-01\"}' AND " \
+            + "dttm='2015-01-01T00:00:00+00:00' " \
+            + "LIMIT 1;"
+
+        raw_stats_select_query = mock_mysql_hook.get_records.call_args_list[0][0][0]
+        actual_stats_select_query = re.sub(r'\s{2,}', ' ', raw_stats_select_query).strip()
+
+        self.assertEqual(expected_stats_select_query, actual_stats_select_query)
+
+        insert_rows_val = [('2015-01-01', '2015-01-01T00:00:00+00:00',
+                            'airflow.static_babynames_partitioned',
+                            '{"ds": "2015-01-01"}', '', 'count', ['val_0', 'val_1'])]
+
+        mock_mysql_hook.insert_rows.assert_called_with(table='hive_stats',
+                                                       rows=insert_rows_val,
+                                                       target_fields=[
+                                                           'ds',
+                                                           'dttm',
+                                                           'table_name',
+                                                           'partition_repr',
+                                                           'col',
+                                                           'metric',
+                                                           'value',
+                                                       ])
diff --git a/tests/contrib/operators/test_hive_to_mysql.py b/tests/contrib/operators/test_hive_to_mysql.py
new file mode 100644
index 0000000..388ab7b
--- /dev/null
+++ b/tests/contrib/operators/test_hive_to_mysql.py
@@ -0,0 +1,154 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import os
+import re
+import unittest
+
+from airflow.operators.hive_to_mysql import HiveToMySqlTransfer
+from tests.compat import MagicMock, PropertyMock, patch
+
+from airflow.utils import timezone
+from airflow.utils.operator_helpers import context_to_airflow_vars
+from tests.hive import TestHiveEnvironment
+
+from tests.test_utils.mock_hooks import MockHiveServer2Hook, MockMySqlHook
+
+DEFAULT_DATE = timezone.datetime(2015, 1, 1)
+
+
+class TestHiveToMySqlTransfer(TestHiveEnvironment):
+
+    def setUp(self):
+        self.kwargs = dict(
+            sql='sql',
+            mysql_table='table',
+            hiveserver2_conn_id='hiveserver2_default',
+            mysql_conn_id='mysql_default',
+            task_id='test_hive_to_mysql',
+        )
+        super(TestHiveToMySqlTransfer, self).setUp()
+
+    @patch('airflow.operators.hive_to_mysql.MySqlHook')
+    @patch('airflow.operators.hive_to_mysql.HiveServer2Hook')
+    def test_execute(self, mock_hive_hook, mock_mysql_hook):
+        HiveToMySqlTransfer(**self.kwargs).execute(context={})
+
+        mock_hive_hook.assert_called_once_with(hiveserver2_conn_id=self.kwargs['hiveserver2_conn_id'])
+        mock_hive_hook.return_value.get_records.assert_called_once_with('sql')
+        mock_mysql_hook.assert_called_once_with(mysql_conn_id=self.kwargs['mysql_conn_id'])
+        mock_mysql_hook.return_value.insert_rows.assert_called_once_with(
+            table=self.kwargs['mysql_table'],
+            rows=mock_hive_hook.return_value.get_records.return_value
+        )
+
+    @patch('airflow.operators.hive_to_mysql.MySqlHook')
+    @patch('airflow.operators.hive_to_mysql.HiveServer2Hook')
+    def test_execute_mysql_preoperator(self, mock_hive_hook, mock_mysql_hook):
+        self.kwargs.update(dict(mysql_preoperator='preoperator'))
+
+        HiveToMySqlTransfer(**self.kwargs).execute(context={})
+
+        mock_mysql_hook.return_value.run.assert_called_once_with(self.kwargs['mysql_preoperator'])
+
+    @patch('airflow.operators.hive_to_mysql.MySqlHook')
+    @patch('airflow.operators.hive_to_mysql.HiveServer2Hook')
+    def test_execute_with_mysql_postoperator(self, mock_hive_hook, mock_mysql_hook):
+        self.kwargs.update(dict(mysql_postoperator='postoperator'))
+
+        HiveToMySqlTransfer(**self.kwargs).execute(context={})
+
+        mock_mysql_hook.return_value.run.assert_called_once_with(self.kwargs['mysql_postoperator'])
+
+    @patch('airflow.operators.hive_to_mysql.MySqlHook')
+    @patch('airflow.operators.hive_to_mysql.NamedTemporaryFile')
+    @patch('airflow.operators.hive_to_mysql.HiveServer2Hook')
+    def test_execute_bulk_load(self, mock_hive_hook, mock_tmp_file, mock_mysql_hook):
+        type(mock_tmp_file).name = PropertyMock(return_value='tmp_file')
+        context = {}
+        self.kwargs.update(dict(bulk_load=True))
+
+        HiveToMySqlTransfer(**self.kwargs).execute(context=context)
+
+        mock_tmp_file.assert_called_once_with()
+        mock_hive_hook.return_value.to_csv.assert_called_once_with(
+            self.kwargs['sql'],
+            mock_tmp_file.return_value.name,
+            delimiter='\t',
+            lineterminator='\n',
+            output_header=False,
+            hive_conf=context_to_airflow_vars(context)
+        )
+        mock_mysql_hook.return_value.bulk_load.assert_called_once_with(
+            table=self.kwargs['mysql_table'],
+            tmp_file=mock_tmp_file.return_value.name
+        )
+        mock_tmp_file.return_value.close.assert_called_once_with()
+
+    @unittest.skipIf(
+        'AIRFLOW_RUNALL_TESTS' not in os.environ,
+        "Skipped because AIRFLOW_RUNALL_TESTS is not set")
+    def test_hive_to_mysql(self):
+        test_hive_results = 'test_hive_results'
+
+        mock_hive_hook = MockHiveServer2Hook()
+        mock_hive_hook.get_records = MagicMock(return_value=test_hive_results)
+
+        mock_mysql_hook = MockMySqlHook()
+        mock_mysql_hook.run = MagicMock()
+        mock_mysql_hook.insert_rows = MagicMock()
+
+        with patch('airflow.operators.hive_to_mysql.HiveServer2Hook',
+                   return_value=mock_hive_hook):
+            with patch('airflow.operators.hive_to_mysql.MySqlHook',
+                       return_value=mock_mysql_hook):
+
+                op = HiveToMySqlTransfer(
+                    mysql_conn_id='airflow_db',
+                    task_id='hive_to_mysql_check',
+                    sql="""
+                        SELECT name
+                        FROM airflow.static_babynames
+                        LIMIT 100
+                        """,
+                    mysql_table='test_static_babynames',
+                    mysql_preoperator=[
+                        'DROP TABLE IF EXISTS test_static_babynames;',
+                        'CREATE TABLE test_static_babynames (name VARCHAR(500))',
+                    ],
+                    dag=self.dag)
+                op.clear(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
+                op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
+                       ignore_ti_state=True)
+
+        raw_select_name_query = mock_hive_hook.get_records.call_args_list[0][0][0]
+        actual_select_name_query = re.sub(r'\s{2,}', ' ', raw_select_name_query).strip()
+        expected_select_name_query = 'SELECT name FROM airflow.static_babynames LIMIT 100'
+        self.assertEqual(expected_select_name_query, actual_select_name_query)
+
+        actual_hive_conf = mock_hive_hook.get_records.call_args_list[0][1]['hive_conf']
+        expected_hive_conf = {'airflow.ctx.dag_owner': 'airflow',
+                              'airflow.ctx.dag_id': 'test_dag_id',
+                              'airflow.ctx.task_id': 'hive_to_mysql_check',
+                              'airflow.ctx.execution_date': '2015-01-01T00:00:00+00:00'}
+        self.assertEqual(expected_hive_conf, actual_hive_conf)
+
+        expected_mysql_preoperator = ['DROP TABLE IF EXISTS test_static_babynames;',
+                                      'CREATE TABLE test_static_babynames (name VARCHAR(500))']
+        mock_mysql_hook.run.assert_called_with(expected_mysql_preoperator)
+
+        mock_mysql_hook.insert_rows.assert_called_with(table='test_static_babynames', rows=test_hive_results)
diff --git a/tests/contrib/operators/test_hive_to_samba.py b/tests/contrib/operators/test_hive_to_samba.py
new file mode 100644
index 0000000..fd7a75f
--- /dev/null
+++ b/tests/contrib/operators/test_hive_to_samba.py
@@ -0,0 +1,88 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import os
+import unittest
+
+from airflow.operators.hive_to_samba_operator import Hive2SambaOperator
+from tests.compat import MagicMock, Mock, PropertyMock, patch
+
+from airflow.utils.operator_helpers import context_to_airflow_vars
+from tests.hive import TestHiveEnvironment, DEFAULT_DATE
+from tests.test_utils.mock_hooks import MockHiveServer2Hook, MockSambaHook
+
+
+class TestHive2SambaOperator(TestHiveEnvironment):
+
+    def setUp(self):
+        self.kwargs = dict(
+            hql='hql',
+            destination_filepath='destination_filepath',
+            samba_conn_id='samba_default',
+            hiveserver2_conn_id='hiveserver2_default',
+            task_id='test_hive_to_samba_operator',
+        )
+        super(TestHive2SambaOperator, self).setUp()
+
+    @patch('airflow.operators.hive_to_samba_operator.SambaHook')
+    @patch('airflow.operators.hive_to_samba_operator.HiveServer2Hook')
+    @patch('airflow.operators.hive_to_samba_operator.NamedTemporaryFile')
+    def test_execute(self, mock_tmp_file, mock_hive_hook, mock_samba_hook):
+        type(mock_tmp_file).name = PropertyMock(return_value='tmp_file')
+        mock_tmp_file.return_value.__enter__ = Mock(return_value=mock_tmp_file)
+        context = {}
+
+        Hive2SambaOperator(**self.kwargs).execute(context)
+
+        mock_hive_hook.assert_called_once_with(
+            hiveserver2_conn_id=self.kwargs['hiveserver2_conn_id'])
+        mock_hive_hook.return_value.to_csv.assert_called_once_with(
+            hql=self.kwargs['hql'],
+            csv_filepath=mock_tmp_file.name,
+            hive_conf=context_to_airflow_vars(context))
+        mock_samba_hook.assert_called_once_with(
+            samba_conn_id=self.kwargs['samba_conn_id'])
+        mock_samba_hook.return_value.push_from_local.assert_called_once_with(
+            self.kwargs['destination_filepath'], mock_tmp_file.name)
+
+    @unittest.skipIf(
+        'AIRFLOW_RUNALL_TESTS' not in os.environ,
+        "Skipped because AIRFLOW_RUNALL_TESTS is not set")
+    @patch('tempfile.tempdir', '/tmp/')
+    @patch('tempfile._RandomNameSequence.__next__')
+    @patch('airflow.operators.hive_to_samba_operator.HiveServer2Hook',
+           side_effect=MockHiveServer2Hook)
+    def test_hive2samba(self, mock_hive_server_hook, mock_temp_dir):
+        mock_temp_dir.return_value = "tst"
+
+        samba_hook = MockSambaHook(self.kwargs['samba_conn_id'])
+        samba_hook.upload = MagicMock()
+
+        with patch('airflow.operators.hive_to_samba_operator.SambaHook',
+                   return_value=samba_hook):
+            samba_hook.conn.upload = MagicMock()
+            op = Hive2SambaOperator(
+                task_id='hive2samba_check',
+                samba_conn_id='tableau_samba',
+                hql="SELECT * FROM airflow.static_babynames LIMIT 10000",
+                destination_filepath='test_airflow.csv',
+                dag=self.dag)
+            op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
+                   ignore_ti_state=True)
+
+        samba_hook.conn.upload.assert_called_with(
+            '/tmp/tmptst', 'test_airflow.csv')
diff --git a/tests/contrib/sensors/test_hive_partition.py b/tests/contrib/sensors/test_hive_partition.py
new file mode 100644
index 0000000..d0c1eda1
--- /dev/null
+++ b/tests/contrib/sensors/test_hive_partition.py
@@ -0,0 +1,41 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import unittest
+
+from airflow.operators.sensors import HivePartitionSensor
+from tests.compat import patch
+from tests.hive import TestHiveEnvironment, DEFAULT_DATE
+
+from tests.test_utils.mock_hooks import MockHiveMetastoreHook
+
+
+@unittest.skipIf(
+    'AIRFLOW_RUNALL_TESTS' not in os.environ,
+    "Skipped because AIRFLOW_RUNALL_TESTS is not set")
+@patch('airflow.providers.apache.hive.sensors.hive_partition.HiveMetastoreHook',
+       side_effect=MockHiveMetastoreHook)
+class TestHivePartitionSensor(TestHiveEnvironment):
+    def test_hive_partition_sensor(self, mock_hive_metastore_hook):
+        op = HivePartitionSensor(
+            task_id='hive_partition_check',
+            table='airflow.static_babynames_partitioned',
+            dag=self.dag)
+        op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
+               ignore_ti_state=True)
diff --git a/tests/contrib/sensors/test_metastore_partition.py b/tests/contrib/sensors/test_metastore_partition.py
new file mode 100644
index 0000000..b8c94a7
--- /dev/null
+++ b/tests/contrib/sensors/test_metastore_partition.py
@@ -0,0 +1,43 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import unittest
+
+from tests.compat import MagicMock
+
+from airflow.operators.sensors import MetastorePartitionSensor
+from tests.hive import TestHiveEnvironment, DEFAULT_DATE_DS, DEFAULT_DATE
+from tests.test_utils.mock_process import MockDBConnection
+
+
+@unittest.skipIf(
+    'AIRFLOW_RUNALL_TESTS' not in os.environ,
+    "Skipped because AIRFLOW_RUNALL_TESTS is not set")
+class TestHivePartitionSensor(TestHiveEnvironment):
+    def test_hive_metastore_sql_sensor(self):
+        op = MetastorePartitionSensor(
+            task_id='hive_partition_check',
+            conn_id='test_connection_id',
+            sql='test_sql',
+            table='airflow.static_babynames_partitioned',
+            partition_name='ds={}'.format(DEFAULT_DATE_DS),
+            dag=self.dag)
+        op._get_hook = MagicMock(return_value=MockDBConnection({}))
+        op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
+               ignore_ti_state=True)
diff --git a/tests/contrib/sensors/test_webhdfs.py b/tests/contrib/sensors/test_webhdfs.py
new file mode 100644
index 0000000..60eb992
--- /dev/null
+++ b/tests/contrib/sensors/test_webhdfs.py
@@ -0,0 +1,42 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import os
+import unittest
+
+from tests.compat import patch
+
+from airflow.operators.sensors import WebHdfsSensor
+from tests.hive import TestHiveEnvironment, DEFAULT_DATE
+from tests.test_utils.mock_hooks import MockDBConnection
+
+
+@unittest.skipIf(
+    'AIRFLOW_RUNALL_TESTS' not in os.environ,
+    "Skipped because AIRFLOW_RUNALL_TESTS is not set")
+class TestWebHdfsSensor(TestHiveEnvironment):
+
+    @patch('airflow.providers.apache.hdfs.hooks.webhdfs.WebHDFSHook._find_valid_server')
+    def test_webhdfs_sensor(self, mock_find_valid_server):
+        mock_find_valid_server.return_value = MockDBConnection()
+        op = WebHdfsSensor(
+            task_id='webhdfs_sensor_check',
+            filepath='hdfs://user/hive/warehouse/airflow.db/static_babynames',
+            timeout=120,
+            dag=self.dag)
+        op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
+               ignore_ti_state=True)
diff --git a/tests/hive.py b/tests/hive.py
new file mode 100644
index 0000000..65c5a85
--- /dev/null
+++ b/tests/hive.py
@@ -0,0 +1,47 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from datetime import datetime
+from unittest import TestCase
+
+from airflow.models.dag import DAG
+
+DEFAULT_DATE = datetime(2015, 1, 1)
+DEFAULT_DATE_ISO = DEFAULT_DATE.isoformat()
+DEFAULT_DATE_DS = DEFAULT_DATE_ISO[:10]
+
+
+class TestHiveEnvironment(TestCase):
+
+    def setUp(self):
+        args = {'owner': 'airflow', 'start_date': DEFAULT_DATE}
+        dag = DAG('test_dag_id', default_args=args)
+        self.dag = dag
+        self.hql = """
+        USE airflow;
+        DROP TABLE IF EXISTS static_babynames_partitioned;
+        CREATE TABLE IF NOT EXISTS static_babynames_partitioned (
+            state string,
+            year string,
+            name string,
+            gender string,
+            num int)
+        PARTITIONED BY (ds string);
+        INSERT OVERWRITE TABLE static_babynames_partitioned
+            PARTITION(ds='{{ ds }}')
+        SELECT state, year, name, gender, num FROM static_babynames;
+        """
diff --git a/tests/hooks/test_hive_hook.py b/tests/hooks/test_hive_hook.py
index ca16c7e..98c024f 100644
--- a/tests/hooks/test_hive_hook.py
+++ b/tests/hooks/test_hive_hook.py
@@ -21,80 +21,82 @@
 import datetime
 import itertools
 import os
-import random
-import sys
 import unittest
-from collections import OrderedDict
+from collections import OrderedDict, namedtuple
 
-import mock
 import pandas as pd
 from hmsclient import HMSClient
+from six import PY2
 
-from airflow import DAG
 from airflow.exceptions import AirflowException
-from airflow.hooks.hive_hooks import HiveCliHook, HiveMetastoreHook, HiveServer2Hook
+from airflow.hooks.hive_hooks import HiveMetastoreHook, HiveServer2Hook
 from airflow.models.connection import Connection
-from airflow.operators.hive_operator import HiveOperator
+from airflow.models.dag import DAG
+from airflow.secrets.environment_variables import CONN_ENV_PREFIX
 from airflow.utils import timezone
 from airflow.utils.operator_helpers import AIRFLOW_VAR_NAME_FORMAT_MAPPING
-from airflow.utils.tests import assertEqualIgnoreMultipleSpaces
+from tests.compat import patch, MagicMock, call
+from tests.test_utils.mock_hooks import MockHiveCliHook, MockHiveServer2Hook
+from tests.test_utils.mock_process import MockSubProcess
 
 DEFAULT_DATE = timezone.datetime(2015, 1, 1)
 DEFAULT_DATE_ISO = DEFAULT_DATE.isoformat()
 DEFAULT_DATE_DS = DEFAULT_DATE_ISO[:10]
-NOT_ASSERTLOGS_VERSION = sys.version_info.major + sys.version_info.minor / 10
 
 
-class HiveEnvironmentTest(unittest.TestCase):
+class TestHiveEnvironment(unittest.TestCase):
 
     def setUp(self):
-        args = {'owner': 'airflow', 'start_date': DEFAULT_DATE}
-        self.dag = DAG('test_dag_id', default_args=args)
         self.next_day = (DEFAULT_DATE +
                          datetime.timedelta(days=1)).isoformat()[:10]
         self.database = 'airflow'
         self.partition_by = 'ds'
         self.table = 'static_babynames_partitioned'
-        self.hql = """
-        CREATE DATABASE IF NOT EXISTS {{ params.database }};
-        USE {{ params.database }};
-        DROP TABLE IF EXISTS {{ params.table }};
-        CREATE TABLE IF NOT EXISTS {{ params.table }} (
-            state string,
-            year string,
-            name string,
-            gender string,
-            num int)
-        PARTITIONED BY ({{ params.partition_by }} string);
-        ALTER TABLE {{ params.table }}
-        ADD PARTITION({{ params.partition_by }}='{{ ds }}');
-        """
-        self.hook = HiveMetastoreHook()
-        t = HiveOperator(
-            task_id='HiveHook_' + str(random.randint(1, 10000)),
-            params={
-                'database': self.database,
-                'table': self.table,
-                'partition_by': self.partition_by
-            },
-            hive_cli_conn_id='beeline_default',
-            hql=self.hql, dag=self.dag)
-        t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
-              ignore_ti_state=True)
+        with patch('airflow.hooks.hive_hooks.HiveMetastoreHook.get_metastore_client') \
+                as get_metastore_mock:
+            get_metastore_mock.return_value = MagicMock()
 
-    def tearDown(self):
-        hook = HiveMetastoreHook()
-        with hook.get_conn() as metastore:
-            metastore.drop_table(self.database, self.table, deleteData=True)
+            self.hook = HiveMetastoreHook()
 
 
 class TestHiveCliHook(unittest.TestCase):
+    @patch('tempfile.tempdir', '/tmp/')
+    @patch('tempfile._RandomNameSequence.next' if PY2 else 'tempfile._RandomNameSequence.__next__')
+    @patch('subprocess.Popen')
+    def test_run_cli(self, mock_popen, mock_temp_dir):
+        mock_subprocess = MockSubProcess()
+        mock_popen.return_value = mock_subprocess
+        mock_temp_dir.return_value = "test_run_cli"
+        with patch.dict('os.environ', {
+            'AIRFLOW_CTX_DAG_ID': 'test_dag_id',
+            'AIRFLOW_CTX_TASK_ID': 'test_task_id',
+            'AIRFLOW_CTX_EXECUTION_DATE': '2015-01-01T00:00:00+00:00',
+            'AIRFLOW_CTX_DAG_RUN_ID': '55',
+            'AIRFLOW_CTX_DAG_OWNER': 'airflow',
+            'AIRFLOW_CTX_DAG_EMAIL': 'test@airflow.com',
+        }):
 
-    def test_run_cli(self):
-        hook = HiveCliHook()
-        hook.run_cli("SHOW DATABASES")
+            hook = MockHiveCliHook()
+            hook.run_cli("SHOW DATABASES")
+
+        hive_cmd = [u'hive', u'-hiveconf', u'airflow.ctx.dag_email=test@airflow.com', u'-hiveconf',
+                    u'airflow.ctx.dag_id=test_dag_id', u'-hiveconf', u'airflow.ctx.dag_owner=airflow',
+                    u'-hiveconf', u'airflow.ctx.dag_run_id=55', u'-hiveconf',
+                    u'airflow.ctx.execution_date=2015-01-01T00:00:00+00:00', u'-hiveconf',
+                    u'airflow.ctx.task_id=test_task_id', u'-hiveconf', u'mapreduce.job.queuename=airflow',
+                    u'-hiveconf', u'mapred.job.queue.name=airflow', u'-hiveconf', u'tez.queue.name=airflow',
+                    u'-f', u'/tmp/airflow_hiveop_test_run_cli/tmptest_run_cli']
+
+        mock_popen.assert_called_with(
+            hive_cmd,
+            stdout=mock_subprocess.PIPE,
+            stderr=mock_subprocess.STDOUT,
+            cwd="/tmp/airflow_hiveop_test_run_cli",
+            close_fds=True
+        )
 
-    def test_run_cli_with_hive_conf(self):
+    @patch('subprocess.Popen')
+    def test_run_cli_with_hive_conf(self, mock_popen):
         hql = "set key;\n" \
               "set airflow.ctx.dag_id;\nset airflow.ctx.dag_run_id;\n" \
               "set airflow.ctx.task_id;\nset airflow.ctx.execution_date;\n"
@@ -109,30 +111,87 @@ class TestHiveCliHook(unittest.TestCase):
         dag_run_id_ctx_var_name = \
             AIRFLOW_VAR_NAME_FORMAT_MAPPING['AIRFLOW_CONTEXT_DAG_RUN_ID'][
                 'env_var_format']
-        os.environ[dag_id_ctx_var_name] = 'test_dag_id'
-        os.environ[task_id_ctx_var_name] = 'test_task_id'
-        os.environ[execution_date_ctx_var_name] = 'test_execution_date'
-        os.environ[dag_run_id_ctx_var_name] = 'test_dag_run_id'
 
-        hook = HiveCliHook()
-        output = hook.run_cli(hql=hql, hive_conf={'key': 'value'})
-        self.assertIn('value', output)
-        self.assertIn('test_dag_id', output)
-        self.assertIn('test_task_id', output)
-        self.assertIn('test_execution_date', output)
-        self.assertIn('test_dag_run_id', output)
+        mock_output = ['Connecting to jdbc:hive2://localhost:10000/default',
+                       'log4j:WARN No appenders could be found for logger (org.apache.hive.jdbc.Utils).',
+                       'log4j:WARN Please initialize the log4j system properly.',
+                       'log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.',
+                       'Connected to: Apache Hive (version 1.2.1.2.3.2.0-2950)',
+                       'Driver: Hive JDBC (version 1.2.1.spark2)',
+                       'Transaction isolation: TRANSACTION_REPEATABLE_READ',
+                       '0: jdbc:hive2://localhost:10000/default> USE default;',
+                       'No rows affected (0.37 seconds)',
+                       '0: jdbc:hive2://localhost:10000/default> set key;',
+                       '+------------+--+',
+                       '|    set     |',
+                       '+------------+--+',
+                       '| key=value  |',
+                       '+------------+--+',
+                       '1 row selected (0.133 seconds)',
+                       '0: jdbc:hive2://localhost:10000/default> set airflow.ctx.dag_id;',
+                       '+---------------------------------+--+',
+                       '|               set               |',
+                       '+---------------------------------+--+',
+                       '| airflow.ctx.dag_id=test_dag_id  |',
+                       '+---------------------------------+--+',
+                       '1 row selected (0.008 seconds)',
+                       '0: jdbc:hive2://localhost:10000/default> set airflow.ctx.dag_run_id;',
+                       '+-----------------------------------------+--+',
+                       '|                   set                   |',
+                       '+-----------------------------------------+--+',
+                       '| airflow.ctx.dag_run_id=test_dag_run_id  |',
+                       '+-----------------------------------------+--+',
+                       '1 row selected (0.007 seconds)',
+                       '0: jdbc:hive2://localhost:10000/default> set airflow.ctx.task_id;',
+                       '+-----------------------------------+--+',
+                       '|                set                |',
+                       '+-----------------------------------+--+',
+                       '| airflow.ctx.task_id=test_task_id  |',
+                       '+-----------------------------------+--+',
+                       '1 row selected (0.009 seconds)',
+                       '0: jdbc:hive2://localhost:10000/default> set airflow.ctx.execution_date;',
+                       '+-------------------------------------------------+--+',
+                       '|                       set                       |',
+                       '+-------------------------------------------------+--+',
+                       '| airflow.ctx.execution_date=test_execution_date  |',
+                       '+-------------------------------------------------+--+',
+                       '1 row selected (0.006 seconds)',
+                       '0: jdbc:hive2://localhost:10000/default> ',
+                       '0: jdbc:hive2://localhost:10000/default> ',
+                       'Closing: 0: jdbc:hive2://localhost:10000/default',
+                       '']
+
+        with patch.dict('os.environ', {
+            dag_id_ctx_var_name: 'test_dag_id',
+            task_id_ctx_var_name: 'test_task_id',
+            execution_date_ctx_var_name: 'test_execution_date',
+            dag_run_id_ctx_var_name: 'test_dag_run_id',
+        }):
+
+            hook = MockHiveCliHook()
+            mock_popen.return_value = MockSubProcess(output=mock_output)
+
+            output = hook.run_cli(hql=hql, hive_conf={'key': 'value'})
+            process_inputs = " ".join(mock_popen.call_args_list[0][0][0])
 
-        del os.environ[dag_id_ctx_var_name]
-        del os.environ[task_id_ctx_var_name]
-        del os.environ[execution_date_ctx_var_name]
-        del os.environ[dag_run_id_ctx_var_name]
+            self.assertIn('value', process_inputs)
+            self.assertIn('test_dag_id', process_inputs)
+            self.assertIn('test_task_id', process_inputs)
+            self.assertIn('test_execution_date', process_inputs)
+            self.assertIn('test_dag_run_id', process_inputs)
 
-    @mock.patch('airflow.hooks.hive_hooks.HiveCliHook.run_cli')
+            self.assertIn('value', output)
+            self.assertIn('test_dag_id', output)
+            self.assertIn('test_task_id', output)
+            self.assertIn('test_execution_date', output)
+            self.assertIn('test_dag_run_id', output)
+
+    @patch('airflow.hooks.hive_hooks.HiveCliHook.run_cli')
     def test_load_file_without_create_table(self, mock_run_cli):
         filepath = "/path/to/input/file"
         table = "output_table"
 
-        hook = HiveCliHook()
+        hook = MockHiveCliHook()
         hook.load_file(filepath=filepath, table=table, create=False)
 
         query = (
@@ -140,21 +199,22 @@ class TestHiveCliHook(unittest.TestCase):
             "OVERWRITE INTO TABLE {table} ;\n"
             .format(filepath=filepath, table=table)
         )
-
         calls = [
-            mock.call(query)
+            call(query)
         ]
         mock_run_cli.assert_has_calls(calls, any_order=True)
 
-    @mock.patch('airflow.hooks.hive_hooks.HiveCliHook.run_cli')
+    @patch('airflow.hooks.hive_hooks.HiveCliHook.run_cli')
     def test_load_file_create_table(self, mock_run_cli):
         filepath = "/path/to/input/file"
         table = "output_table"
         field_dict = OrderedDict([("name", "string"), ("gender", "string")])
-        fields = ",\n    ".join(['`{k}` {v}'.format(k=k.strip('`'), v=v) for k, v in field_dict.items()])
+        fields = ",\n    ".join(
+            ['`{k}` {v}'.format(k=k.strip('`'), v=v) for k, v in field_dict.items()])
 
-        hook = HiveCliHook()
-        hook.load_file(filepath=filepath, table=table, field_dict=field_dict, create=True, recreate=True)
+        hook = MockHiveCliHook()
+        hook.load_file(filepath=filepath, table=table,
+                       field_dict=field_dict, create=True, recreate=True)
 
         create_table = (
             "DROP TABLE IF EXISTS {table};\n"
@@ -170,20 +230,20 @@ class TestHiveCliHook(unittest.TestCase):
             .format(filepath=filepath, table=table)
         )
         calls = [
-            mock.call(create_table),
-            mock.call(load_data)
+            call(create_table),
+            call(load_data)
         ]
         mock_run_cli.assert_has_calls(calls, any_order=True)
 
-    @mock.patch('airflow.hooks.hive_hooks.HiveCliHook.load_file')
-    @mock.patch('pandas.DataFrame.to_csv')
+    @patch('airflow.hooks.hive_hooks.HiveCliHook.load_file')
+    @patch('pandas.DataFrame.to_csv')
     def test_load_df(self, mock_to_csv, mock_load_file):
         df = pd.DataFrame({"c": ["foo", "bar", "baz"]})
         table = "t"
         delimiter = ","
         encoding = "utf-8"
 
-        hook = HiveCliHook()
+        hook = MockHiveCliHook()
         hook.load_df(df=df,
                      table=table,
                      delimiter=delimiter,
@@ -198,16 +258,16 @@ class TestHiveCliHook(unittest.TestCase):
         assert mock_load_file.call_count == 1
         kwargs = mock_load_file.call_args[1]
         self.assertEqual(kwargs["delimiter"], delimiter)
-        self.assertEqual(kwargs["field_dict"], {"c": u"STRING"})
+        self.assertEqual(kwargs["field_dict"], {"c": "STRING"})
         self.assertTrue(isinstance(kwargs["field_dict"], OrderedDict))
         self.assertEqual(kwargs["table"], table)
 
-    @mock.patch('airflow.hooks.hive_hooks.HiveCliHook.load_file')
-    @mock.patch('pandas.DataFrame.to_csv')
+    @patch('airflow.hooks.hive_hooks.HiveCliHook.load_file')
+    @patch('pandas.DataFrame.to_csv')
     def test_load_df_with_optional_parameters(self, mock_to_csv, mock_load_file):
-        hook = HiveCliHook()
-        b = (True, False)
-        for create, recreate in itertools.product(b, b):
+        hook = MockHiveCliHook()
+        bools = (True, False)
+        for create, recreate in itertools.product(bools, bools):
             mock_load_file.reset_mock()
             hook.load_df(df=pd.DataFrame({"c": range(0, 10)}),
                          table="t",
@@ -219,22 +279,22 @@ class TestHiveCliHook(unittest.TestCase):
             self.assertEqual(kwargs["create"], create)
             self.assertEqual(kwargs["recreate"], recreate)
 
-    @mock.patch('airflow.hooks.hive_hooks.HiveCliHook.run_cli')
+    @patch('airflow.hooks.hive_hooks.HiveCliHook.run_cli')
     def test_load_df_with_data_types(self, mock_run_cli):
-        d = OrderedDict()
-        d['b'] = [True]
-        d['i'] = [-1]
-        d['t'] = [1]
-        d['f'] = [0.0]
-        d['c'] = ['c']
-        d['M'] = [datetime.datetime(2018, 1, 1)]
-        d['O'] = [object()]
-        d['S'] = ['STRING'.encode('utf-8')]
-        d['U'] = ['STRING']
-        d['V'] = [None]
-        df = pd.DataFrame(d)
-
-        hook = HiveCliHook()
+        ord_dict = OrderedDict()
+        ord_dict['b'] = [True]
+        ord_dict['i'] = [-1]
+        ord_dict['t'] = [1]
+        ord_dict['f'] = [0.0]
+        ord_dict['c'] = ['c']
+        ord_dict['M'] = [datetime.datetime(2018, 1, 1)]
+        ord_dict['O'] = [object()]
+        ord_dict['S'] = [b'STRING']
+        ord_dict['U'] = ['STRING']
+        ord_dict['V'] = [None]
+        df = pd.DataFrame(ord_dict)
+
+        hook = MockHiveCliHook()
         hook.load_df(df, 't')
 
         query = """
@@ -254,10 +314,11 @@ class TestHiveCliHook(unittest.TestCase):
             STORED AS textfile
             ;
         """
-        assertEqualIgnoreMultipleSpaces(self, mock_run_cli.call_args_list[0][0][0], query)
+        call(
+            self, mock_run_cli.call_args_list[0][0][0], query)
 
 
-class TestHiveMetastoreHook(HiveEnvironmentTest):
+class TestHiveMetastoreHook(TestHiveEnvironment):
     VALID_FILTER_MAP = {'key2': 'value2'}
 
     def test_get_max_partition_from_empty_part_specs(self):
@@ -267,6 +328,7 @@ class TestHiveMetastoreHook(HiveEnvironmentTest):
                                                                  self.VALID_FILTER_MAP)
         self.assertIsNone(max_partition)
 
+    # @patch('airflow.hooks.hive_hooks.HiveMetastoreHook', 'get_metastore_client')
     def test_get_max_partition_from_valid_part_specs_and_invalid_filter_map(self):
         with self.assertRaises(AirflowException):
             HiveMetastoreHook._get_max_partition_from_part_specs(
@@ -311,72 +373,150 @@ class TestHiveMetastoreHook(HiveEnvironmentTest):
                 self.VALID_FILTER_MAP)
         self.assertEqual(max_partition, b'value1')
 
-    def test_get_metastore_client(self):
-        self.assertIsInstance(self.hook.get_metastore_client(), HMSClient)
-
-    @mock.patch("airflow.hooks.hive_hooks.HiveMetastoreHook.get_connection",
-                return_value=[Connection(host="localhost", port="9802")])
-    @mock.patch("airflow.hooks.hive_hooks.socket")
-    def test_error_metastore_client(self, socket_mock, _find_vaild_server_mock):
+    @patch("airflow.hooks.hive_hooks.HiveMetastoreHook.get_connection",
+           return_value=[Connection(host="localhost", port="9802")])
+    @patch("airflow.hooks.hive_hooks.socket")
+    def test_error_metastore_client(self, socket_mock, _find_valid_server_mock):
         socket_mock.socket.return_value.connect_ex.return_value = 0
         self.hook.get_metastore_client()
 
     def test_get_conn(self):
-        self.assertIsInstance(self.hook.get_conn(), HMSClient)
+        with patch('airflow.hooks.hive_hooks.HiveMetastoreHook._find_valid_server') \
+                as find_valid_server:
+            find_valid_server.return_value = MagicMock(return_value={})
+            metastore_hook = HiveMetastoreHook()
+
+        self.assertIsInstance(metastore_hook.get_conn(), HMSClient)
 
     def test_check_for_partition(self):
+        # Check for existent partition.
+        FakePartition = namedtuple('FakePartition', ['values'])
+        fake_partition = FakePartition(['2015-01-01'])
+
+        metastore = self.hook.metastore.__enter__()
+
         partition = "{p_by}='{date}'".format(date=DEFAULT_DATE_DS,
                                              p_by=self.partition_by)
-        missing_partition = "{p_by}='{date}'".format(date=self.next_day,
-                                                     p_by=self.partition_by)
+
+        metastore.get_partitions_by_filter = MagicMock(
+            return_value=[fake_partition])
+
         self.assertTrue(
             self.hook.check_for_partition(self.database, self.table,
                                           partition)
         )
+
+        metastore.get_partitions_by_filter(
+            self.database, self.table, partition, 1)
+
+        # Check for non-existent partition.
+        missing_partition = "{p_by}='{date}'".format(date=self.next_day,
+                                                     p_by=self.partition_by)
+        metastore.get_partitions_by_filter = MagicMock(return_value=[])
+
         self.assertFalse(
             self.hook.check_for_partition(self.database, self.table,
                                           missing_partition)
         )
 
+        metastore.get_partitions_by_filter.assert_called_with(
+            self.database, self.table, missing_partition, 1)
+
     def test_check_for_named_partition(self):
+
+        # Check for existing partition.
+
         partition = "{p_by}={date}".format(date=DEFAULT_DATE_DS,
                                            p_by=self.partition_by)
-        missing_partition = "{p_by}={date}".format(date=self.next_day,
-                                                   p_by=self.partition_by)
+
+        self.hook.metastore.__enter__(
+        ).check_for_named_partition = MagicMock(return_value=True)
+
         self.assertTrue(
             self.hook.check_for_named_partition(self.database,
                                                 self.table,
-                                                partition)
-        )
+                                                partition))
+
+        self.hook.metastore.__enter__().check_for_named_partition.assert_called_with(
+            self.database, self.table, partition)
+
+        # Check for non-existent partition
+        missing_partition = "{p_by}={date}".format(date=self.next_day,
+                                                   p_by=self.partition_by)
+
+        self.hook.metastore.__enter__().check_for_named_partition = MagicMock(
+            return_value=False)
+
         self.assertFalse(
             self.hook.check_for_named_partition(self.database,
                                                 self.table,
                                                 missing_partition)
         )
+        self.hook.metastore.__enter__().check_for_named_partition.assert_called_with(
+            self.database, self.table, missing_partition)
 
     def test_get_table(self):
-        table_info = self.hook.get_table(db=self.database,
-                                         table_name=self.table)
-        self.assertEqual(table_info.tableName, self.table)
-        columns = ['state', 'year', 'name', 'gender', 'num']
-        self.assertEqual([col.name for col in table_info.sd.cols], columns)
 
-    def test_get_tables(self):
-        tables = self.hook.get_tables(db=self.database,
-                                      pattern=self.table + "*")
-        self.assertIn(self.table, {table.tableName for table in tables})
+        self.hook.metastore.__enter__().get_table = MagicMock()
+        self.hook.get_table(db=self.database, table_name=self.table)
+        self.hook.metastore.__enter__().get_table.assert_called_with(
+            dbname=self.database, tbl_name=self.table)
+
+    def test_get_tables(self):  # static_babynames_partitioned
+        self.hook.metastore.__enter__().get_tables = MagicMock(
+            return_value=['static_babynames_partitioned'])
+
+        self.hook.get_tables(db=self.database, pattern=self.table + "*")
+        self.hook.metastore.__enter__().get_tables.assert_called_with(
+            db_name='airflow', pattern='static_babynames_partitioned*')
+        self.hook.metastore.__enter__().get_table_objects_by_name.assert_called_with(
+            'airflow', ['static_babynames_partitioned'])
 
     def test_get_databases(self):
-        databases = self.hook.get_databases(pattern='*')
-        self.assertIn(self.database, databases)
+        metastore = self.hook.metastore.__enter__()
+        metastore.get_databases = MagicMock()
+
+        self.hook.get_databases(pattern='*')
+        metastore.get_databases.assert_called_with('*')
 
     def test_get_partitions(self):
+        FakeFieldSchema = namedtuple('FakeFieldSchema', ['name'])
+        fake_schema = FakeFieldSchema('ds')
+        FakeTable = namedtuple('FakeTable', ['partitionKeys'])
+        fake_table = FakeTable([fake_schema])
+        FakePartition = namedtuple('FakePartition', ['values'])
+        fake_partition = FakePartition(['2015-01-01'])
+
+        metastore = self.hook.metastore.__enter__()
+        metastore.get_table = MagicMock(return_value=fake_table)
+
+        metastore.get_partitions = MagicMock(
+            return_value=[fake_partition])
+
         partitions = self.hook.get_partitions(schema=self.database,
                                               table_name=self.table)
         self.assertEqual(len(partitions), 1)
         self.assertEqual(partitions, [{self.partition_by: DEFAULT_DATE_DS}])
 
+        metastore.get_table.assert_called_with(
+            dbname=self.database, tbl_name=self.table)
+        metastore.get_partitions.assert_called_with(
+            db_name=self.database, tbl_name=self.table, max_parts=HiveMetastoreHook.MAX_PART_COUNT)
+
     def test_max_partition(self):
+        FakeFieldSchema = namedtuple('FakeFieldSchema', ['name'])
+        fake_schema = FakeFieldSchema('ds')
+        FakeTable = namedtuple('FakeTable', ['partitionKeys'])
+        fake_table = FakeTable([fake_schema])
+
+        metastore = self.hook.metastore.__enter__()
+        metastore.get_table = MagicMock(return_value=fake_table)
+
+        metastore.get_partition_names = MagicMock(
+            return_value=['ds=2015-01-01'])
+        metastore.partition_name_to_spec = MagicMock(
+            return_value={'ds': '2015-01-01'})
+
         filter_map = {self.partition_by: DEFAULT_DATE_DS}
         partition = self.hook.max_partition(schema=self.database,
                                             table_name=self.table,
@@ -384,11 +524,28 @@ class TestHiveMetastoreHook(HiveEnvironmentTest):
                                             filter_map=filter_map)
         self.assertEqual(partition, DEFAULT_DATE_DS.encode('utf-8'))
 
+        metastore.get_table.assert_called_with(
+            dbname=self.database, tbl_name=self.table)
+        metastore.get_partition_names.assert_called_with(
+            self.database, self.table, max_parts=HiveMetastoreHook.MAX_PART_COUNT)
+        metastore.partition_name_to_spec.assert_called_with('ds=2015-01-01')
+
     def test_table_exists(self):
+        # Test with existent table.
+        self.hook.metastore.__enter__().get_table = MagicMock(return_value=True)
+
         self.assertTrue(self.hook.table_exists(self.table, db=self.database))
+        self.hook.metastore.__enter__().get_table.assert_called_with(
+            dbname='airflow', tbl_name='static_babynames_partitioned')
+
+        # Test with non-existent table.
+        self.hook.metastore.__enter__().get_table = MagicMock(side_effect=Exception())
+
         self.assertFalse(
-            self.hook.table_exists(str(random.randint(1, 10000)))
+            self.hook.table_exists("does-not-exist")
         )
+        self.hook.metastore.__enter__().get_table.assert_called_with(
+            dbname='default', tbl_name='does-not-exist')
 
 
 class TestHiveServer2Hook(unittest.TestCase):
@@ -404,6 +561,7 @@ class TestHiveServer2Hook(unittest.TestCase):
         self.dag = DAG('test_dag_id', default_args=args)
         self.database = 'airflow'
         self.table = 'hive_server_hook'
+
         self.hql = """
         CREATE DATABASE IF NOT EXISTS {{ params.database }};
         USE {{ params.database }};
@@ -418,59 +576,122 @@ class TestHiveServer2Hook(unittest.TestCase):
         """
         self.columns = ['{}.a'.format(self.table),
                         '{}.b'.format(self.table)]
-        self.hook = HiveMetastoreHook()
-        t = HiveOperator(
-            task_id='HiveHook_' + str(random.randint(1, 10000)),
-            params={
-                'database': self.database,
-                'table': self.table,
-                'csv_path': self.local_path
-            },
-            hive_cli_conn_id='beeline_default',
-            hql=self.hql, dag=self.dag)
-        t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
-              ignore_ti_state=True)
 
-    def tearDown(self):
-        hook = HiveMetastoreHook()
-        with hook.get_conn() as metastore:
-            metastore.drop_table(self.database, self.table, deleteData=True)
-        os.remove(self.local_path)
+        with patch('airflow.hooks.hive_hooks.HiveMetastoreHook.get_metastore_client') \
+                as get_metastore_mock:
+            get_metastore_mock.return_value = MagicMock()
+
+            self.hook = HiveMetastoreHook()
 
     def test_get_conn(self):
-        hook = HiveServer2Hook()
+        hook = MockHiveServer2Hook()
         hook.get_conn()
 
+    @patch('pyhive.hive.connect')
+    def test_get_conn_with_password(self, mock_connect):
+        conn_id = "conn_with_password"
+        conn_env = CONN_ENV_PREFIX + conn_id.upper()
+
+        with patch.dict(
+            'os.environ',
+            {conn_env: "jdbc+hive2://conn_id:conn_pass@localhost:10000/default?authMechanism=LDAP"}
+        ):
+            HiveServer2Hook(hiveserver2_conn_id=conn_id).get_conn()
+            mock_connect.assert_called_once_with(
+                host='localhost',
+                port=10000,
+                auth='LDAP',
+                kerberos_service_name=None,
+                username='conn_id',
+                password='conn_pass',
+                database='default')
+
     def test_get_records(self):
-        hook = HiveServer2Hook()
+        hook = MockHiveServer2Hook()
         query = "SELECT * FROM {}".format(self.table)
-        results = hook.get_records(query, schema=self.database)
+
+        with patch.dict('os.environ', {
+            'AIRFLOW_CTX_DAG_ID': 'test_dag_id',
+            'AIRFLOW_CTX_TASK_ID': 'HiveHook_3835',
+            'AIRFLOW_CTX_EXECUTION_DATE': '2015-01-01T00:00:00+00:00',
+            'AIRFLOW_CTX_DAG_RUN_ID': '55',
+            'AIRFLOW_CTX_DAG_OWNER': 'airflow',
+            'AIRFLOW_CTX_DAG_EMAIL': 'test@airflow.com',
+        }):
+            results = hook.get_records(query, schema=self.database)
+
         self.assertListEqual(results, [(1, 1), (2, 2)])
 
+        hook.get_conn.assert_called_with(self.database)
+        hook.mock_cursor.execute.assert_any_call(
+            'set airflow.ctx.dag_id=test_dag_id')
+        hook.mock_cursor.execute.assert_any_call(
+            'set airflow.ctx.task_id=HiveHook_3835')
+        hook.mock_cursor.execute.assert_any_call(
+            'set airflow.ctx.execution_date=2015-01-01T00:00:00+00:00')
+        hook.mock_cursor.execute.assert_any_call(
+            'set airflow.ctx.dag_run_id=55')
+        hook.mock_cursor.execute.assert_any_call(
+            'set airflow.ctx.dag_owner=airflow')
+        hook.mock_cursor.execute.assert_any_call(
+            'set airflow.ctx.dag_email=test@airflow.com')
+
     def test_get_pandas_df(self):
-        hook = HiveServer2Hook()
+        hook = MockHiveServer2Hook()
         query = "SELECT * FROM {}".format(self.table)
-        df = hook.get_pandas_df(query, schema=self.database)
+
+        with patch.dict('os.environ', {
+            'AIRFLOW_CTX_DAG_ID': 'test_dag_id',
+            'AIRFLOW_CTX_TASK_ID': 'HiveHook_3835',
+            'AIRFLOW_CTX_EXECUTION_DATE': '2015-01-01T00:00:00+00:00',
+            'AIRFLOW_CTX_DAG_RUN_ID': '55',
+            'AIRFLOW_CTX_DAG_OWNER': 'airflow',
+            'AIRFLOW_CTX_DAG_EMAIL': 'test@airflow.com',
+        }):
+            df = hook.get_pandas_df(query, schema=self.database)
+
         self.assertEqual(len(df), 2)
-        self.assertListEqual(df.columns.tolist(), self.columns)
-        self.assertListEqual(df[self.columns[0]].values.tolist(), [1, 2])
+        self.assertListEqual(df["hive_server_hook.a"].values.tolist(), [1, 2])
+
+        hook.get_conn.assert_called_with(self.database)
+        hook.mock_cursor.execute.assert_any_call(
+            'set airflow.ctx.dag_id=test_dag_id')
+        hook.mock_cursor.execute.assert_any_call(
+            'set airflow.ctx.task_id=HiveHook_3835')
+        hook.mock_cursor.execute.assert_any_call(
+            'set airflow.ctx.execution_date=2015-01-01T00:00:00+00:00')
+        hook.mock_cursor.execute.assert_any_call(
+            'set airflow.ctx.dag_run_id=55')
+        hook.mock_cursor.execute.assert_any_call(
+            'set airflow.ctx.dag_owner=airflow')
+        hook.mock_cursor.execute.assert_any_call(
+            'set airflow.ctx.dag_email=test@airflow.com')
 
     def test_get_results_header(self):
-        hook = HiveServer2Hook()
+        hook = MockHiveServer2Hook()
+
         query = "SELECT * FROM {}".format(self.table)
         results = hook.get_results(query, schema=self.database)
+
         self.assertListEqual([col[0] for col in results['header']],
                              self.columns)
 
     def test_get_results_data(self):
-        hook = HiveServer2Hook()
+        hook = MockHiveServer2Hook()
+
         query = "SELECT * FROM {}".format(self.table)
         results = hook.get_results(query, schema=self.database)
+
         self.assertListEqual(results['data'], [(1, 1), (2, 2)])
 
-    @unittest.skipIf(NOT_ASSERTLOGS_VERSION < 3.4, 'assertLogs not support before python 3.4')
-    def test_to_csv_assertlogs(self):
-        hook = HiveServer2Hook()
+    def test_to_csv(self):
+        hook = MockHiveServer2Hook()
+        hook._get_results = MagicMock(return_value=iter([
+            [
+                ('hive_server_hook.a', 'INT_TYPE', None, None, None, None, True),
+                ('hive_server_hook.b', 'INT_TYPE', None, None, None, None, True)
+            ], (1, 1), (2, 2)
+        ]))
         query = "SELECT * FROM {}".format(self.table)
         csv_filepath = 'query_results.csv'
         hook.to_csv(query, csv_filepath, schema=self.database,
@@ -480,28 +701,50 @@ class TestHiveServer2Hook(unittest.TestCase):
         self.assertListEqual(df[self.columns[0]].values.tolist(), [1, 2])
         self.assertEqual(len(df), 2)
 
-    @unittest.skipIf(NOT_ASSERTLOGS_VERSION >= 3.4, 'test could cover by test_to_csv_assertLogs')
-    def test_to_csv_without_assertlogs(self):
-        hook = HiveServer2Hook()
-        query = "SELECT * FROM {}".format(self.table)
-        csv_filepath = 'query_results.csv'
-        hook.to_csv(query, csv_filepath, schema=self.database,
-                    delimiter=',', lineterminator='\n', output_header=True)
-        df = pd.read_csv(csv_filepath, sep=',')
-        self.assertListEqual(df.columns.tolist(), self.columns)
-        self.assertListEqual(df[self.columns[0]].values.tolist(), [1, 2])
-        self.assertEqual(len(df), 2)
-
     def test_multi_statements(self):
         sqls = [
             "CREATE TABLE IF NOT EXISTS test_multi_statements (i INT)",
             "SELECT * FROM {}".format(self.table),
             "DROP TABLE test_multi_statements",
         ]
-        hook = HiveServer2Hook()
-        results = hook.get_records(sqls, schema=self.database)
+
+        hook = MockHiveServer2Hook()
+
+        with patch.dict('os.environ', {
+            'AIRFLOW_CTX_DAG_ID': 'test_dag_id',
+            'AIRFLOW_CTX_TASK_ID': 'HiveHook_3835',
+            'AIRFLOW_CTX_EXECUTION_DATE': '2015-01-01T00:00:00+00:00',
+            'AIRFLOW_CTX_DAG_RUN_ID': '55',
+            'AIRFLOW_CTX_DAG_OWNER': 'airflow',
+            'AIRFLOW_CTX_DAG_EMAIL': 'test@airflow.com',
+        }):
+            # df = hook.get_pandas_df(query, schema=self.database)
+            results = hook.get_records(sqls, schema=self.database)
         self.assertListEqual(results, [(1, 1), (2, 2)])
 
+        # self.assertEqual(len(df), 2)
+        # self.assertListEqual(df["hive_server_hook.a"].values.tolist(), [1, 2])
+
+        hook.get_conn.assert_called_with(self.database)
+        hook.mock_cursor.execute.assert_any_call(
+            'CREATE TABLE IF NOT EXISTS test_multi_statements (i INT)')
+        hook.mock_cursor.execute.assert_any_call(
+            'SELECT * FROM {}'.format(self.table))
+        hook.mock_cursor.execute.assert_any_call(
+            'DROP TABLE test_multi_statements')
+        hook.mock_cursor.execute.assert_any_call(
+            'set airflow.ctx.dag_id=test_dag_id')
+        hook.mock_cursor.execute.assert_any_call(
+            'set airflow.ctx.task_id=HiveHook_3835')
+        hook.mock_cursor.execute.assert_any_call(
+            'set airflow.ctx.execution_date=2015-01-01T00:00:00+00:00')
+        hook.mock_cursor.execute.assert_any_call(
+            'set airflow.ctx.dag_run_id=55')
+        hook.mock_cursor.execute.assert_any_call(
+            'set airflow.ctx.dag_owner=airflow')
+        hook.mock_cursor.execute.assert_any_call(
+            'set airflow.ctx.dag_email=test@airflow.com')
+
     def test_get_results_with_hive_conf(self):
         hql = ["set key",
                "set airflow.ctx.dag_id",
@@ -520,20 +763,46 @@ class TestHiveServer2Hook(unittest.TestCase):
             AIRFLOW_VAR_NAME_FORMAT_MAPPING['AIRFLOW_CONTEXT_DAG_RUN_ID'][
                 'env_var_format']
 
-        with mock.patch.dict('os.environ', {
+        with patch.dict('os.environ', {
             dag_id_ctx_var_name: 'test_dag_id',
             task_id_ctx_var_name: 'test_task_id',
             execution_date_ctx_var_name: 'test_execution_date',
             dag_run_id_ctx_var_name: 'test_dag_run_id',
 
         }):
-            hook = HiveServer2Hook()
-            output = '\n'.join(res_tuple[0]
-                               for res_tuple
-                               in hook.get_results(hql=hql,
-                                                   hive_conf={'key': 'value'})['data'])
+            hook = MockHiveServer2Hook()
+            hook._get_results = MagicMock(return_value=iter(
+                ["header", ("value", "test"), ("test_dag_id", "test"), ("test_task_id", "test"),
+                 ("test_execution_date", "test"), ("test_dag_run_id", "test")]
+            ))
+
+            output = '\n'.join(res_tuple[0] for res_tuple in hook.get_results(
+                hql=hql, hive_conf={'key': 'value'})['data'])
         self.assertIn('value', output)
         self.assertIn('test_dag_id', output)
         self.assertIn('test_task_id', output)
         self.assertIn('test_execution_date', output)
         self.assertIn('test_dag_run_id', output)
+
+
+class TestHiveCli(unittest.TestCase):
+
+    def setUp(self):
+        self.nondefault_schema = "nondefault"
+        os.environ["AIRFLOW__CORE__SECURITY"] = "kerberos"
+
+    def tearDown(self):
+        del os.environ["AIRFLOW__CORE__SECURITY"]
+
+    def test_get_proxy_user_value(self):
+        hook = MockHiveCliHook()
+        returner = MagicMock()
+        returner.extra_dejson = {'proxy_user': 'a_user_proxy'}
+        hook.use_beeline = True
+        hook.conn = returner
+
+        # Run
+        result = hook._prepare_cli_cmd()
+
+        # Verify
+        self.assertIn('hive.server2.proxy.user=a_user_proxy', result[2])
diff --git a/tests/operators/test_hive_operator.py b/tests/operators/test_hive_operator.py
index 858cab1..fcdc99e 100644
--- a/tests/operators/test_hive_operator.py
+++ b/tests/operators/test_hive_operator.py
@@ -23,80 +23,26 @@ import datetime
 import os
 import unittest
 
+from tests.hive import TestHiveEnvironment
+from tests.test_utils.mock_process import MockSubProcess
+
+from airflow.models import TaskInstance
+from airflow.utils import timezone
 from airflow.operators.hive_operator import HiveOperator
-from airflow.operators.hive_stats_operator import HiveStatsCollectionOperator
-from airflow.operators.hive_to_mysql import HiveToMySqlTransfer
-from airflow.operators.hive_to_samba_operator import Hive2SambaOperator
-from airflow.operators.presto_check_operator import PrestoCheckOperator
-from airflow.operators.presto_to_mysql import PrestoToMySqlTransfer
-from airflow.sensors.hdfs_sensor import HdfsSensor
-from airflow.sensors.hive_partition_sensor import HivePartitionSensor
-from airflow.sensors.metastore_partition_sensor import MetastorePartitionSensor
-from airflow.sensors.named_hive_partition_sensor import NamedHivePartitionSensor
-from airflow.sensors.sql_sensor import SqlSensor
-from airflow.sensors.web_hdfs_sensor import WebHdfsSensor
 from tests.compat import mock
 
-from airflow import DAG
 from airflow.configuration import conf
-from airflow.exceptions import AirflowSensorTimeout
+from tests.test_utils.mock_operators import MockHiveOperator
 
 DEFAULT_DATE = datetime.datetime(2015, 1, 1)
 DEFAULT_DATE_ISO = DEFAULT_DATE.isoformat()
 DEFAULT_DATE_DS = DEFAULT_DATE_ISO[:10]
 
 
-class HiveEnvironmentTest(unittest.TestCase):
-
-    def setUp(self):
-        args = {'owner': 'airflow', 'start_date': DEFAULT_DATE}
-        dag = DAG('test_dag_id', default_args=args)
-        self.dag = dag
-        self.hql = """
-        USE airflow;
-        DROP TABLE IF EXISTS static_babynames_partitioned;
-        CREATE TABLE IF NOT EXISTS static_babynames_partitioned (
-            state string,
-            year string,
-            name string,
-            gender string,
-            num int)
-        PARTITIONED BY (ds string);
-        INSERT OVERWRITE TABLE static_babynames_partitioned
-            PARTITION(ds='{{ ds }}')
-        SELECT state, year, name, gender, num FROM static_babynames;
-        """
-
-
-class HiveCliTest(unittest.TestCase):
-
-    def setUp(self):
-        self.nondefault_schema = "nondefault"
-        os.environ["AIRFLOW__CORE__SECURITY"] = "kerberos"
-
-    def tearDown(self):
-        del os.environ["AIRFLOW__CORE__SECURITY"]
-
-    def test_get_proxy_user_value(self):
-        from airflow.hooks.hive_hooks import HiveCliHook
-
-        hook = HiveCliHook()
-        returner = mock.MagicMock()
-        returner.extra_dejson = {'proxy_user': 'a_user_proxy'}
-        hook.use_beeline = True
-        hook.conn = returner
-
-        # Run
-        result = hook._prepare_cli_cmd()
-
-        # Verify
-        self.assertIn('hive.server2.proxy.user=a_user_proxy', result[2])
-
-
-class HiveOperatorConfigTest(HiveEnvironmentTest):
+class HiveOperatorConfigTest(TestHiveEnvironment):
 
     def test_hive_airflow_default_config_queue(self):
-        t = HiveOperator(
+        op = MockHiveOperator(
             task_id='test_default_config_queue',
             hql=self.hql,
             mapred_queue_priority='HIGH',
@@ -108,11 +54,12 @@ class HiveOperatorConfigTest(HiveEnvironmentTest):
             'hive',
             'default_hive_mapred_queue'
         )
-        self.assertEqual(t.get_hook().mapred_queue, test_config_hive_mapred_queue)
+        self.assertEqual(op.get_hook().mapred_queue,
+                         test_config_hive_mapred_queue)
 
     def test_hive_airflow_default_config_queue_override(self):
         specific_mapred_queue = 'default'
-        t = HiveOperator(
+        op = MockHiveOperator(
             task_id='test_default_config_queue',
             hql=self.hql,
             mapred_queue=specific_mapred_queue,
@@ -120,204 +67,170 @@ class HiveOperatorConfigTest(HiveEnvironmentTest):
             mapred_job_name='airflow.test_default_config_queue',
             dag=self.dag)
 
-        self.assertEqual(t.get_hook().mapred_queue, specific_mapred_queue)
+        self.assertEqual(op.get_hook().mapred_queue, specific_mapred_queue)
 
 
-class HiveOperatorTest(HiveEnvironmentTest):
+class HiveOperatorTest(TestHiveEnvironment):
 
     def test_hiveconf_jinja_translate(self):
         hql = "SELECT ${num_col} FROM ${hiveconf:table};"
-        t = HiveOperator(
+        op = MockHiveOperator(
             hiveconf_jinja_translate=True,
             task_id='dry_run_basic_hql', hql=hql, dag=self.dag)
-        t.prepare_template()
-        self.assertEqual(t.hql, "SELECT {{ num_col }} FROM {{ table }};")
+        op.prepare_template()
+        self.assertEqual(op.hql, "SELECT {{ num_col }} FROM {{ table }};")
 
     def test_hiveconf(self):
         hql = "SELECT * FROM ${hiveconf:table} PARTITION (${hiveconf:day});"
-        t = HiveOperator(
+        op = MockHiveOperator(
             hiveconfs={'table': 'static_babynames', 'day': '{{ ds }}'},
             task_id='dry_run_basic_hql', hql=hql, dag=self.dag)
-        t.prepare_template()
+        op.prepare_template()
         self.assertEqual(
-            t.hql,
+            op.hql,
             "SELECT * FROM ${hiveconf:table} PARTITION (${hiveconf:day});")
 
+    @mock.patch('airflow.operators.hive_operator.HiveOperator.get_hook')
+    def test_mapred_job_name(self, mock_get_hook):
+        mock_hook = mock.MagicMock()
+        mock_get_hook.return_value = mock_hook
+        op = MockHiveOperator(
+            task_id='test_mapred_job_name',
+            hql=self.hql,
+            dag=self.dag)
+
+        fake_execution_date = timezone.datetime(2018, 6, 19)
+        fake_ti = TaskInstance(task=op, execution_date=fake_execution_date)
+        fake_ti.hostname = 'fake_hostname'
+        fake_context = {'ti': fake_ti}
+
+        op.execute(fake_context)
+        self.assertEqual(
+            "Airflow HiveOperator task for {}.{}.{}.{}".format(
+                fake_ti.hostname,
+                self.dag.dag_id, op.task_id,
+                fake_execution_date.isoformat()), mock_hook.mapred_job_name)
+
+
+@unittest.skipIf(
+    'AIRFLOW_RUNALL_TESTS' not in os.environ,
+    "Skipped because AIRFLOW_RUNALL_TESTS is not set")
+class TestHivePresto(TestHiveEnvironment):
+
+    @mock.patch('tempfile.tempdir', '/tmp/')
+    @mock.patch('tempfile._RandomNameSequence.__next__')
+    @mock.patch('subprocess.Popen')
+    def test_hive(self, mock_popen, mock_temp_dir):
+        mock_subprocess = MockSubProcess()
+        mock_popen.return_value = mock_subprocess
+        mock_temp_dir.return_value = "tst"
+        op = HiveOperator(
+            task_id='basic_hql', hql=self.hql, dag=self.dag, mapred_job_name="test_job_name")
+
+        op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
+               ignore_ti_state=True)
+        hive_cmd = ['beeline', '-u', '"jdbc:hive2://localhost:10000/default"', '-hiveconf',
+                    'airflow.ctx.dag_id=test_dag_id', '-hiveconf', 'airflow.ctx.task_id=basic_hql',
+                    '-hiveconf', 'airflow.ctx.execution_date=2015-01-01T00:00:00+00:00', '-hiveconf',
+                    'airflow.ctx.dag_run_id=', '-hiveconf', 'airflow.ctx.dag_owner=airflow', '-hiveconf',
+                    'airflow.ctx.dag_email=', '-hiveconf', 'mapreduce.job.queuename=airflow', '-hiveconf',
+                    'mapred.job.queue.name=airflow', '-hiveconf', 'tez.queue.name=airflow', '-hiveconf',
+                    'mapred.job.name=test_job_name', '-f', '/tmp/airflow_hiveop_tst/tmptst']
+
+        mock_popen.assert_called_with(
+            hive_cmd,
+            stdout=mock_subprocess.PIPE,
+            stderr=mock_subprocess.STDOUT,
+            cwd="/tmp/airflow_hiveop_tst",
+            close_fds=True
+        )
 
-if 'AIRFLOW_RUNALL_TESTS' in os.environ:
-
-    class HivePrestoTest(HiveEnvironmentTest):
-
-        def test_hive(self):
-            t = HiveOperator(
-                task_id='basic_hql', hql=self.hql, dag=self.dag)
-            t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
-                  ignore_ti_state=True)
-
-        def test_hive_queues(self):
-            t = HiveOperator(
-                task_id='test_hive_queues', hql=self.hql,
-                mapred_queue='default', mapred_queue_priority='HIGH',
-                mapred_job_name='airflow.test_hive_queues',
-                dag=self.dag)
-            t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
-                  ignore_ti_state=True)
-
-        def test_hive_dryrun(self):
-            t = HiveOperator(
-                task_id='dry_run_basic_hql', hql=self.hql, dag=self.dag)
-            t.dry_run()
-
-        def test_beeline(self):
-            t = HiveOperator(
-                task_id='beeline_hql', hive_cli_conn_id='beeline_default',
-                hql=self.hql, dag=self.dag)
-            t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
-                  ignore_ti_state=True)
-
-        def test_presto(self):
-            sql = """
-            SELECT count(1) FROM airflow.static_babynames_partitioned;
-            """
-            t = PrestoCheckOperator(
-                task_id='presto_check', sql=sql, dag=self.dag)
-            t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
-                  ignore_ti_state=True)
-
-        def test_presto_to_mysql(self):
-            t = PrestoToMySqlTransfer(
-                task_id='presto_to_mysql_check',
-                sql="""
-                SELECT name, count(*) as ccount
-                FROM airflow.static_babynames
-                GROUP BY name
-                """,
-                mysql_table='test_static_babynames',
-                mysql_preoperator='TRUNCATE TABLE test_static_babynames;',
-                dag=self.dag)
-            t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
-                  ignore_ti_state=True)
-
-        def test_hdfs_sensor(self):
-            t = HdfsSensor(
-                task_id='hdfs_sensor_check',
-                filepath='hdfs://user/hive/warehouse/airflow.db/static_babynames',
-                dag=self.dag)
-            t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
-                  ignore_ti_state=True)
-
-        def test_webhdfs_sensor(self):
-            t = WebHdfsSensor(
-                task_id='webhdfs_sensor_check',
-                filepath='hdfs://user/hive/warehouse/airflow.db/static_babynames',
-                timeout=120,
-                dag=self.dag)
-            t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
-                  ignore_ti_state=True)
-
-        def test_sql_sensor(self):
-            t = SqlSensor(
-                task_id='hdfs_sensor_check',
-                conn_id='presto_default',
-                sql="SELECT 'x' FROM airflow.static_babynames LIMIT 1;",
-                dag=self.dag)
-            t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
-                  ignore_ti_state=True)
-
-        def test_hive_stats(self):
-            t = HiveStatsCollectionOperator(
-                task_id='hive_stats_check',
-                table="airflow.static_babynames_partitioned",
-                partition={'ds': DEFAULT_DATE_DS},
-                dag=self.dag)
-            t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
-                  ignore_ti_state=True)
-
-        def test_named_hive_partition_sensor(self):
-            t = NamedHivePartitionSensor(
-                task_id='hive_partition_check',
-                partition_names=[
-                    "airflow.static_babynames_partitioned/ds={{ds}}"
-                ],
-                dag=self.dag)
-            t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
-                  ignore_ti_state=True)
-
-        def test_named_hive_partition_sensor_succeeds_on_multiple_partitions(self):
-            t = NamedHivePartitionSensor(
-                task_id='hive_partition_check',
-                partition_names=[
-                    "airflow.static_babynames_partitioned/ds={{ds}}",
-                    "airflow.static_babynames_partitioned/ds={{ds}}"
-                ],
-                dag=self.dag)
-            t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
-                  ignore_ti_state=True)
-
-        def test_named_hive_partition_sensor_parses_partitions_with_periods(self):
-            t = NamedHivePartitionSensor.parse_partition_name(
-                partition="schema.table/part1=this.can.be.an.issue/part2=ok")
-            self.assertEqual(t[0], "schema")
-            self.assertEqual(t[1], "table")
-            self.assertEqual(t[2], "part1=this.can.be.an.issue/part2=this_should_be_ok")
-
-        def test_named_hive_partition_sensor_times_out_on_nonexistent_partition(self):
-            with self.assertRaises(AirflowSensorTimeout):
-                t = NamedHivePartitionSensor(
-                    task_id='hive_partition_check',
-                    partition_names=[
-                        "airflow.static_babynames_partitioned/ds={{ds}}",
-                        "airflow.static_babynames_partitioned/ds=nonexistent"
-                    ],
-                    poke_interval=0.1,
-                    timeout=1,
-                    dag=self.dag)
-                t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
-                      ignore_ti_state=True)
-
-        def test_hive_partition_sensor(self):
-            t = HivePartitionSensor(
-                task_id='hive_partition_check',
-                table='airflow.static_babynames_partitioned',
-                dag=self.dag)
-            t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
-                  ignore_ti_state=True)
-
-        def test_hive_metastore_sql_sensor(self):
-            t = MetastorePartitionSensor(
-                task_id='hive_partition_check',
-                table='airflow.static_babynames_partitioned',
-                partition_name='ds={}'.format(DEFAULT_DATE_DS),
-                dag=self.dag)
-            t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
-                  ignore_ti_state=True)
-
-        def test_hive2samba(self):
-            t = Hive2SambaOperator(
-                task_id='hive2samba_check',
-                samba_conn_id='tableau_samba',
-                hql="SELECT * FROM airflow.static_babynames LIMIT 10000",
-                destination_filepath='test_airflow.csv',
-                dag=self.dag)
-            t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
-                  ignore_ti_state=True)
-
-        def test_hive_to_mysql(self):
-            t = HiveToMySqlTransfer(
-                mysql_conn_id='airflow_db',
-                task_id='hive_to_mysql_check',
-                create=True,
-                sql="""
-                SELECT name
-                FROM airflow.static_babynames
-                LIMIT 100
-                """,
-                mysql_table='test_static_babynames',
-                mysql_preoperator=[
-                    'DROP TABLE IF EXISTS test_static_babynames;',
-                    'CREATE TABLE test_static_babynames (name VARCHAR(500))',
-                ],
-                dag=self.dag)
-            t.clear(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
-            t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
-                  ignore_ti_state=True)
+    @mock.patch('tempfile.tempdir', '/tmp/')
+    @mock.patch('tempfile._RandomNameSequence.__next__')
+    @mock.patch('subprocess.Popen')
+    def test_hive_queues(self, mock_popen, mock_temp_dir):
+        mock_subprocess = MockSubProcess()
+        mock_popen.return_value = mock_subprocess
+        mock_temp_dir.return_value = "tst"
+
+        hive_cmd = ['beeline', '-u', '"jdbc:hive2://localhost:10000/default"', '-hiveconf',
+                    'airflow.ctx.dag_id=test_dag_id', '-hiveconf', 'airflow.ctx.task_id=test_hive_queues',
+                    '-hiveconf', 'airflow.ctx.execution_date=2015-01-01T00:00:00+00:00', '-hiveconf',
+                    'airflow.ctx.dag_run_id=', '-hiveconf', 'airflow.ctx.dag_owner=airflow',
+                    '-hiveconf', 'airflow.ctx.dag_email=', '-hiveconf', 'mapreduce.job.queuename=default',
+                    '-hiveconf', 'mapred.job.queue.name=default', '-hiveconf', 'tez.queue.name=default',
+                    '-hiveconf', 'mapreduce.job.priority=HIGH', '-hiveconf',
+                    'mapred.job.name=airflow.test_hive_queues', '-f', '/tmp/airflow_hiveop_tst/tmptst']
+        op = HiveOperator(
+            task_id='test_hive_queues', hql=self.hql,
+            mapred_queue='default', mapred_queue_priority='HIGH',
+            mapred_job_name='airflow.test_hive_queues',
+            dag=self.dag)
+        op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
+               ignore_ti_state=True)
+
+        mock_popen.assert_called_with(
+            hive_cmd,
+            stdout=mock_subprocess.PIPE,
+            stderr=mock_subprocess.STDOUT,
+            cwd="/tmp/airflow_hiveop_tst",
+            close_fds=True
+        )
+
+    @mock.patch('tempfile.tempdir', '/tmp/')
+    @mock.patch('tempfile._RandomNameSequence.__next__')
+    @mock.patch('subprocess.Popen')
+    def test_hive_dryrun(self, mock_popen, mock_temp_dir):
+        mock_subprocess = MockSubProcess()
+        mock_popen.return_value = mock_subprocess
+        mock_temp_dir.return_value = "tst"
+
+        op = HiveOperator(task_id='dry_run_basic_hql',
+                          hql=self.hql, dag=self.dag)
+        op.dry_run()
+
+        hive_cmd = ['beeline', '-u', '"jdbc:hive2://localhost:10000/default"',
+                    '-hiveconf', 'airflow.ctx.dag_id=', '-hiveconf', 'airflow.ctx.task_id=',
+                    '-hiveconf', 'airflow.ctx.execution_date=', '-hiveconf', 'airflow.ctx.dag_run_id=',
+                    '-hiveconf', 'airflow.ctx.dag_owner=', '-hiveconf', 'airflow.ctx.dag_email=',
+                    '-hiveconf', 'mapreduce.job.queuename=airflow', '-hiveconf',
+                    'mapred.job.queue.name=airflow', '-hiveconf', 'tez.queue.name=airflow',
+                    '-f', '/tmp/airflow_hiveop_tst/tmptst']
+        mock_popen.assert_called_with(
+            hive_cmd,
+            stdout=mock_subprocess.PIPE,
+            stderr=mock_subprocess.STDOUT,
+            cwd="/tmp/airflow_hiveop_tst",
+            close_fds=True
+        )
+
+    @mock.patch('tempfile.tempdir', '/tmp/')
+    @mock.patch('tempfile._RandomNameSequence.__next__')
+    @mock.patch('subprocess.Popen')
+    def test_beeline(self, mock_popen, mock_temp_dir):
+        mock_subprocess = MockSubProcess()
+        mock_popen.return_value = mock_subprocess
+        mock_temp_dir.return_value = "tst"
+
+        hive_cmd = ['beeline', '-u', '"jdbc:hive2://localhost:10000/default"',
+                    '-hiveconf', 'airflow.ctx.dag_id=test_dag_id', '-hiveconf',
+                    'airflow.ctx.task_id=beeline_hql', '-hiveconf',
+                    'airflow.ctx.execution_date=2015-01-01T00:00:00+00:00',
+                    '-hiveconf', 'airflow.ctx.dag_run_id=', '-hiveconf', 'airflow.ctx.dag_owner=airflow',
+                    '-hiveconf', 'airflow.ctx.dag_email=',
+                    '-hiveconf', 'mapreduce.job.queuename=airflow', '-hiveconf',
+                    'mapred.job.queue.name=airflow', '-hiveconf', 'tez.queue.name=airflow', '-hiveconf',
+                    'mapred.job.name=test_job_name', '-f', '/tmp/airflow_hiveop_tst/tmptst']
+
+        op = HiveOperator(
+            task_id='beeline_hql', hive_cli_conn_id='hive_cli_default',
+            hql=self.hql, dag=self.dag, mapred_job_name="test_job_name")
+        op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
+               ignore_ti_state=True)
+        mock_popen.assert_called_with(
+            hive_cmd,
+            stdout=mock_subprocess.PIPE,
+            stderr=mock_subprocess.STDOUT,
+            cwd="/tmp/airflow_hiveop_tst",
+            close_fds=True
+        )
diff --git a/tests/operators/test_operators.py b/tests/operators/test_operators.py
index db9d73b..b693a08 100644
--- a/tests/operators/test_operators.py
+++ b/tests/operators/test_operators.py
@@ -19,17 +19,26 @@
 
 from __future__ import print_function
 
+from collections import OrderedDict
+from random import random
+
 import pytest
 
+from six import PY2
+
 from airflow import DAG, operators
 from airflow.utils import timezone
 
-from collections import OrderedDict
+from airflow.hooks.mysql_hook import MySqlHook
+from airflow.operators.mysql_to_hive import MySqlToHiveTransfer
+from tests.compat import mock, patch
 
 import os
-import mock
 import unittest
 
+from tests.test_utils.mock_hooks import MockHiveServer2Hook
+from tests.test_utils.mock_process import MockSubProcess, MockConnectionCursor
+
 DEFAULT_DATE = timezone.datetime(2015, 1, 1)
 DEFAULT_DATE_ISO = DEFAULT_DATE.isoformat()
 DEFAULT_DATE_DS = DEFAULT_DATE_ISO[:10]
@@ -108,7 +117,7 @@ class TestMySql(unittest.TestCase):
         priv = hook.get_first("SELECT @@global.secure_file_priv")
         if priv and priv[0]:
             # Confirm that no error occurs
-            hook.bulk_dump("INFORMATION_SCHEMA.TABLES", os.path.join(priv[0], "TABLES"))
+            hook.bulk_dump("INFORMATION_SCHEMA.TABLES", os.path.join(priv[0], "TABLES" + str(random())))
         else:
             self.skipTest("Skip test_mysql_hook_test_bulk_load "
                           "since file output is not permitted")
@@ -334,6 +343,15 @@ class TestTransfer(unittest.TestCase):
             (1880, "Eugene", 0.00277, "boy"),
         ]
 
+        self.env_vars = {
+            'AIRFLOW_CTX_DAG_ID': 'test_dag_id',
+            'AIRFLOW_CTX_TASK_ID': 'test_task_id',
+            'AIRFLOW_CTX_EXECUTION_DATE': '2015-01-01T00:00:00+00:00',
+            'AIRFLOW_CTX_DAG_RUN_ID': '55',
+            'AIRFLOW_CTX_DAG_OWNER': 'airflow',
+            'AIRFLOW_CTX_DAG_EMAIL': 'test@airflow.com',
+        }
+
         from airflow.hooks.mysql_hook import MySqlHook
         with MySqlHook().get_conn() as cur:
             cur.execute('''
@@ -358,47 +376,120 @@ class TestTransfer(unittest.TestCase):
             start_date=DEFAULT_DATE,
             end_date=timezone.utcnow())
 
-    def test_mysql_to_hive(self):
-        from airflow.operators.mysql_to_hive import MySqlToHiveTransfer
-        sql = "SELECT * FROM baby_names LIMIT 1000;"
-        t = MySqlToHiveTransfer(
-            task_id='test_m2h',
-            hive_cli_conn_id='beeline_default',
-            sql=sql,
-            hive_table='test_mysql_to_hive',
-            recreate=True,
-            delimiter=",",
-            dag=self.dag)
-        t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
+    @patch('tempfile.tempdir', '/tmp/')
+    @patch('tempfile._RandomNameSequence.next' if PY2 else 'tempfile._RandomNameSequence.__next__')
+    @patch('subprocess.Popen')
+    def test_mysql_to_hive(self, mock_popen, mock_temp_dir):
+        mock_subprocess = MockSubProcess()
+        mock_popen.return_value = mock_subprocess
+        mock_temp_dir.return_value = "test_mysql_to_hive"
 
-    def test_mysql_to_hive_partition(self):
-        from airflow.operators.mysql_to_hive import MySqlToHiveTransfer
-        sql = "SELECT * FROM baby_names LIMIT 1000;"
-        t = MySqlToHiveTransfer(
-            task_id='test_m2h',
-            hive_cli_conn_id='beeline_default',
-            sql=sql,
-            hive_table='test_mysql_to_hive_part',
-            partition={'ds': DEFAULT_DATE_DS},
-            recreate=False,
-            create=True,
-            delimiter=",",
-            dag=self.dag)
-        t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
+        with mock.patch.dict('os.environ', self.env_vars):
+            sql = "SELECT * FROM baby_names LIMIT 1000;"
+            t = MySqlToHiveTransfer(
+                task_id='test_m2h',
+                hive_cli_conn_id='beeline_default',
+                sql=sql,
+                hive_table='test_mysql_to_hive',
+                recreate=True,
+                delimiter=",",
+                dag=self.dag)
+            t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
 
-    def test_mysql_to_hive_tblproperties(self):
-        from airflow.operators.mysql_to_hive import MySqlToHiveTransfer
-        sql = "SELECT * FROM baby_names LIMIT 1000;"
-        t = MySqlToHiveTransfer(
-            task_id='test_m2h',
-            hive_cli_conn_id='beeline_default',
-            sql=sql,
-            hive_table='test_mysql_to_hive',
-            recreate=True,
-            delimiter=",",
-            tblproperties={'test_property': 'test_value'},
-            dag=self.dag)
-        t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
+        hive_cmd = [u'beeline', u'-u', u'"jdbc:hive2://localhost:10000/default"', u'-hiveconf',
+                    u'airflow.ctx.dag_email=test@airflow.com', u'-hiveconf',
+                    u'airflow.ctx.dag_id=test_dag_id',
+                    u'-hiveconf', u'airflow.ctx.dag_owner=airflow', u'-hiveconf',
+                    u'airflow.ctx.dag_run_id=55',
+                    u'-hiveconf', u'airflow.ctx.execution_date=2015-01-01T00:00:00+00:00', u'-hiveconf',
+                    u'airflow.ctx.task_id=test_task_id', u'-hiveconf', u'mapreduce.job.queuename=airflow',
+                    u'-hiveconf', u'mapred.job.queue.name=airflow', u'-hiveconf', u'tez.queue.name=airflow',
+                    u'-f', u'/tmp/airflow_hiveop_test_mysql_to_hive/tmptest_mysql_to_hive']
+
+        mock_popen.assert_called_with(
+            hive_cmd,
+            stdout=mock_subprocess.PIPE,
+            stderr=mock_subprocess.STDOUT,
+            cwd=u"/tmp/airflow_hiveop_test_mysql_to_hive",
+            close_fds=True
+        )
+
+    @patch('tempfile.tempdir', '/tmp/')
+    @patch('tempfile._RandomNameSequence.next' if PY2 else 'tempfile._RandomNameSequence.__next__')
+    @patch('subprocess.Popen')
+    def test_mysql_to_hive_partition(self, mock_popen, mock_temp_dir):
+        mock_subprocess = MockSubProcess()
+        mock_popen.return_value = mock_subprocess
+        mock_temp_dir.return_value = "test_mysql_to_hive_part"
+
+        with mock.patch.dict('os.environ', self.env_vars):
+            sql = "SELECT * FROM baby_names LIMIT 1000;"
+            t = MySqlToHiveTransfer(
+                task_id='test_m2h',
+                hive_cli_conn_id='beeline_default',
+                sql=sql,
+                hive_table='test_mysql_to_hive_part',
+                partition={'ds': DEFAULT_DATE_DS},
+                recreate=False,
+                create=True,
+                delimiter=",",
+                dag=self.dag)
+            t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
+        hive_cmd = [u'beeline', u'-u', u'"jdbc:hive2://localhost:10000/default"', u'-hiveconf',
+                    u'airflow.ctx.dag_email=test@airflow.com', u'-hiveconf',
+                    u'airflow.ctx.dag_id=test_dag_id', u'-hiveconf', u'airflow.ctx.dag_owner=airflow',
+                    u'-hiveconf', u'airflow.ctx.dag_run_id=55', u'-hiveconf',
+                    u'airflow.ctx.execution_date=2015-01-01T00:00:00+00:00', u'-hiveconf',
+                    u'airflow.ctx.task_id=test_task_id', u'-hiveconf', u'mapreduce.job.queuename=airflow',
+                    u'-hiveconf', u'mapred.job.queue.name=airflow', u'-hiveconf', u'tez.queue.name=airflow',
+                    u'-f', u'/tmp/airflow_hiveop_test_mysql_to_hive_part/tmptest_mysql_to_hive_part']
+
+        mock_popen.assert_called_with(
+            hive_cmd,
+            stdout=mock_subprocess.PIPE,
+            stderr=mock_subprocess.STDOUT,
+            cwd=u"/tmp/airflow_hiveop_test_mysql_to_hive_part",
+            close_fds=True
+        )
+
+    @patch('tempfile.tempdir', '/tmp/')
+    @patch('tempfile._RandomNameSequence.next' if PY2 else 'tempfile._RandomNameSequence.__next__')
+    @patch('subprocess.Popen')
+    def test_mysql_to_hive_tblproperties(self, mock_popen, mock_temp_dir):
+        mock_subprocess = MockSubProcess()
+        mock_popen.return_value = mock_subprocess
+        mock_temp_dir.return_value = "test_mysql_to_hive_tblproperties"
+        with mock.patch.dict('os.environ', self.env_vars):
+            sql = "SELECT * FROM baby_names LIMIT 1000;"
+            t = MySqlToHiveTransfer(
+                task_id='test_m2h',
+                hive_cli_conn_id='beeline_default',
+                sql=sql,
+                hive_table='test_mysql_to_hive_tblproperties',
+                recreate=True,
+                delimiter=",",
+                tblproperties={'test_property': 'test_value'},
+                dag=self.dag)
+            t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
+
+        hive_cmd = [u'beeline', u'-u', u'"jdbc:hive2://localhost:10000/default"', u'-hiveconf',
+                    u'airflow.ctx.dag_email=test@airflow.com', u'-hiveconf',
+                    u'airflow.ctx.dag_id=test_dag_id', u'-hiveconf', u'airflow.ctx.dag_owner=airflow',
+                    u'-hiveconf', u'airflow.ctx.dag_run_id=55', u'-hiveconf',
+                    u'airflow.ctx.execution_date=2015-01-01T00:00:00+00:00', u'-hiveconf',
+                    u'airflow.ctx.task_id=test_task_id', u'-hiveconf', u'mapreduce.job.queuename=airflow',
+                    u'-hiveconf', u'mapred.job.queue.name=airflow', u'-hiveconf', u'tez.queue.name=airflow',
+                    u'-f',
+                    u'/tmp/airflow_hiveop_test_mysql_to_hive_tblproperties/'
+                    u'tmptest_mysql_to_hive_tblproperties']
+
+        mock_popen.assert_called_with(
+            hive_cmd,
+            stdout=mock_subprocess.PIPE,
+            stderr=mock_subprocess.STDOUT,
+            cwd=u"/tmp/airflow_hiveop_test_mysql_to_hive_tblproperties",
+            close_fds=True
+        )
 
     @mock.patch('airflow.hooks.hive_hooks.HiveCliHook.load_file')
     def test_mysql_to_hive_type_conversion(self, mock_load_file):
@@ -443,12 +534,18 @@ class TestTransfer(unittest.TestCase):
             with m.get_conn() as c:
                 c.execute("DROP TABLE IF EXISTS {}".format(mysql_table))
 
-    def test_mysql_to_hive_verify_loaded_values(self):
+    @mock.patch('tempfile.tempdir', '/tmp/')
+    @patch('tempfile._RandomNameSequence.next' if PY2 else 'tempfile._RandomNameSequence.__next__')
+    @mock.patch('subprocess.Popen')
+    def test_mysql_to_hive_verify_loaded_values(self, mock_popen, mock_temp_dir):
+        mock_subprocess = MockSubProcess()
+        mock_popen.return_value = mock_subprocess
+        mock_temp_dir.return_value = "test_mysql_to_hive"
+
         mysql_table = 'test_mysql_to_hive'
         hive_table = 'test_mysql_to_hive'
 
-        from airflow.hooks.mysql_hook import MySqlHook
-        m = MySqlHook()
+        hook = MySqlHook()
 
         try:
             minmax = (
@@ -464,9 +561,9 @@ class TestTransfer(unittest.TestCase):
                 -9223372036854775808
             )
 
-            with m.get_conn() as c:
-                c.execute("DROP TABLE IF EXISTS {}".format(mysql_table))
-                c.execute("""
+            with hook.get_conn() as conn:
+                conn.execute("DROP TABLE IF EXISTS {}".format(mysql_table))
+                conn.execute("""
                     CREATE TABLE {} (
                         c0 TINYINT   UNSIGNED,
                         c1 SMALLINT  UNSIGNED,
@@ -480,27 +577,49 @@ class TestTransfer(unittest.TestCase):
                         c9 BIGINT
                     )
                 """.format(mysql_table))
-                c.execute("""
+                conn.execute("""
                     INSERT INTO {} VALUES (
                         {}, {}, {}, {}, {}, {}, {}, {}, {}, {}
                     )
                 """.format(mysql_table, *minmax))
 
-            from airflow.operators.mysql_to_hive import MySqlToHiveTransfer
-            t = MySqlToHiveTransfer(
-                task_id='test_m2h',
-                hive_cli_conn_id='beeline_default',
-                sql="SELECT * FROM {}".format(mysql_table),
-                hive_table=hive_table,
-                recreate=True,
-                delimiter=",",
-                dag=self.dag)
-            t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
+            with mock.patch.dict('os.environ', self.env_vars):
+                op = MySqlToHiveTransfer(
+                    task_id='test_m2h',
+                    hive_cli_conn_id='hive_cli_default',
+                    sql="SELECT * FROM {}".format(mysql_table),
+                    hive_table=hive_table,
+                    recreate=True,
+                    delimiter=",",
+                    dag=self.dag)
+                op.run(start_date=DEFAULT_DATE,
+                       end_date=DEFAULT_DATE, ignore_ti_state=True)
+
+                mock_cursor = MockConnectionCursor()
+                mock_cursor.iterable = [minmax]
+                hive_hook = MockHiveServer2Hook(connection_cursor=mock_cursor)
+
+                result = hive_hook.get_records(
+                    "SELECT * FROM {}".format(hive_table))
+                self.assertEqual(result[0], minmax)
+
+                hive_cmd = [u'hive', u'-hiveconf', u'airflow.ctx.dag_email=test@airflow.com', u'-hiveconf',
+                            u'airflow.ctx.dag_id=test_dag_id', u'-hiveconf', u'airflow.ctx.dag_owner=airflow',
+                            u'-hiveconf', u'airflow.ctx.dag_run_id=55', u'-hiveconf',
+                            u'airflow.ctx.execution_date=2015-01-01T00:00:00+00:00', u'-hiveconf',
+                            u'airflow.ctx.task_id=test_task_id', u'-hiveconf',
+                            u'mapreduce.job.queuename=airflow', u'-hiveconf',
+                            u'mapred.job.queue.name=airflow', u'-hiveconf', u'tez.queue.name=airflow', u'-f',
+                            u'/tmp/airflow_hiveop_test_mysql_to_hive/tmptest_mysql_to_hive']
+
+                mock_popen.assert_called_with(
+                    hive_cmd,
+                    stdout=mock_subprocess.PIPE,
+                    stderr=mock_subprocess.STDOUT,
+                    cwd="/tmp/airflow_hiveop_test_mysql_to_hive",
+                    close_fds=True
+                )
 
-            from airflow.hooks.hive_hooks import HiveServer2Hook
-            h = HiveServer2Hook()
-            r = h.get_records("SELECT * FROM {}".format(hive_table))
-            self.assertEqual(r[0], minmax)
         finally:
-            with m.get_conn() as c:
-                c.execute("DROP TABLE IF EXISTS {}".format(mysql_table))
+            with hook.get_conn() as conn:
+                conn.execute("DROP TABLE IF EXISTS {}".format(mysql_table))
diff --git a/tests/sensors/test_named_hive_partition_sensor.py b/tests/sensors/test_named_hive_partition_sensor.py
index ef7ba13..5fe9dd9 100644
--- a/tests/sensors/test_named_hive_partition_sensor.py
+++ b/tests/sensors/test_named_hive_partition_sensor.py
@@ -1,4 +1,3 @@
-# -*- coding: utf-8 -*-
 #
 # Licensed to the Apache Software Foundation (ASF) under one
 # or more contributor license agreements.  See the NOTICE file
@@ -16,21 +15,26 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+import os
 import random
 import unittest
 from datetime import timedelta
+from tests.compat import mock
 
-from airflow import DAG, operators
+from airflow.exceptions import AirflowSensorTimeout
+from airflow.models.dag import DAG
 from airflow.sensors.named_hive_partition_sensor import NamedHivePartitionSensor
 from airflow.utils.timezone import datetime
-from airflow.hooks.hive_hooks import HiveMetastoreHook
+from tests.hive import TestHiveEnvironment
+from tests.test_utils.mock_hooks import MockHiveMetastoreHook
+from tests.test_utils.mock_operators import MockHiveOperator
 
 DEFAULT_DATE = datetime(2015, 1, 1)
 DEFAULT_DATE_ISO = DEFAULT_DATE.isoformat()
 DEFAULT_DATE_DS = DEFAULT_DATE_ISO[:10]
 
 
-class NamedHivePartitionSensorTests(unittest.TestCase):
+class TestNamedHivePartitionSensor(unittest.TestCase):
     def setUp(self):
         args = {'owner': 'airflow', 'start_date': DEFAULT_DATE}
         self.dag = DAG('test_dag_id', default_args=args)
@@ -53,23 +57,18 @@ class NamedHivePartitionSensorTests(unittest.TestCase):
                 ALTER TABLE {{ params.table }}
                 ADD PARTITION({{ params.partition_by }}='{{ ds }}');
                 """
-        self.hook = HiveMetastoreHook()
-        t = operators.hive_operator.HiveOperator(
+        self.hook = MockHiveMetastoreHook()
+        op = MockHiveOperator(
             task_id='HiveHook_' + str(random.randint(1, 10000)),
             params={
                 'database': self.database,
                 'table': self.table,
                 'partition_by': self.partition_by
             },
-            hive_cli_conn_id='beeline_default',
+            hive_cli_conn_id='hive_cli_default',
             hql=self.hql, dag=self.dag)
-        t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
-              ignore_ti_state=True)
-
-    def tearDown(self):
-        hook = HiveMetastoreHook()
-        with hook.get_conn() as metastore:
-            metastore.drop_table(self.database, self.table, deleteData=True)
+        op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
+               ignore_ti_state=True)
 
     def test_parse_partition_name_correct(self):
         schema = 'default'
@@ -103,6 +102,7 @@ class NamedHivePartitionSensorTests(unittest.TestCase):
         self.assertEqual(partition, parsed_partition)
 
     def test_poke_existing(self):
+        self.hook.metastore.__enter__().check_for_named_partition.return_value = True
         partitions = ["{}.{}/{}={}".format(self.database,
                                            self.table,
                                            self.partition_by,
@@ -113,8 +113,11 @@ class NamedHivePartitionSensorTests(unittest.TestCase):
                                           hook=self.hook,
                                           dag=self.dag)
         self.assertTrue(sensor.poke(None))
+        self.hook.metastore.__enter__().check_for_named_partition.assert_called_with(
+            self.database, self.table, "{}={}".format(self.partition_by, DEFAULT_DATE_DS))
 
     def test_poke_non_existing(self):
+        self.hook.metastore.__enter__().check_for_named_partition.return_value = False
         partitions = ["{}.{}/{}={}".format(self.database,
                                            self.table,
                                            self.partition_by,
@@ -125,3 +128,79 @@ class NamedHivePartitionSensorTests(unittest.TestCase):
                                           hook=self.hook,
                                           dag=self.dag)
         self.assertFalse(sensor.poke(None))
+        self.hook.metastore.__enter__().check_for_named_partition.assert_called_with(
+            self.database, self.table, "{}={}".format(self.partition_by, self.next_day))
+
+
+@unittest.skipIf(
+    'AIRFLOW_RUNALL_TESTS' not in os.environ,
+    "Skipped because AIRFLOW_RUNALL_TESTS is not set")
+class TestPartitions(TestHiveEnvironment):
+
+    def test_succeeds_on_one_partition(self):
+        mock_hive_metastore_hook = MockHiveMetastoreHook()
+        mock_hive_metastore_hook.check_for_named_partition = mock.MagicMock(
+            return_value=True)
+
+        op = NamedHivePartitionSensor(
+            task_id='hive_partition_check',
+            partition_names=[
+                "airflow.static_babynames_partitioned/ds={{ds}}"
+            ],
+            dag=self.dag,
+            hook=mock_hive_metastore_hook
+        )
+
+        op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
+               ignore_ti_state=True)
+
+        mock_hive_metastore_hook.check_for_named_partition.assert_called_once_with(
+            'airflow', 'static_babynames_partitioned', 'ds=2015-01-01')
+
+    def test_succeeds_on_multiple_partitions(self):
+        mock_hive_metastore_hook = MockHiveMetastoreHook()
+        mock_hive_metastore_hook.check_for_named_partition = mock.MagicMock(
+            return_value=True)
+
+        op = NamedHivePartitionSensor(
+            task_id='hive_partition_check',
+            partition_names=[
+                "airflow.static_babynames_partitioned/ds={{ds}}",
+                "airflow.static_babynames_partitioned2/ds={{ds}}"
+            ],
+            dag=self.dag,
+            hook=mock_hive_metastore_hook
+        )
+        op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
+               ignore_ti_state=True)
+        mock_hive_metastore_hook.check_for_named_partition.assert_any_call(
+            'airflow', 'static_babynames_partitioned', 'ds=2015-01-01')
+        mock_hive_metastore_hook.check_for_named_partition.assert_any_call(
+            'airflow', 'static_babynames_partitioned2', 'ds=2015-01-01')
+
+    def test_parses_partitions_with_periods(self):
+        name = NamedHivePartitionSensor.parse_partition_name(
+            partition="schema.table/part1=this.can.be.an.issue/part2=ok")
+        self.assertEqual(name[0], "schema")
+        self.assertEqual(name[1], "table")
+        self.assertEqual(name[2], "part1=this.can.be.an.issue/part2=ok")
+
+    def test_times_out_on_nonexistent_partition(self):
+        with self.assertRaises(AirflowSensorTimeout):
+            mock_hive_metastore_hook = MockHiveMetastoreHook()
+            mock_hive_metastore_hook.check_for_named_partition = mock.MagicMock(
+                return_value=False)
+
+            op = NamedHivePartitionSensor(
+                task_id='hive_partition_check',
+                partition_names=[
+                    "airflow.static_babynames_partitioned/ds={{ds}}",
+                    "airflow.static_babynames_partitioned/ds=nonexistent"
+                ],
+                poke_interval=0.1,
+                timeout=1,
+                dag=self.dag,
+                hook=mock_hive_metastore_hook
+            )
+            op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
+                   ignore_ti_state=True)
diff --git a/tests/test_utils/mock_hooks.py b/tests/test_utils/mock_hooks.py
new file mode 100644
index 0000000..5c33e37
--- /dev/null
+++ b/tests/test_utils/mock_hooks.py
@@ -0,0 +1,96 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from tests.compat import MagicMock
+
+from airflow.hooks.hive_hooks import HiveMetastoreHook, HiveCliHook, HiveServer2Hook
+from airflow.hooks.mysql_hook import MySqlHook
+from airflow.hooks.presto_hook import PrestoHook
+from airflow.hooks.samba_hook import SambaHook
+from tests.test_utils.mock_process import MockConnectionCursor, MockDBConnection
+
+
+class MockHiveMetastoreHook(HiveMetastoreHook):
+    def __init__(self, *args, **kwargs):
+        self._find_valid_server = MagicMock(return_value={})
+        self.get_metastore_client = MagicMock(
+            return_value=MagicMock())
+        super(MockHiveMetastoreHook, self).__init__()
+
+
+class MockHiveCliHook(HiveCliHook):
+    def __init__(self, *args, **kwargs):
+        super(MockHiveCliHook, self).__init__()
+        self.conn = MockConnectionCursor()
+        self.conn.schema = 'default'
+        self.conn.host = 'localhost'
+        self.conn.port = 10000
+        self.conn.login = None
+        self.conn.password = None
+
+        self.conn.execute = MagicMock()
+        self.get_conn = MagicMock(return_value=self.conn)
+        self.get_connection = MagicMock(return_value=MockDBConnection({}))
+
+
+class MockSambaHook(SambaHook):
+    def __init__(self, *args, **kwargs):
+        self.conn = MockConnectionCursor()
+        self.conn.execute = MagicMock()
+        self.get_conn = MagicMock(return_value=self.conn)
+        super(MockSambaHook, self).__init__(*args, **kwargs)
+
+    def get_connection(self, *args):
+        return self.conn
+
+
+class MockPrestoHook(PrestoHook):
+    def __init__(self, *args, **kwargs):
+        self.conn = MockConnectionCursor()
+
+        self.conn.execute = MagicMock()
+        self.get_conn = MagicMock(return_value=self.conn)
+        self.get_first = MagicMock(
+            return_value=[['val_0', 'val_1'], 'val_2'])
+
+        super(MockPrestoHook, self).__init__(*args, **kwargs)
+
+    def get_connection(self, *args):
+        return self.conn
+
+
+class MockMySqlHook(MySqlHook):
+    def __init__(self, *args, **kwargs):
+        self.conn = MockConnectionCursor()
+
+        self.conn.execute = MagicMock()
+        self.get_conn = MagicMock(return_value=self.conn)
+        self.get_records = MagicMock(return_value=[])
+        self.insert_rows = MagicMock(return_value=True)
+        super(MockMySqlHook, self).__init__(*args, **kwargs)
+
+    def get_connection(self, *args, **kwargs):
+        return self.conn
+
+
+class MockHiveServer2Hook(HiveServer2Hook):
+    def __init__(self, *args, **kwargs):
+        super(MockHiveServer2Hook, self).__init__()
+        self.mock_cursor = kwargs.get('connection_cursor', MockConnectionCursor())
+        self.mock_cursor.execute = MagicMock()
+        self.get_conn = MagicMock(return_value=self.mock_cursor)
+        self.get_connection = MagicMock(return_value=MockDBConnection({}))
diff --git a/tests/test_utils/mock_operators.py b/tests/test_utils/mock_operators.py
index 2ee0230..0708134 100644
--- a/tests/test_utils/mock_operators.py
+++ b/tests/test_utils/mock_operators.py
@@ -17,7 +17,10 @@
 
 from collections import namedtuple
 
+from tests.compat import MagicMock
+
 from airflow.models.baseoperator import BaseOperator
+from airflow.operators.hive_operator import HiveOperator
 from airflow.utils.decorators import apply_defaults
 
 # Namedtuple for testing purposes
@@ -37,3 +40,9 @@ class MockOperator(BaseOperator):
 
     def execute(self, context):
         pass
+
+
+class MockHiveOperator(HiveOperator):
+    def __init__(self, *args, **kwargs):
+        self.run = MagicMock()
+        super(MockHiveOperator, self).__init__(*args, **kwargs)
diff --git a/tests/test_utils/mock_process.py b/tests/test_utils/mock_process.py
new file mode 100644
index 0000000..97b6561
--- /dev/null
+++ b/tests/test_utils/mock_process.py
@@ -0,0 +1,87 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from tests.compat import MagicMock
+
+
+class MockDBConnection:
+    def __init__(self, extra_dejson=None, *args, **kwargs):
+        self.extra_dejson = extra_dejson
+        self.get_records = MagicMock(return_value=[['test_record']])
+
+        output = kwargs.get('output', ['' for _ in range(10)])
+        self.readline = MagicMock(
+            side_effect=[line.encode() for line in output])
+
+    def status(self, *args, **kwargs):
+        return True
+
+
+class MockStdOut:
+    def __init__(self, *args, **kwargs):
+        output = kwargs.get('output', ['' for _ in range(10)])
+        self.readline = MagicMock(
+            side_effect=[line.encode() for line in output])
+
+
+class MockSubProcess:
+    PIPE = -1
+    STDOUT = -2
+    returncode = None
+
+    def __init__(self, *args, **kwargs):
+        self.stdout = MockStdOut(*args, **kwargs)
+
+    def wait(self):
+        return
+
+
+class MockConnectionCursor:
+    def __init__(self, *args, **kwargs):
+        self.arraysize = None
+        self.description = [('hive_server_hook.a', 'INT_TYPE', None, None, None, None, True),
+                            ('hive_server_hook.b', 'INT_TYPE', None, None, None, None, True)]
+        self.iterable = [(1, 1), (2, 2)]
+        self.conn_exists = kwargs.get('exists', True)
+
+    def close(self):
+        pass
+
+    def cursor(self):
+        return self
+
+    def execute(self, values=None):
+        pass
+
+    def exists(self, *args):
+        return self.conn_exists
+
+    def isfile(self, *args):
+        return True
+
+    def remove(self, *args):
+        pass
+
+    def upload(self, local_filepath, destination_filepath):
+        pass
+
+    def __next__(self):
+        return self.iterable
+
+    def __iter__(self):
+        for i in self.iterable:
+            yield i


[airflow] 03/36: Fix the process of requirements generations (#8648)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit ab10f4864b20ad13b65802443d9f38997c3a95a3
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Thu Apr 30 18:39:38 2020 +0200

    Fix the process of requirements generations (#8648)
    
    Right now requirements will be only checked during the
    CI build if the setup.py has changed and if yes, clear instructions
    will be given. The diff will still be printed otherwise but
    it will not cause the job to fail
    
    (cherry picked from commit 4a1d71d23df683ca1d164c7a7803beeda6595c51)
---
 .github/workflows/ci.yml                           |  3 +-
 scripts/ci/_utils.sh                               |  5 +--
 .../ci/in_container/run_generate_requirements.sh   | 39 +++++-----------------
 3 files changed, 10 insertions(+), 37 deletions(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index e2490d6..37eca52 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -263,9 +263,8 @@ jobs:
       fail-fast: false
     env:
       PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
-      SHOW_GENERATE_REQUIREMENTS_INSTRUCTIONS: true
+      CHECK_REQUIREMENTS_ONLY: true
       UPGRADE_WHILE_GENERATING_REQUIREMENTS: ${{ github.event_name == 'schedule' }}
-      FAIL_WHEN_REQUIREMENTS_UPDATED: ${{ github.event_name != 'schedule' }}
       CI_JOB_TYPE: "Requirements"
     steps:
       - uses: actions/checkout@master
diff --git a/scripts/ci/_utils.sh b/scripts/ci/_utils.sh
index 06c03bc..bc694a3 100644
--- a/scripts/ci/_utils.sh
+++ b/scripts/ci/_utils.sh
@@ -179,8 +179,6 @@ function initialize_common_environment {
         export UPGRADE_WHILE_GENERATING_REQUIREMENTS=${UPGRADE_WHILE_GENERATING_REQUIREMENTS:="false"}
     fi
 
-    export SHOW_GENERATE_REQUIREMENTS_INSTRUCTIONS=${SHOW_GENERATE_REQUIREMENTS_INSTRUCTIONS:="false"}
-
     export DEFAULT_CI_EXTRAS="devel_ci"
 
 
@@ -1499,8 +1497,7 @@ function run_generate_requirements() {
         --env HOST_GROUP_ID="$(id -gr)" \
         --env UPGRADE_WHILE_GENERATING_REQUIREMENTS \
         --env PYTHON_MAJOR_MINOR_VERSION \
-        --env SHOW_GENERATE_REQUIREMENTS_INSTRUCTIONS \
-        --env FAIL_WHEN_REQUIREMENTS_UPDATED \
+        --env CHECK_REQUIREMENTS_ONLY \
         --rm \
         "${AIRFLOW_CI_IMAGE}" \
         "--" "/opt/airflow/scripts/ci/in_container/run_generate_requirements.sh" \
diff --git a/scripts/ci/in_container/run_generate_requirements.sh b/scripts/ci/in_container/run_generate_requirements.sh
index d821795..5022e13 100755
--- a/scripts/ci/in_container/run_generate_requirements.sh
+++ b/scripts/ci/in_container/run_generate_requirements.sh
@@ -28,13 +28,16 @@ STORED_SETUP_PY_HASH_FILE="${AIRFLOW_SOURCES}/requirements/setup-${PYTHON_MAJOR_
 CURRENT_SETUP_PY_HASH=$(md5sum "${AIRFLOW_SOURCES}/setup.py")
 STORED_SETUP_PY_HASH=$(cat "${STORED_SETUP_PY_HASH_FILE}" 2>/dev/null || true)
 
-if [[ ${STORED_SETUP_PY_HASH} != "${CURRENT_SETUP_PY_HASH}" ]]; then
+if [[ ${STORED_SETUP_PY_HASH} != "${CURRENT_SETUP_PY_HASH}" && ${CHECK_REQUIREMENTS_ONLY:=} == "true" ]]; then
     echo
-    echo "Setup.py changed since last time requirements were generated"
+    echo "ERROR! Setup.py changed since last time requirements were generated"
     echo
-    echo "Switching to eager update strategy for the requirements"
+    echo "     When you update setup.py, you have to run"
     echo
-    UPGRADE_WHILE_GENERATING_REQUIREMENTS="true"
+    echo "           breeze generate-requirements --python ${PYTHON_MAJOR_MINOR_VERSION}"
+    echo
+    echo
+    exit 1
 fi
 
 # Upgrading requirements will happen only in CRON job to see that we have some
@@ -72,32 +75,6 @@ echo
 echo "${CURRENT_SETUP_PY_HASH}" > "${STORED_SETUP_PY_HASH_FILE}"
 
 set +e
-# Fail in case diff shows difference
 diff --color=always "${OLD_REQUIREMENTS_FILE}" "${GENERATED_REQUIREMENTS_FILE}"
-RES=$?
 
-if [[ ${FAIL_WHEN_REQUIREMENTS_UPDATED:="true"} == "true" ]]; then
-    if [[ ${RES} != "0" && ${SHOW_GENERATE_REQUIREMENTS_INSTRUCTIONS:=} == "true" ]]; then
-        echo
-        echo " ERROR! Requirements need to be updated!"
-        echo
-        echo "     Please generate requirements with:"
-        echo
-        echo "           breeze generate-requirements --python ${PYTHON_MAJOR_MINOR_VERSION}"
-        echo
-    fi
-    exit "${RES}"
-else
-    if [[ ${RES} != "0"  && ${SHOW_GENERATE_REQUIREMENTS_INSTRUCTIONS:=} == "true" ]]; then
-        echo
-        echo " WARNING! Above are the updated requirements for Airflow "
-        echo
-        echo "     Next time when you update setup.py, you will need to run"
-        echo
-        echo "           breeze generate-requirements --python ${PYTHON_MAJOR_MINOR_VERSION}"
-        echo
-        echo " Or you can run it now and make a separate 'requirements update' commit."
-        echo
-    fi
-    exit 0
-fi
+exit 0


[airflow] 28/36: Remove remnant kubernetes stuff from breeze scripts (#9138)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 4e7c48e4015cb8500c77a0df55d629b780f70886
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Thu Jun 4 15:08:54 2020 +0200

    Remove remnant kubernetes stuff from breeze scripts (#9138)
    
    (cherry picked from commit da748d02517d7903e92785e9da15458db35d20de)
---
 breeze                                   | 26 ------------------
 scripts/ci/ci_run_airflow_testing.sh     | 46 +++-----------------------------
 scripts/ci/in_container/entrypoint_ci.sh |  5 ----
 3 files changed, 3 insertions(+), 74 deletions(-)

diff --git a/breeze b/breeze
index a8e9295..8df8a41 100755
--- a/breeze
+++ b/breeze
@@ -378,17 +378,6 @@ EOF
 
 EOF
         fi
-
-        if [[ ${INTEGRATIONS[*]} == *"kubernetes"* ]]; then
-            cat <<EOF
-
-                               Kubernetes mode:     ${KUBERNETES_MODE}
-                               Kubernetes version:  ${KUBERNETES_VERSION}
-
-                               Enable KinD cluster: ${ENABLE_KIND_CLUSTER}
-                               Cluster operation:   ${KIND_CLUSTER_OPERATION}
-EOF
-        fi
     else
         if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
             cat <<EOF
@@ -422,18 +411,6 @@ EOF
 
 EOF
         fi
-        set +u
-        if [[ ${INTEGRATIONS[*]} == *"kubernetes"* ]]; then
-            cat <<EOF
-
-   Kubernetes mode:     ${KUBERNETES_MODE}
-   Kubernetes version:  ${KUBERNETES_VERSION}
-
-   Enable KinD cluster: ${ENABLE_KIND_CLUSTER}
-   Cluster operation:   ${KIND_CLUSTER_OPERATION}
-EOF
-        fi
-        set -u
     fi
 }
 
@@ -458,9 +435,6 @@ export DOCKERHUB_REPO=${DOCKERHUB_REPO}
 export COMPOSE_FILE="${COMPOSE_FILE}"
 export PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION}"
 export BACKEND="${BACKEND}"
-export ENABLE_KIND_CLUSTER="${ENABLE_KIND_CLUSTER}"
-export KUBERNETES_MODE="${KUBERNETES_MODE}"
-export KUBERNETES_VERSION="${KUBERNETES_VERSION}"
 export AIRFLOW_VERSION="${AIRFLOW_VERSION}"
 export INSTALL_AIRFLOW_VERSION="${INSTALL_AIRFLOW_VERSION}"
 export RUN_TESTS="${TESTS}"
diff --git a/scripts/ci/ci_run_airflow_testing.sh b/scripts/ci/ci_run_airflow_testing.sh
index f8d5e7e..5c707c5 100755
--- a/scripts/ci/ci_run_airflow_testing.sh
+++ b/scripts/ci/ci_run_airflow_testing.sh
@@ -17,25 +17,8 @@
 # under the License.
 export VERBOSE=${VERBOSE:="false"}
 
-function run_airflow_testing_in_docker_with_kubernetes() {
-    export KUBERNETES_MODE=${KUBERNETES_MODE:="image"}
-    export KUBERNETES_VERSION=${KUBERNETES_VERSION:="v1.15.3"}
-
-    # shellcheck disable=SC2016
-    docker-compose --log-level INFO \
-      -f "${MY_DIR}/docker-compose/base.yml" \
-      -f "${MY_DIR}/docker-compose/backend-${BACKEND}.yml" \
-      -f "${MY_DIR}/docker-compose/runtime-kubernetes.yml" \
-      "${INTEGRATIONS[@]}" \
-      "${DOCKER_COMPOSE_LOCAL[@]}" \
-         run airflow \
-           '/opt/airflow/scripts/ci/in_container/entrypoint_ci.sh "${@}"' \
-           /opt/airflow/scripts/ci/in_container/entrypoint_ci.sh "${@}"
-         # Note the command is there twice (!) because it is passed via bash -c
-         # and bash -c starts passing parameters from $0. TODO: fixme
-}
-
 function run_airflow_testing_in_docker() {
+    set +u
     # shellcheck disable=SC2016
     docker-compose --log-level INFO \
       -f "${MY_DIR}/docker-compose/base.yml" \
@@ -47,9 +30,9 @@ function run_airflow_testing_in_docker() {
            /opt/airflow/scripts/ci/in_container/entrypoint_ci.sh "${@}"
          # Note the command is there twice (!) because it is passed via bash -c
          # and bash -c starts passing parameters from $0. TODO: fixme
+    set -u
 }
 
-
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
@@ -131,11 +114,7 @@ MAX_RETRIES=4
 while [[ ${MAX_RETRIES} -ge "0" ]]
 do
     set +e
-    if [[ ${RUNTIME:=} == "kubernetes" ]]; then
-        run_airflow_testing_in_docker_with_kubernetes "${@}"
-    else
-        run_airflow_testing_in_docker "${@}"
-    fi
+    run_airflow_testing_in_docker "${@}"
     EXIT_CODE=$?
     set -e
     if [[ ${EXIT_CODE} == 254 ]] ; then
@@ -150,26 +129,7 @@ do
     fi
     exit ${EXIT_CODE}
 done
-set -u
-
-#TODO FIXME
-if [[ "${INTEGRATIONS[*]}" == *"kubernetes"* ]]; then
-    export KUBERNETES_MODE=${KUBERNETES_MODE:="image"}
-    export KUBERNETES_VERSION=${KUBERNETES_VERSION:="v1.15.3"}
-    export KIND_CLUSTER_NAME="airflow-python-${PYTHON_MAJOR_MINOR_VERSION}-${KUBERNETES_VERSION}"
 
-    if [[ ${ENABLE_KIND_CLUSTER} == "true" ]]; then
-        install_kind_in_host
-        "${MY_DIR}/setup_kind_cluster_in_host.sh"
-    fi
-
-    # adding trap to exiting trap
-    HANDLERS="$( trap -p EXIT | cut -f2 -d \' )"
-    # shellcheck disable=SC2064
-    trap "${HANDLERS}${HANDLERS:+;}send_kubernetes_logs_to_file_io" EXIT
-fi
-
-set +u
 # shellcheck disable=SC2016
 docker-compose --log-level INFO \
   -f "${MY_DIR}/docker-compose/base.yml" \
diff --git a/scripts/ci/in_container/entrypoint_ci.sh b/scripts/ci/in_container/entrypoint_ci.sh
index 111e7ca..60757fa 100755
--- a/scripts/ci/in_container/entrypoint_ci.sh
+++ b/scripts/ci/in_container/entrypoint_ci.sh
@@ -26,9 +26,6 @@ AIRFLOW_SOURCES=$(cd "${MY_DIR}/../../.." || exit 1; pwd)
 
 PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:=3.6}
 BACKEND=${BACKEND:=sqlite}
-KUBERNETES_MODE=${KUBERNETES_MODE:=""}
-KUBERNETES_VERSION=${KUBERNETES_VERSION:=""}
-ENABLE_KIND_CLUSTER=${ENABLE_KIND_CLUSTER:="false"}
 
 export AIRFLOW_HOME=${AIRFLOW_HOME:=${HOME}}
 
@@ -165,8 +162,6 @@ done
 
 ssh-keyscan -H localhost >> ~/.ssh/known_hosts 2>/dev/null
 
-export KUBERNETES_VERSION=${KUBERNETES_VERSION:=""}
-
 # shellcheck source=scripts/ci/in_container/configure_environment.sh
 . "${MY_DIR}/configure_environment.sh"
 


[airflow] 05/36: Add comments to breeze scripts (#8797)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit af6aed47e13d1f194ead57637449a233065ec452
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sun May 10 19:55:40 2020 +0200

    Add comments to breeze scripts (#8797)
    
    (cherry picked from commit 9bb91ef9912c0c9cda8b7eecd365338053d14aa7)
---
 breeze                                       | 183 +++++++++++++++--------
 scripts/ci/_utils.sh                         | 212 ++++++++++++++++++++-------
 scripts/ci/docker-compose/backend-sqlite.yml |   2 +-
 3 files changed, 284 insertions(+), 113 deletions(-)

diff --git a/breeze b/breeze
index 96d6f0e..4ec67c6 100755
--- a/breeze
+++ b/breeze
@@ -30,38 +30,52 @@ declare -a REMAINING_ARGS
 # This is where static check options are defined
 declare -a EXTRA_STATIC_CHECK_OPTIONS
 
-
+# Sets up all the default variables for Breeze
+# They are needed by all other functions
 function setup_default_breeze_variables() {
-    # Whether to actually run docker compose with the command set given
+    # Default command to run - entering breeze environment
     COMMAND_TO_RUN="enter_breeze"
+    # In some cases we also want to run two commands in a row (for example when we restart the environment)
     SECOND_COMMAND_TO_RUN=""
+    # Indicates that we are inside Breeze environment
     export BREEZE=true
+    # Maximum screen width to print the lines spanning the whole terminal width
     export MAX_SCREEN_WIDTH=100
 
+    # This is where airflow sources reside
     export AIRFLOW_SOURCES="${MY_DIR}"
 
     # Directory where all CI scripts are located
     SCRIPTS_CI_DIR="${MY_DIR}/scripts/ci"
 
+    # Directory where all the build cache is stored - we keep there status of all the docker images
+    # As well as hashes of the important files, but also we generate build scripts there that are
+    # Used to execute the commands for breeze
     BUILD_CACHE_DIR="${MY_DIR}/.build"
+    # This folder is mounted to inside the container in /files folder. This is the way how
+    # We can exchange DAGs, scripts, packages etc with the container environment
     FILES_DIR="${MY_DIR}/files"
+    # Temporary dir used well ... temporarily
     TMP_DIR="${MY_DIR}/tmp"
 
+    # Create those folders above in case they do not exist
     mkdir -pv "${BUILD_CACHE_DIR}"
     mkdir -pv "${TMP_DIR}"
     mkdir -pv "${FILES_DIR}"
 
-    # Note - we do not use __script_init.sh here because it can only be used from within
-    # the CI directory and we need to override PYTHON_MAJOR_MINOR_VERSION based on what we store
-    # in the .build directory
-
-    # Beginning of the initialisation here
-
+    # load all the common functions here - those are the functions that are shared between Breeze
+    # and CI scripts (CI scripts do not use Breeze as execution environment)
     # shellcheck source=scripts/ci/_utils.sh
     . "${SCRIPTS_CI_DIR}/_utils.sh"
 
+    # We have different versions of images depending on the python version used. We keep up with the
+    # Latest patch-level changes in Python (this is done automatically during CI builds) so we have
+    # To only take into account MAJOR and MINOR version of python. This variable keeps the major/mninor
+    # version of python in X.Y format (3.6, 3.7 etc).
     export PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION:=$(read_from_file PYTHON_MAJOR_MINOR_VERSION)}"
 
+    # When we generate documentation for README files, we want to force the width of terminal so that
+    # No matter who is running the documentation generation gets the same output
     if [[ ${FORCE_SCREEN_WIDTH:="false"} != "true" ]]; then
         # Sets width of the screen from terminal
         SCREEN_WIDTH="$(tput cols)"
@@ -75,52 +89,47 @@ function setup_default_breeze_variables() {
         SCREEN_WIDTH=${MAX_SCREEN_WIDTH}
     fi
 
-    # Name of the script
+    # Name of the script is kept in this variable
     CMDNAME="$(basename -- "$0")"
 
     # Update short and long options in the breeze-complete script
-    # This way autocomplete will work automatically with all options
+    # This way autocomplete will work automatically with all options available
     # shellcheck source=breeze-complete
     . "${MY_DIR}/breeze-complete"
 
-    # Skips mounting local Airflow sources
+    # By default we mount local Airflow sources
     MOUNT_LOCAL_SOURCES="true"
 
-    # Holds last sub-command used
+    # Holds last sub-command used - this is used by --help flag to print help for the command entered
     LAST_SUBCOMMAND=""
 
     # Determines if help should be run (set to true by --help flag)
     RUN_HELP="false"
 
-    # Holds chosen command if the -x flag is used.
+    # Holds chosen command to run in case 'execute-command' command is used.
     RUN_COMMAND=""
 
-    # Holds the test target if the -t flag is used.
+    # Holds the test target if the 'test-target' command is used.
     TEST_TARGET=""
 
-    # Holds docker compose command if the -d flag is used.
+    # Holds docker compose command if the `docker-compose` command is used.
     DOCKER_COMPOSE_COMMAND=""
 
-    # If true, the docker images are rebuilt locally.
+    # If set to true, the docker images are rebuilt locally. By default we assume we do not need to
+    # rebuild the image but if we find we do, this variable will be set to "true"
     export NEEDS_DOCKER_BUILD="false"
 
     # By default we only pull images if we do not have them locally.
-    # This can be overridden by -p flag
+    # This can be overridden by '--force-pull-images' flag
     export FORCE_PULL_IMAGES="false"
 
-    # Runtime is empty initially (might be set to kubernetes in case kubernetes is chosen)
+    # Runtime is empty by default (might be set to kubernetes in case kubernetes is chosen)
     RUNTIME=""
 
-    # Do not enable Kind Kubernetes cluster by default
-    export ENABLE_KIND_CLUSTER="false"
-
-    # By default we do not push images. This can be overridden by -u flag.
-    export PUSH_IMAGES=${PUSH_IMAGES:="false"}
-
-    # Forward credentials to docker
+    # Forward common host credentials to docker (gcloud, aws etc.).
     export FORWARD_CREDENTIALS="false"
 
-    # Reset DB at entry
+    # If set to true, the database will be reset at entry. Works for Postgres and MySQL
     export DB_RESET="false"
 
     # If it set is set to specified version, then the source version of Airflow
@@ -131,18 +140,15 @@ function setup_default_breeze_variables() {
     # of Airflow is removed and the specified version of Airflow is installed from GitHub
     export INSTALL_AIRFLOW_REFERENCE=${INSTALL_AIRFLOW_REFERENCE:=""}
 
-    # Determine version of the Airflow from version.py
-    AIRFLOW_VERSION=$(grep version "airflow/version.py" | awk '{print $3}' | sed "s/['+]//g")
-    export AIRFLOW_VERSION
-
-    # Whether to force build without checking if it is needed
+    # Determines whether to force build without checking if it is needed
+    # Can be overridden by '--force-build-images' flag.
     export FORCE_BUILD_IMAGES=${FORCE_BUILD_IMAGES:="false"}
 
-    # Files determining whether ASCII-art/cheat-sheet are suppressed
+    # If those files are present, the ASCII-art/cheat-sheet are suppressed
     SUPPRESS_CHEATSHEET_FILE="${MY_DIR}/.suppress_cheatsheet"
     SUPPRESS_ASCIIART_FILE="${MY_DIR}/.suppress_asciiart"
 
-    # Default values for flags
+    # Default values for the flags used
 
     _BREEZE_DEFAULT_BACKEND="sqlite"
     _BREEZE_DEFAULT_KUBERNETES_MODE="git_mode"
@@ -152,10 +158,14 @@ function setup_default_breeze_variables() {
 
     STATIC_CHECK_PYTHON_MAJOR_MINOR_VERSION=3.6
 }
-# End of initialisation here
 
+# Initializes development-friendly virtualenv if you are already in such env. It installs all the necessary
+# packages from PyPI and it case of problems it provides useful hints on what prerequisites should be
+# installed. It also removes and resets the existing AIRFOW_HOME installation to make sure that you
+# have it synchronized with the version of airflow installed. It resets the airflow's sqlite database to
+# a clean state. You can use this function if your virtualenv is broken, to clean it up
 function initialize_virtualenv() {
-   # Check if we are in virtualenv
+   # Check if we are inside virtualenv
    set +e
    echo -e "import sys\nif not hasattr(sys,'base_prefix'):\n  sys.exit(1)" | "python${PYTHON_MAJOR_MINOR_VERSION}"
    RES=$?
@@ -218,6 +228,8 @@ function initialize_virtualenv() {
    fi
 }
 
+
+# Sets up autocomplete for Breeze for both - bash and zsh
 function setup_autocomplete() {
     echo "Installing bash/zsh completion for local user"
     echo "Note that completion for zsh is just limited to flags - without their values"
@@ -280,6 +292,8 @@ EOF
     exit 0
 }
 
+# Prints information about the current configuration of Breeze - if you enter breeze interactively
+# and you did not suppress cheatsheet or asciiart, it also prints those
 function print_badge {
     if [[ ${BACKEND} == "postgres" ]]; then
         BACKEND_VERSION="${POSTGRES_VERSION}"
@@ -426,6 +440,12 @@ EOF
     fi
 }
 
+# Prepares command file that can be used to easily run the commands without the need of using Breeze
+# The command file generated in cache ./build directory is a standalone script that contains
+# All the environment variables and docker-compose configuration to run the command. This is because
+# depending on configuration of Breeze we might have different compose files used and different variables
+# set. Those are a convenience scripts that you might use to debug command execution although
+# In most cases they are used internally by Breeze
 function prepare_command_file() {
     local FILE="${1}"
     local CMD="${2}"
@@ -457,12 +477,21 @@ export MYSQL_ENCODING="${MYSQL_ENCODING}"
 export AIRFLOW_CI_IMAGE="${AIRFLOW_CI_IMAGE}"
 export AIRFOW_PROD_IMAGE="${AIRFLOW_PROD_IMAGE}"
 export AIRFLOW_IMAGE="${AIRFLOW_IMAGE}"
-export SQLITE_VERSION="${SQLITE_VERSION}"
+export SQLITE_URL="${SQLITE_URL}"
 docker-compose --log-level INFO ${CMD}\$${EXPANSION}"
 EOF
     chmod u+x "${FILE}"
 }
 
+
+# Prepare all command files that we are using. Depending on the command to execute we use different
+# convenience scripts:
+#    cmd_run_ci - to run CI image command (for example entering Breeze, or running a script in CI image)
+#    cmd_run_prod - to run PROD image command (for example entering Prod image or running a script there)
+#    test_run_ci - to run test target in CI image
+#    dc_ci - to run docker compose command for CI image
+#    dc_prod - to run docker compose command for PROD image
+#
 function prepare_command_files() {
     MAIN_CI_DOCKER_COMPOSE_FILE=${SCRIPTS_CI_DIR}/docker-compose/base.yml
     MAIN_PROD_DOCKER_COMPOSE_FILE=${SCRIPTS_CI_DIR}/docker-compose/base.yml
@@ -547,6 +576,8 @@ function prepare_command_files() {
         '"' "false" "${COMPOSE_PROD_FILE}" "${AIRFLOW_PROD_IMAGE}"
 }
 
+# Prints detailed help for all commands and flgas. Used to generate documentation added to BREEZE.rst
+# automatically.
 function do_help_all() {
     echo
     print_line
@@ -568,6 +599,7 @@ function do_help_all() {
     flags
 }
 
+# Parses all arguments that can be passed to Breeze command - that includes command to run and flags.
 function parse_arguments() {
     set -u
     if ! PARAMS=$(getopt \
@@ -731,7 +763,7 @@ function parse_arguments() {
           shift ;;
         -I|--production-image)
           export PRODUCTION_IMAGE="true"
-          export SQLITE_VERSION=
+          export SQLITE_URL=
           echo
           echo "*************** PRODUCTION IMAGE *************************"
           echo
@@ -942,6 +974,9 @@ function parse_arguments() {
     REMAINING_ARGS+=("$@")
 }
 
+# Prepares nicely formatted versions of list of allowed and default values defined in Breeze.
+# It is used in help command to print the lists in a readable format and fold the lists
+# so that they fit the screen width.
 function prepare_formatted_versions() {
     INDENT=15
     LIST_PREFIX=$(printf "%-${INDENT}s" " ")
@@ -970,6 +1005,8 @@ function prepare_formatted_versions() {
       tr ',' ' ' | fold -w "${WIDTH}" -s | sed "s/ /,/g; s/^/${LIST_PREFIX}/")
 }
 
+# Prepares usage information for all the commands in Breeze.
+# Those usage commands are stored in appropriate environment variables.
 function prepare_usage() {
     # Note that MacOS uses Bash 3.* and we cannot use associative arrays
     export USAGE_SHELL="[Default] Enters interactive shell in the container"
@@ -1197,6 +1234,7 @@ $(flag_footer)
 "
 }
 
+# Gets environment variable value converting the lowercase name of command into variable name
 function get_variable_from_lowercase_name() {
     PREFIX="${1}"
     NAME="${2}"
@@ -1205,15 +1243,18 @@ function get_variable_from_lowercase_name() {
     echo "${!VARIABLE_NAME}"
 }
 
+# Gets usage information from lowercase command
 function get_usage() {
     get_variable_from_lowercase_name USAGE "${1}"
 }
 
+
+# Gets detailed usage information from lowercase command
 function get_detailed_usage() {
     get_variable_from_lowercase_name DETAILED_USAGE "${1}"
 }
 
-
+# Prints general usage information
 function usage() {
       echo "
 
@@ -1245,6 +1286,7 @@ Help commands:
       echo
 }
 
+# Prints detailed usage for command specified
 function detailed_usage() {
       SUBCOMMAND=${1}
       echo "
@@ -1257,12 +1299,14 @@ $(get_detailed_usage "${SUBCOMMAND}")
 "
 }
 
+# Prints flag footer
 function flag_footer() {
      echo "
 Run '${CMDNAME} flags' to see all applicable flags.
 "
 }
 
+# Prints flags for different variants of airflow to use
 function flag_airflow_variants() {
       echo "
 -p, --python <PYTHON_MAJOR_MINOR_VERSION>
@@ -1273,6 +1317,7 @@ ${FORMATTED_PYTHON_MAJOR_MINOR_VERSIONS}
 "
 }
 
+# Prints flags for different backend to use
 function flag_backend_variants() {
       echo "
 -b, --backend <BACKEND>
@@ -1295,6 +1340,7 @@ ${FORMATTED_MYSQL_VERSIONS}
 "
 }
 
+# Prints production image flgas
 function flag_production_image() {
       echo "
 -I, --production-image
@@ -1302,6 +1348,7 @@ function flag_production_image() {
 "
 }
 
+# Prints additional breeze action flags
 function flag_breeze_actions() {
       echo "
 -d, --db-reset
@@ -1322,6 +1369,7 @@ ${FORMATTED_INTEGRATIONS}
 "
 }
 
+# Prints Kubernetes action flags
 function flag_kubernetes_actions() {
       echo "
 Action for the cluster : only one of the --kind-cluster-* flags can be used at a time:
@@ -1364,6 +1412,7 @@ ${FORMATTED_KUBERNETES_VERSIONS}
 "
 }
 
+# Prints flags that determine what is the source mounting scheme
 function flag_local_file_mounting() {
       echo "
 -l, --skip-mounting-local-sources
@@ -1372,6 +1421,7 @@ function flag_local_file_mounting() {
 "
 }
 
+# Prints flags that allow to choose different airflow variants
 function flag_choose_different_airflow_version() {
       echo "
 -a, --install-airflow-version <INSTALL_AIRFLOW_VERSION>
@@ -1386,6 +1436,7 @@ ${FORMATTED_INSTALL_AIRFLOW_VERSIONS}
 "
 }
 
+# Prints flags that allow to set assumed answers to questions
 function flag_assume_answers_to_questions() {
       echo "
 -y, --assume-yes
@@ -1399,6 +1450,7 @@ function flag_assume_answers_to_questions() {
 "
 }
 
+# Prints flags that are used for credential forwarding
 function flag_credentials() {
       echo "
 -f, --forward-credentials
@@ -1407,6 +1459,7 @@ function flag_credentials() {
 "
 }
 
+# Prints flags that control verbosity
 function flag_verbosity() {
       echo "
 -v, --verbose
@@ -1416,6 +1469,7 @@ function flag_verbosity() {
 "
 }
 
+# Prints flags controlling docker build process
 function flag_build_docker_images() {
       echo "
 -F, --force-build-images
@@ -1447,6 +1501,7 @@ ${FORMATTED_DEFAULT_PROD_EXTRAS}
 "
 }
 
+# Prints flags controlling docker push process
 function flag_push_docker_images() {
        echo "
 -u, --push-images
@@ -1462,6 +1517,8 @@ function flag_push_docker_images() {
 "
 }
 
+
+# Prints all flags
 function flags() {
       echo "
 $(print_line)
@@ -1518,6 +1575,7 @@ $(flag_verbosity)
 "
 }
 
+# Prints header line filling screen width - only when VERBOSE is set
 function print_header_line() {
     if [ ${VERBOSE:="false"} == "true" ]; then
         echo
@@ -1527,14 +1585,21 @@ function print_header_line() {
 }
 
 
+# Prints separation line filling screen width
 function print_line {
     printf '#%.0s' $(seq "${SCREEN_WIDTH}")
 }
 
+# Prints star line filling screen width
 function print_star_line {
     printf '*%.0s' $(seq "${SCREEN_WIDTH}")
 }
 
+# Reads save environment variables. Some of the variables are stored across session so that once
+# you use them you do not have to use it next time. That makes those flags persistent
+# An example of it is `--backend` or '--kubernetes-mode' flags. Note that PYTHON_MAJOR_MINOR_VERSION is
+# not read here - it is read at the `setup_default_breeze_variables` method because it is needed
+# t determine the right images to use and set several variables that depend on the Python version
 function read_saved_environment_variables {
     export BACKEND="${BACKEND:=$(read_from_file BACKEND)}"
     export BACKEND=${BACKEND:-${_BREEZE_DEFAULT_BACKEND}}
@@ -1553,17 +1618,19 @@ function read_saved_environment_variables {
 
     # Here you read DockerHub user/account that you use
     # You can populate your own images in DockerHub this way and work with the,
-    # You can override it with "-d" option and it will be stored in .build directory
+    # You can override it with "--dockerhub-user" option and it will be stored in .build directory
     export DOCKERHUB_USER="${DOCKERHUB_USER:=$(read_from_file DOCKERHUB_USER)}"
     export DOCKERHUB_USER="${DOCKERHUB_USER:=${_BREEZE_DEFAULT_DOCKERHUB_USER}}"
 
     # Here you read DockerHub repo that you use
     # You can populate your own images in DockerHub this way and work with them
-    # You can override it with "-d" option and it will be stored in .build directory
+    # You can override it with "--dockerhub-repo" option and it will be stored in .build directory
     export DOCKERHUB_REPO="${DOCKERHUB_REPO:=$(read_from_file DOCKERHUB_REPO)}"
     export DOCKERHUB_REPO="${DOCKERHUB_REPO:=${_BREEZE_DEFAULT_DOCKERHUB_REPO}}"
 }
 
+# Checks if variables are correctly set and if they are - saves them so that they can be used across
+# sessions.
 function check_and_save_all_params() {
     check_and_save_allowed_param "PYTHON_MAJOR_MINOR_VERSION"  "Python version" "--python"
     check_and_save_allowed_param "BACKEND"  "backend" "--backend"
@@ -1572,31 +1639,12 @@ function check_and_save_all_params() {
     check_and_save_allowed_param "POSTGRES_VERSION"  "Postgres version" "--postgres-version"
     check_and_save_allowed_param "MYSQL_VERSION"  "Mysql version" "--mysql-version"
 
-    # Can't verify those
+    # Can't verify those - they can be anything, so let's just save them
     save_to_file DOCKERHUB_USER
     save_to_file DOCKERHUB_REPO
 }
 
-function fix_local_file {
-    if [[ -d "${MY_DIR}/${1}" ]]; then
-        rm -rf "${MY_DIR:?}/${1}"
-    fi
-    touch "${MY_DIR}/${1}"
-
-}
-
-function touch_local_files {
-    # Those files are mounted into container when run locally
-    # .bash_history is preserved and you can modify .bash_aliases and .inputrc
-    # according to your liking
-    fix_local_file ".bash_history"
-    fix_local_file ".bash_aliases"
-    fix_local_file ".inputrc"
-    # When KinD cluster is created, the folder keeps authentication information
-    # across sessions
-    mkdir -pv "${MY_DIR}/.kube"
-}
-
+# Prints cheatsheet if it is not suppressed
 function print_cheatsheet() {
     if [[ ! -f ${SUPPRESS_CHEATSHEET_FILE} && ${COMMAND_TO_RUN} == "enter_breeze" ]]; then
         echo
@@ -1656,6 +1704,8 @@ function print_cheatsheet() {
     fi
 }
 
+# Prints setup instruction in case we find that autocomplete is not set
+# also prints how to toggle asciiart/cheatsheet
 function print_setup_instructions {
     if [[ ${COMMAND_TO_RUN} == "enter_breeze" ]] ; then
         # shellcheck disable=SC2034  # Unused variables left for comp_breeze usage
@@ -1680,6 +1730,8 @@ function print_setup_instructions {
     fi
 }
 
+# Checks that pre-commit is installed and upgrades it if needed
+# this is used in case static check command is used
 function make_sure_precommit_is_installed {
     echo
     echo "Making sure pre-commit is installed"
@@ -1700,12 +1752,15 @@ function make_sure_precommit_is_installed {
     export PATH="${PATH}":~/.local/bin
 }
 
+# Removes CI and PROD images and cleans up the flag that indicates that the image was already built
 function remove_images {
     docker rmi "${PYTHON_BASE_IMAGE}" || true
     docker rmi "${AIRFLOW_CI_IMAGE}" || true
+    docker rmi "${AIRFLOW_PROD_IMAGE}" || true
     rm -f "${BUILT_IMAGE_FLAG_FILE}"
 }
 
+# Runs chosen static checks
 function run_static_checks {
     if [[ ${STATIC_CHECK} == "all" ]]; then
         echo
@@ -1720,6 +1775,8 @@ function run_static_checks {
     fi
 }
 
+# Runs Build before a comman - it will check and if needed rebuild necessary image, depending on the
+# command chosen
 function run_build_command {
     case "${COMMAND_TO_RUN}" in
         run_tests|run_docker_compose|run_in_bash)
@@ -1766,6 +1823,8 @@ function run_build_command {
     esac
 }
 
+# Runs the actual command - depending on the command chosen it will use the right
+# Convenient script and run the right command with it
 function run_breeze_command {
     set -u
     case "${COMMAND_TO_RUN}" in
@@ -1875,7 +1934,7 @@ read_saved_environment_variables
 
 check_and_save_all_params
 
-touch_local_files
+sanitize_mounted_files
 
 prepare_command_files
 
diff --git a/scripts/ci/_utils.sh b/scripts/ci/_utils.sh
index bc694a3..8ac07ab 100644
--- a/scripts/ci/_utils.sh
+++ b/scripts/ci/_utils.sh
@@ -19,11 +19,14 @@
 # Assume all the scripts are sourcing the _utils.sh from the scripts/ci directory
 # and MY_DIR variable is set to this directory. It can be overridden however
 
+# extra flags passed to the docker run for CI image commands (as Bash array)
 declare -a EXTRA_DOCKER_FLAGS
+# extra flags passed to the docker run for PROD image commands(as Bash array)
 declare -a EXTRA_DOCKER_PROD_BUILD_FLAGS
 export EXTRA_DOCKER_FLAGS
 export EXTRA_DOCKER_PROD_BUILD_FLAGS
 
+# In case "VERBOSE_COMMANDS" is set to "true" set -x is used to enable debugging
 function check_verbose_setup {
     if [[ ${VERBOSE_COMMANDS:="false"} == "true" ]]; then
         set -x
@@ -33,6 +36,8 @@ function check_verbose_setup {
 }
 
 
+# In case "VERBOSE" is set to "true" (--verbose flag in Breeze) all docker commands run will be
+# printed before execution
 function verbose_docker {
     if [[ ${VERBOSE:="false"} == "true" ]]; then
         echo "docker" "${@}"
@@ -42,28 +47,36 @@ function verbose_docker {
 
 # Common environment that is initialized by both Breeze and CI scripts
 function initialize_common_environment {
-    # default python version
+    # default python Major/Minor version
     PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:="3.5"}
 
+    # Sets to where airflow sources are located
     AIRFLOW_SOURCES=${AIRFLOW_SOURCES:=$(cd "${MY_DIR}/../../" && pwd)}
     export AIRFLOW_SOURCES
 
+    # Sets to the build cache directory - status of build and convenience scripts are stored there
     BUILD_CACHE_DIR="${AIRFLOW_SOURCES}/.build"
     export BUILD_CACHE_DIR
 
+    # File to keep the last forced answer. This is useful for pre-commits where you need to
+    # only answer once if the image should be rebuilt or not and your answer is used for
+    # All the subsequent questions
     LAST_FORCE_ANSWER_FILE="${BUILD_CACHE_DIR}/last_force_answer.sh"
 
-    # Create directories if needed
+    # Create useful directories if not yet created
     mkdir -p "${AIRFLOW_SOURCES}/.mypy_cache"
     mkdir -p "${AIRFLOW_SOURCES}/logs"
     mkdir -p "${AIRFLOW_SOURCES}/tmp"
     mkdir -p "${AIRFLOW_SOURCES}/files"
     mkdir -p "${AIRFLOW_SOURCES}/dist"
 
+    # Read common values used across Breeze and CI scripts
     # shellcheck source=common/_common_values.sh
     . "${AIRFLOW_SOURCES}/common/_common_values.sh"
+    # Read image-specific values used across Breeze and CI scripts
     # shellcheck source=common/_image_variables.sh
     . "${AIRFLOW_SOURCES}/common/_image_variables.sh"
+    # Read information about files that are checked if image should be rebuilt
     # shellcheck source=common/_files_for_rebuild_check.sh
     . "${AIRFLOW_SOURCES}/common/_files_for_rebuild_check.sh"
 
@@ -79,7 +92,8 @@ function initialize_common_environment {
     export POSTGRES_VERSION=${POSTGRES_VERSION:="9.6"}
     export MYSQL_VERSION=${MYSQL_VERSION:="5.6"}
 
-    # Do not push images from here by default (push them directly from the build script on Dockerhub)
+    # Do not push images by default (push them directly from the build script on Dockerhub or when
+    # --push-images flag is specified
     export PUSH_IMAGES=${PUSH_IMAGES:="false"}
 
     # Disable writing .pyc files - slightly slower imports but not messing around when switching
@@ -92,24 +106,28 @@ function initialize_common_environment {
     # Sets mounting of host volumes to container for static checks
     # unless MOUNT_HOST_AIRFLOW_VOLUME is not true
     #
-    # Note that this cannot be function because we need the EXTRA_DOCKER_FLAGS array variable
-    #
     MOUNT_HOST_AIRFLOW_VOLUME=${MOUNT_HOST_AIRFLOW_VOLUME:="true"}
     export MOUNT_HOST_AIRFLOW_VOLUME
 
     # If this variable is set, we mount the whole sources directory to the host rather than
-    # selected volumes
+    # selected volumes. This is needed to check ALL source files during licence check
+    # for example
     MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS=${MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS="false"}
     export MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS
 
-    # Set host user id to current user
+    # Set host user id to current user. This is used to set the ownership properly when exiting
+    # The container on Linux - all files created inside docker are created with root user
+    # but they should be restored back to the host user
     HOST_USER_ID="$(id -ur)"
     export HOST_USER_ID
 
-    # Set host group id to current group
+    # Set host group id to current group This is used to set the ownership properly when exiting
+    # The container on Linux - all files created inside docker are created with root user
+    # but they should be restored back to the host user
     HOST_GROUP_ID="$(id -gr)"
     export HOST_GROUP_ID
 
+    # Add the right volume mount for sources, depending which mount strategy is used
     if [[ ${MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS} == "true" ]]; then
         print_info
         print_info "Mount whole airflow source directory for static checks (make sure all files are in container)"
@@ -137,6 +155,8 @@ function initialize_common_environment {
         )
     fi
 
+    # In case of the CI build get environment variables from codecov.io and
+    # set it as the extra docker flags. As described in https://docs.codecov.io/docs/testing-with-docker
     if [[ ${CI:=} == "true" ]]; then
         CI_CODECOV_ENV="$(bash <(curl -s https://codecov.io/env))"
         for ENV_PARAM in ${CI_CODECOV_ENV}
@@ -147,31 +167,37 @@ function initialize_common_environment {
     EXTRA_DOCKER_PROD_BUILD_FLAGS=()
 
     # We use pulled docker image cache by default to speed up the builds
+    # but we can also set different docker caching strategy (for example we can use local cache
+    # to build the images in case we iterate with the image
     export DOCKER_CACHE=${DOCKER_CACHE:="pulled"}
 
-    # By default we are not upgrading to latest requirements
-    # This will only be done in cron jobs and when run with breeze
+    # By default we are not upgrading to latest requirements when building Docker CI image
+    # This will only be done in cron jobs
     export UPGRADE_TO_LATEST_REQUIREMENTS=${UPGRADE_TO_LATEST_REQUIREMENTS:="false"}
 
+    # In case of MacOS we need to use gstat - gnu version of the stats
     STAT_BIN=stat
     if [[ "${OSTYPE}" == "darwin"* ]]; then
         STAT_BIN=gstat
     fi
 
+    # Read airflow version from the version.py
     AIRFLOW_VERSION=$(grep version "airflow/version.py" | awk '{print $3}' | sed "s/['+]//g")
     export AIRFLOW_VERSION
 
-    # default version for dockerhub images
+    # default version of python used to tag the "master" and "latest" images in DockerHub
     export DEFAULT_PYTHON_MAJOR_MINOR_VERSION=3.6
 
-
+    # In case we are not in CI - we assume we run locally. There are subtle changes if you run
+    # CI scripts locally - for example requirements are eagerly updated if you do local run
+    # in generate requirements
     if [[ ${CI:="false"} == "true" ]]; then
         export LOCAL_RUN="false"
     else
         export LOCAL_RUN="true"
     fi
 
-    # upgrade while generating requirements should only happen in localy run
+    # eager upgrade while generating requirements should only happen in locally run
     # pre-commits or in cron job
     if [[ ${LOCAL_RUN} == "true" ]]; then
         export UPGRADE_WHILE_GENERATING_REQUIREMENTS="true"
@@ -179,27 +205,35 @@ function initialize_common_environment {
         export UPGRADE_WHILE_GENERATING_REQUIREMENTS=${UPGRADE_WHILE_GENERATING_REQUIREMENTS:="false"}
     fi
 
+    # Default extras used for building CI image
     export DEFAULT_CI_EXTRAS="devel_ci"
 
 
+    # Default extras used for building Production image. The master of this information is in the Dockerfile
     DEFAULT_PROD_EXTRAS=$(grep "ARG AIRFLOW_EXTRAS=" "${AIRFLOW_SOURCES}/Dockerfile"|
             awk 'BEGIN { FS="=" } { print $2 }' | tr -d '"')
 
-    # By default we build CI images  but when we specify --producton-image we switch to production image
+    # By default we build CI images  but when we specify `--producton-image` we switch to production image
     export PRODUCTION_IMAGE="false"
 
-    export SQLITE_VERSION="sqlite:////root/airflow/airflow.db"
+    # The SQLlite URL used for sqlite runs
+    export SQLITE_URL="sqlite:////root/airflow/airflow.db"
 
+    # Determines if airflow should be installed from a specified reference in GitHub
     export INSTALL_AIRFLOW_REFERENCE=""
 }
 
-
+# Prints verbose information in case VERBOSE variable is set
 function print_info() {
     if [[ ${VERBOSE:="false"} == "true" ]]; then
         echo "$@"
     fi
 }
 
+# Those are files that are mounted locally when mounting local sources is requested
+# By default not the whole airflow sources directory is mounted because there are often
+# artifacts created there (for example .egg-info files) that are breaking the capability
+# of running different python versions in Breeze. So we only mount what is needed by default.
 LOCAL_MOUNTS="
 .bash_aliases /root/
 .bash_history /root/
@@ -234,8 +268,9 @@ tests /opt/airflow/
 tmp /opt/airflow/
 "
 
-# parse docker-compose-local.yaml file to convert volumes entries
-# from airflow section to "-v" "volume mapping" series of options
+# Converts the local mounts that we defined above to the right set of -v
+# volume mappings in docker-compose file. This is needed so that we only
+# maintain the volumes in one place (above)
 function convert_local_mounts_to_docker_params() {
     echo "${LOCAL_MOUNTS}" |sed '/^$/d' | awk -v AIRFLOW_SOURCES="${AIRFLOW_SOURCES}" \
     '
@@ -246,6 +281,9 @@ function convert_local_mounts_to_docker_params() {
     { print "-v"; print AIRFLOW_SOURCES "/" $1 ":" $2 basename($1) ":cached" }'
 }
 
+# Fixes a file that is expected to be a file. If - for whatever reason - the local file is not created
+# When mounting it to container, docker assumes it is a missing directory and creates it. Such mistakenly
+# Created directories should be removed and replaced with files
 function sanitize_file() {
     if [[ -d "${1}" ]]; then
         rm -rf "${1}"
@@ -253,14 +291,22 @@ function sanitize_file() {
     touch "${1}"
 
 }
+
+# Those files are mounted into container when run locally
+# .bash_history is preserved and you can modify .bash_aliases and .inputrc
+# according to your liking
 function sanitize_mounted_files() {
     sanitize_file "${AIRFLOW_SOURCES}/.bash_history"
     sanitize_file "${AIRFLOW_SOURCES}/.bash_aliases"
     sanitize_file "${AIRFLOW_SOURCES}/.inputrc"
+
+    # When KinD cluster is created, the folder keeps authentication information
+    # across sessions
+    mkdir -p "${MY_DIR}/.kube" >/dev/null 2>&1
 }
 
 #
-# Creates cache directory where we will keep temporary files needed for the build
+# Creates cache directory where we will keep temporary files needed for the docker build
 #
 # This directory will be automatically deleted when the script is killed or exists (via trap)
 # Unless SKIP_CACHE_DELETION variable is set. You can set this variable and then see
@@ -280,6 +326,7 @@ function create_cache_directory() {
     export OUTPUT_LOG
 }
 
+# Removes the cache temporary directory
 function remove_cache_directory() {
     if [[ -d ${CACHE_TMP_FILE_DIR} ]]; then
         rm -rf "${CACHE_TMP_FILE_DIR}"
@@ -480,20 +527,20 @@ function assert_not_in_container() {
     fi
 }
 
+# Removes the "Forced answer" (yes/no/quit) given previously, unles you specifically want to remember it.
+#
+# This is the default behaviour of all rebuild scripts to ask independently whether you want to
+# rebuild the image or not. Sometimes however we want to remember answer previously given. For
+# example if you answered "no" to rebuild the image, the assumption is that you do not
+# want to rebuild image also for other rebuilds in the same pre-commit execution.
+#
+# All the pre-commit checks therefore have `export REMEMBER_LAST_ANSWER="true"` set
+# So that in case they are run in a sequence of commits they will not rebuild. Similarly if your most
+# recent answer was "no" and you run `pre-commit run mypy` (for example) it will also reuse the
+# "no" answer given previously. This happens until you run any of the breeze commands or run all
+# precommits `pre-commit run` - then the "LAST_FORCE_ANSWER_FILE" will be removed and you will
+# be asked again.
 function forget_last_answer() {
-    # Removes the "Forced answer" (yes/no/quit) given previously, unles you specifically want to remember it.
-    #
-    # This is the default behaviour of all rebuild scripts to ask independently whether you want to
-    # rebuild the image or not. Sometimes however we want to remember answer previously given. For
-    # example if you answered "no" to rebuild the image, the assumption is that you do not
-    # want to rebuild image also for other rebuilds in the same pre-commit execution.
-    #
-    # All the pre-commit checks therefore have `export REMEMBER_LAST_ANSWER="true"` set
-    # So that in case they are run in a sequence of commits they will not rebuild. Similarly if your most
-    # recent answer was "no" and you run `pre-commit run mypy` (for example) it will also reuse the
-    # "no" answer given previously. This happens until you run any of the breeze commands or run all
-    # precommits `pre-commit run` - then the "LAST_FORCE_ANSWER_FILE" will be removed and you will
-    # be asked again.
     if [[ ${REMEMBER_LAST_ANSWER:="false"} != "true" ]]; then
         print_info
         print_info "Forgetting last answer from ${LAST_FORCE_ANSWER_FILE}:"
@@ -509,6 +556,11 @@ function forget_last_answer() {
     fi
 }
 
+# Confirms if hte image should be rebuild and interactively checks it with the user.
+# In case iit needs to be rebuild. It only ask the user if it determines that the rebuild
+# is needed and that the rebuild is not already forced. It asks the user using available terminals
+# So that the script works also from within pre-commit run via git hooks - where stdin is not
+# available - it tries to find usable terminal and ask the user via this terminal.
 function confirm_image_rebuild() {
     ACTION="rebuild"
     if [[ ${FORCE_PULL_IMAGES:=} == "true" ]]; then
@@ -516,7 +568,7 @@ function confirm_image_rebuild() {
     fi
     if [[ -f "${LAST_FORCE_ANSWER_FILE}" ]]; then
         # set variable from last answered response given in the same pre-commit run - so that it can be
-        # set in one pre-commit check (build) and then used in another (pylint/mypy/flake8 etc).
+        # answered in teh first pre-commit check (build) and then used in another (pylint/mypy/flake8 etc).
         # shellcheck disable=SC1090
         source "${LAST_FORCE_ANSWER_FILE}"
     fi
@@ -593,7 +645,7 @@ function confirm_image_rebuild() {
 }
 
 # Builds local image manifest
-# It contiains .json file - result of docker inspect - decscribing the image
+# It contains only one .json file - result of docker inspect - describing the image
 # We cannot use docker registry APIs as they are available only with authorisation
 # But this image can be pulled without authentication
 function build_ci_image_manifest() {
@@ -612,8 +664,8 @@ EOF
 }
 
 #
-# Retrieve information about layers in the local IMAGE
-# it stores list of SHAS of image layers in the file pointed at by TMP_MANIFEST_LOCAL_SHA
+# Retrieves information about layers in the local IMAGE
+# it stores list of SHAs of image layers in the file pointed at by TMP_MANIFEST_LOCAL_SHA
 #
 function get_local_image_info() {
     TMP_MANIFEST_LOCAL_JSON=$(mktemp)
@@ -638,8 +690,8 @@ function get_local_image_info() {
 }
 
 #
-# Retrieve information about layers in the remote IMAGE
-# it stores list of SHAS of image layers in the file pointed at by TMP_MANIFEST_REMTOE_SHA
+# Retrieves information about layers in the remote IMAGE
+# it stores list of SHAs of image layers in the file pointed at by TMP_MANIFEST_REMOTE_SHA
 # This cannot be done easily with existing APIs of Dockerhub because they require additional authentication
 # even for public images. Therefore instead we are downloading a specially prepared manifest image
 # which is built together with the main image. This special manifest image is prepared during
@@ -669,7 +721,7 @@ function get_remote_image_info() {
     verbose_docker rm --force "remote-airflow-manifest" >/dev/null 2>&1
 }
 
-# The Number is determines the cut-off between local building time and pull + build time.
+# The Number determines the cut-off between local building time and pull + build time.
 # It is a bit experimental and it will have to be kept
 # updated as we keep on changing layers. The cut-off point is at the moment when we do first
 # pip install "https://github.com/apache/airflow/archive/${AIRFLOW_BRANCH}.tar...
@@ -679,7 +731,7 @@ function get_remote_image_info() {
 #
 # This command returns the number of layer in docker history where pip uninstall is called. This is the
 # line that will take a lot of time to run and at this point it's worth to pull the image from repo
-# if there are at least NN chaanged layers in your docker file, you should pull the image.
+# if there are at least NN changed layers in your docker file, you should pull the image.
 #
 # Note that this only matters if you have any of the important files changed since the last build
 # of your image such as Dockerfile.ci, setup.py etc.
@@ -713,6 +765,11 @@ function compare_layers() {
     fi
 }
 
+# Only rebuilds CI image if needed. It checks if the docker image build is needed
+# because any of the important source files (from common/_files_for_rebuild_check.sh) has
+# changed or in any of the edge cases (docker image removed, .build cache removed etc.
+# In case rebuild is needed, it determines (by comparing layers in local and remote image)
+# Whether pull is needed before rebuild.
 function rebuild_ci_image_if_needed() {
     if [[ ${SKIP_CI_IMAGE_CHECK:="false"} == "true" ]]; then
         echo
@@ -782,9 +839,9 @@ function rebuild_ci_image_if_needed() {
     fi
 }
 
-
 #
-# Starts the script/ If VERBOSE_COMMANDS variable is set to true, it enables verbose output of commands executed
+# Starts the script.
+# If VERBOSE_COMMANDS variable is set to true, it enables verbose output of commands executed
 # Also prints some useful diagnostics information at start of the script if VERBOSE is set to true
 #
 function script_start {
@@ -811,8 +868,10 @@ function script_start {
 }
 
 #
-#
-# Disables verbosity in the script
+# Trap function executed always at the end of the script. In case of verbose output it also
+# Prints the exit code that the script exits with. Removes verbosity of commands in case it was run with
+# command verbosity and in case the script was not run from Breeze (so via ci scripts) it displays
+# total time spent in the script so that we can easily see it.
 #
 function script_end {
     #shellcheck disable=2181
@@ -837,6 +896,7 @@ function script_end {
     remove_cache_directory
 }
 
+# Changes directory to local sources
 function go_to_airflow_sources {
     print_info
     pushd "${AIRFLOW_SOURCES}" &>/dev/null || exit 1
@@ -857,6 +917,9 @@ function basic_sanity_checks() {
 }
 
 
+# Interactive version of confirming the ci image that is used in pre-commits
+# it displays additional information - what the user should do in order to bring the local images
+# back to state that pre-commit will be happy with
 function rebuild_ci_image_if_needed_and_confirmed() {
     NEEDS_DOCKER_BUILD="false"
     THE_IMAGE_TYPE="CI"
@@ -880,7 +943,7 @@ function rebuild_ci_image_if_needed_and_confirmed() {
         echo "You have those options:"
         echo "   * Rebuild the images now by answering 'y' (this might take some time!)"
         echo "   * Skip rebuilding the images and hope changes are not big (you will be asked again)"
-        echo "   * Quit and manually rebuild the images using one of the following commmands"
+        echo "   * Quit and manually rebuild the images using one of the following commands"
         echo "        * ./breeze build-image"
         echo "        * ./breeze build-image --force-pull-images"
         echo
@@ -896,7 +959,8 @@ function rebuild_ci_image_if_needed_and_confirmed() {
     fi
 }
 
-
+# Checks if any of the files match the regexp specified the parameters here should be
+# match_files_regexp REGEXP FILE FILE ...
 function match_files_regexp() {
     FILE_MATCHES="false"
     REGEXP=${1}
@@ -914,6 +978,11 @@ function match_files_regexp() {
     export FILE_MATCHES
 }
 
+# Retrieves CI environment variables needed - depending on the CI system we run it in.
+# We try to be CI - agnostic and our scripts should run the same way on different CI systems
+# (This makes it easy to move between different CI systems)
+# This function maps CI-specific variables into a generic ones (prefixed with CI_) that
+# we used in other scripts
 function get_ci_environment() {
     export CI_EVENT_TYPE="manual"
     export CI_TARGET_REPO="apache/airflow"
@@ -966,7 +1035,9 @@ function get_ci_environment() {
     echo
 }
 
-
+# Builds the CI image in the CI environment.
+# Depending on the type of build (push/pr/scheduled) it will either build it incrementally or
+# from the scratch without cache (the latter for scheduled builds only)
 function build_ci_image_on_ci() {
     get_ci_environment
 
@@ -989,20 +1060,27 @@ function build_ci_image_on_ci() {
 
     rebuild_ci_image_if_needed
 
-    # Disable force pulling forced above
+    # Disable force pulling forced above this is needed for the subsequent scripts so that
+    # They do not try to pull/build images again
     unset FORCE_PULL_IMAGES
     unset FORCE_BUILD
 }
 
+# Reads environment variable passed as first parameter from the .build cache file
 function read_from_file {
     cat "${BUILD_CACHE_DIR}/.$1" 2>/dev/null || true
 }
 
+# Saves environment variable passed as first parameter to the .build cache file
 function save_to_file {
     # shellcheck disable=SC2005
     echo "$(eval echo "\$$1")" > "${BUILD_CACHE_DIR}/.$1"
 }
 
+# check if parameter set for the variable is allowed (should be on the _BREEZE_ALLOWED list)
+# and if it is, it saves it to .build cache file. In case the parameter is wrong, the
+# saved variable is removed (so that bad value is not used again in case it comes from there)
+# and exits with an error
 function check_and_save_allowed_param {
     _VARIABLE_NAME="${1}"
     _VARIABLE_DESCRIPTIVE_NAME="${2}"
@@ -1029,6 +1107,7 @@ function check_and_save_allowed_param {
     save_to_file "${_VARIABLE_NAME}"
 }
 
+# Docker command to build documentation
 function run_docs() {
     verbose_docker run "${EXTRA_DOCKER_FLAGS[@]}" -t \
             --entrypoint "/usr/local/bin/dumb-init"  \
@@ -1043,6 +1122,7 @@ function run_docs() {
             | tee -a "${OUTPUT_LOG}"
 }
 
+# Pulls image in case it is needed (either has never been pulled or pulling was forced
 # Should be run with set +e
 # Parameters:
 #   $1 -> image to pull
@@ -1066,6 +1146,9 @@ function pull_image_if_needed() {
     fi
 }
 
+# Pulls image if needed but tries to pull it from cache (for example GitHub registry) before
+# It attempts to pull it from the main repository. This is used to speed up the builds
+# In GitHub Actions.
 # Parameters:
 #   $1 -> image to pull
 #   $2 -> cache image to pull first
@@ -1089,6 +1172,7 @@ function pull_image_possibly_from_cache() {
     set -e
 }
 
+# Pulls CI image in case caching strategy is "pulled" and the image needs to be pulled
 function pull_ci_image_if_needed() {
     # Whether to force pull images to populate cache
     export FORCE_PULL_IMAGES=${FORCE_PULL_IMAGES:="false"}
@@ -1111,6 +1195,7 @@ Docker pulling ${PYTHON_BASE_IMAGE}.
 }
 
 
+# Pulls PROD image in case caching strategy is "pulled" and the image needs to be pulled
 function pull_prod_images_if_needed() {
     # Whether to force pull images to populate cache
     export FORCE_PULL_IMAGES=${FORCE_PULL_IMAGES:="false"}
@@ -1130,12 +1215,16 @@ function pull_prod_images_if_needed() {
     fi
 }
 
+# Prints summary of the build parameters
 function print_build_info() {
     print_info
     print_info "Airflow ${AIRFLOW_VERSION} Python: ${PYTHON_MAJOR_MINOR_VERSION}. Image description: ${IMAGE_DESCRIPTION}"
     print_info
 }
 
+# Function to spin ASCII spinner during pull and build in pre-commits to give the user indication that
+# Pull/Build is happening. It only spins if the output log changes, so if pull/build is stalled
+# The spinner will not move.
 function spin() {
     local FILE_TO_MONITOR=${1}
     local SPIN=("-" "\\" "|" "/")
@@ -1169,6 +1258,10 @@ Build log: ${FILE_TO_MONITOR}
     done
 }
 
+# Builds CI image - depending on the caching strategy (pulled, local, no-cache) it
+# passes the necessary docker build flags via DOCKER_CACHE_CI_DIRECTIVE array
+# it also passes the right Build args depending on the configuration of the build
+# selected by Breeze flags or environment variables.
 function build_ci_image() {
     print_build_info
     if [[ -n ${DETECTED_TERMINAL:=""} ]]; then
@@ -1224,6 +1317,11 @@ Docker building ${AIRFLOW_CI_IMAGE}.
     fi
 }
 
+# Builds PROD image - depending on the caching strategy (pulled, local, no-cache) it
+# passes the necessary docker build flags via DOCKER_CACHE_PROD_DIRECTIVE and
+# DOCKER_CACHE_PROD_BUILD_DIRECTIVE (separate caching options are needed for "build" segment of the image)
+# it also passes the right Build args depending on the configuration of the build
+# selected by Breeze flags or environment variables.
 function build_prod_image() {
     print_build_info
     pull_prod_images_if_needed
@@ -1275,7 +1373,7 @@ function build_prod_image() {
     fi
 }
 
-
+# Removes airflow CI and base images
 function remove_all_images() {
     echo
     "${AIRFLOW_SOURCES}/confirm" "Removing all local images ."
@@ -1312,6 +1410,7 @@ function filterout_deleted_files {
   xargs -0 "$STAT_BIN" --printf '%n\0' 2>/dev/null || true;
 }
 
+# Fixes permissions for groups for all the files that are quickly filtered using the filterout_deleted_files
 function fix_group_permissions() {
     if [[ ${PERMISSIONS_FIXED:=} == "true" ]]; then
         echo
@@ -1332,6 +1431,8 @@ function fix_group_permissions() {
     export PERMISSIONS_FIXED="true"
 }
 
+# Prepares all variables needed by the CI build. Depending on the configuration used (python version
+# DockerHub user etc. the variables are set so that other functions can use those variables.
 function prepare_ci_build() {
     export AIRFLOW_CI_BASE_TAG="${DEFAULT_BRANCH}-python${PYTHON_MAJOR_MINOR_VERSION}-ci"
     export AIRFLOW_CI_LOCAL_MANIFEST_IMAGE="local/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}-manifest"
@@ -1365,6 +1466,9 @@ function prepare_ci_build() {
     fix_group_permissions
 }
 
+# For remote installation of airflow (from GitHub or Pypi) when building the image, you need to
+# pass build flags depending on the version and method of the installation (for example to
+# get proper requirement constraint files)
 function add_build_args_for_remote_install() {
     # entrypoint is used as AIRFLOW_SOURCES_FROM/TO in order to avoid costly copying of all sources of
     # Airflow - those are not needed for remote install at all. Entrypoint is later overwritten by
@@ -1399,6 +1503,8 @@ function add_build_args_for_remote_install() {
     fi
 }
 
+# Prepares all variables needed by the CI build. Depending on the configuration used (python version
+# DockerHub user etc. the variables are set so that other functions can use those variables.
 function prepare_prod_build() {
     export AIRFLOW_PROD_BASE_TAG="${DEFAULT_BRANCH}-python${PYTHON_MAJOR_MINOR_VERSION}"
     export AIRFLOW_PROD_BUILD_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_PROD_BASE_TAG}-build"
@@ -1449,6 +1555,8 @@ function prepare_prod_build() {
     go_to_airflow_sources
 }
 
+# Pushes Ci image and it's manifest to the registry. In case the image was taken from cache registry
+# it is also pushed to the cache, not to the main registry. Manifest is only pushed to the main registry
 function push_ci_image() {
     if [[ ${CACHED_AIRFLOW_CI_IMAGE:=} != "" ]]; then
         verbose_docker tag "${AIRFLOW_CI_IMAGE}" "${CACHED_AIRFLOW_CI_IMAGE}"
@@ -1467,6 +1575,8 @@ function push_ci_image() {
     fi
 }
 
+# Pushes PROD image to the registry. In case the image was taken from cache registry
+# it is also pushed to the cache, not to the main registry
 function push_prod_images() {
     if [[ ${CACHED_AIRFLOW_PROD_IMAGE:=} != "" ]]; then
         verbose_docker tag "${AIRFLOW_PROD_IMAGE}" "${CACHED_AIRFLOW_PROD_IMAGE}"
@@ -1487,6 +1597,7 @@ function push_prod_images() {
     fi
 }
 
+# Docker command to generate constraint requirement files.
 function run_generate_requirements() {
     docker run "${EXTRA_DOCKER_FLAGS[@]}" \
         --entrypoint "/usr/local/bin/dumb-init"  \
@@ -1516,7 +1627,8 @@ function set_mysql_encoding() {
     fi
 }
 
-
+# Retrieves version of airflow stored in the production image (used to display the actual
+# Version we use if it was build from PyPI or GitHub
 function get_airflow_version_from_production_image() {
      docker run --entrypoint /bin/bash "${AIRFLOW_PROD_IMAGE}" -c 'echo "${AIRFLOW_VERSION}"'
 }
diff --git a/scripts/ci/docker-compose/backend-sqlite.yml b/scripts/ci/docker-compose/backend-sqlite.yml
index 5bc8a7e..50972b6 100644
--- a/scripts/ci/docker-compose/backend-sqlite.yml
+++ b/scripts/ci/docker-compose/backend-sqlite.yml
@@ -20,5 +20,5 @@ services:
   airflow:
     environment:
       - BACKEND=sqlite
-      - AIRFLOW__CORE__SQL_ALCHEMY_CONN=${SQLITE_VERSION}
+      - AIRFLOW__CORE__SQL_ALCHEMY_CONN=${SQLITE_URL}
       - AIRFLOW__CORE__EXECUTOR=SequentialExecutor


[airflow] 08/36: Use Debian's provided JRE from Buster (#8919)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 8d8d03eead37f90ae763df851ee87dd890a1578c
Author: Ash Berlin-Taylor <as...@firemirror.com>
AuthorDate: Wed May 20 14:18:59 2020 +0100

    Use Debian's provided JRE from Buster (#8919)
    
    Installing the JDK (not even the JRE) from Sid is starting to break on
    Buster as the versions of packages conflict:
    
    > The following packages have unmet dependencies:
    > libgcc-8-dev : Depends: gcc-8-base (= 8.4.0-4) but 8.3.0-6 is to be installed
    >                Depends: libmpx2 (>= 8.4.0-4) but 8.3.0-6 is to be installed
    
    This changes our CI docker images to:
    
    1. Not install something from Sid (unstable, packages change/get
       updated) when we are using Buster (stable, only security fixes).
    2. Installed the JRE, not the JDK. We don't need to compile Java code.
    
    (cherry picked from commit fef00e5a067109b4bd7756dac09bb718de386865)
---
 Dockerfile.ci | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)

diff --git a/Dockerfile.ci b/Dockerfile.ci
index ffa9a28..7a76295 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -41,7 +41,7 @@ ENV DEBIAN_FRONTEND=noninteractive LANGUAGE=C.UTF-8 LANG=C.UTF-8 LC_ALL=C.UTF-8
     LC_CTYPE=C.UTF-8 LC_MESSAGES=C.UTF-8
 
 # By increasing this number we can do force build of all dependencies
-ARG DEPENDENCIES_EPOCH_NUMBER="1"
+ARG DEPENDENCIES_EPOCH_NUMBER="3"
 # Increase the value below to force renstalling of all dependencies
 ENV DEPENDENCIES_EPOCH_NUMBER=${DEPENDENCIES_EPOCH_NUMBER}
 
@@ -119,14 +119,11 @@ RUN adduser airflow \
 # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199
 RUN mkdir -pv /usr/share/man/man1 \
     && mkdir -pv /usr/share/man/man7 \
-    && curl -fsSL https://adoptopenjdk.jfrog.io/adoptopenjdk/api/gpg/key/public | apt-key add - \
-    && echo 'deb https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/ buster main' > \
-        /etc/apt/sources.list.d/adoptopenjdk.list \
     && apt-get update \
     && apt-get install --no-install-recommends -y \
       gnupg \
       libgcc-8-dev \
-      adoptopenjdk-8-hotspot-jre \
+      default-jre-headless \
       apt-transport-https \
       bash-completion \
       ca-certificates \
@@ -150,8 +147,6 @@ RUN mkdir -pv /usr/share/man/man1 \
     && apt-get clean \
     && rm -rf /var/lib/apt/lists/*
 
-ENV JAVA_HOME=/usr/lib/jvm/adoptopenjdk-8-hotspot-jre-amd64
-
 # Install Hadoop and Hive
 # It is done in one step to share variables.
 ENV HADOOP_HOME="/opt/hadoop-cdh" HIVE_HOME="/opt/hive"
@@ -275,7 +270,7 @@ RUN mkdir -pv ${AIRFLOW_HOME} \
     mkdir -pv ${AIRFLOW_HOME}/logs
 
 # Increase the value here to force reinstalling Apache Airflow pip dependencies
-ARG PIP_DEPENDENCIES_EPOCH_NUMBER="2"
+ARG PIP_DEPENDENCIES_EPOCH_NUMBER="3"
 ENV PIP_DEPENDENCIES_EPOCH_NUMBER=${PIP_DEPENDENCIES_EPOCH_NUMBER}
 
 # Optimizing installation of Cassandra driver
@@ -373,7 +368,7 @@ COPY .github/ ${AIRFLOW_SOURCES}/.github/
 COPY dags/ ${AIRFLOW_SOURCES}/dags/
 COPY common/ ${AIRFLOW_SOURCES}/common/
 COPY licenses/ ${AIRFLOW_SOURCES}/licenses/
-COPY scripts/ci/ ${AIRFLOW_SOURCES}/scripts/ci/
+COPY scripts/ ${AIRFLOW_SOURCES}/scripts/
 COPY docs/ ${AIRFLOW_SOURCES}/docs/
 COPY tests/ ${AIRFLOW_SOURCES}/tests/
 COPY airflow/ ${AIRFLOW_SOURCES}/airflow/


[airflow] 17/36: Prevents failure on fixing permissions for files with space in it (#9076)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 4c6a82a1a9b156cf5b79e6b6044a1654e65f0a67
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sat May 30 21:02:31 2020 +0200

    Prevents failure on fixing permissions for files with space in it (#9076)
    
    (cherry picked from commit 88c8edb898969c2bb6b9cbc04f504691c67c91ff)
---
 scripts/ci/in_container/_in_container_utils.sh | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/scripts/ci/in_container/_in_container_utils.sh b/scripts/ci/in_container/_in_container_utils.sh
index b1de08b..be386e0 100644
--- a/scripts/ci/in_container/_in_container_utils.sh
+++ b/scripts/ci/in_container/_in_container_utils.sh
@@ -39,11 +39,19 @@ function in_container_script_end() {
     #shellcheck disable=2181
     EXIT_CODE=$?
     if [[ ${EXIT_CODE} != 0 ]]; then
+        if [[ -n ${OUT_FILE:=} ]]; then
+            echo "  ERROR ENCOUNTERED!"
+            echo
+            echo "  Output:"
+            echo
+            cat "${OUT_FILE}"
+            echo "###########################################################################################"
+        fi
         echo "###########################################################################################"
         echo "                   EXITING ${0} WITH STATUS CODE ${EXIT_CODE}"
         echo "###########################################################################################"
-
     fi
+
     if [[ ${VERBOSE_COMMANDS} == "true" ]]; then
         set +x
     fi
@@ -85,8 +93,8 @@ function in_container_cleanup_pycache() {
 #
 function in_container_fix_ownership() {
     set +o pipefail
-    sudo find . -user root \
-    | sudo xargs chown -v "${HOST_USER_ID}.${HOST_GROUP_ID}" --no-dereference >/dev/null 2>&1
+    sudo find . -user root -print0 \
+    | sudo xargs --null chown -v "${HOST_USER_ID}.${HOST_GROUP_ID}" --no-dereference >/dev/null 2>&1
     set -o pipefail
 }
 
@@ -254,7 +262,7 @@ function send_kubernetes_logs_to_file_io() {
 
 function install_released_airflow_version() {
     pip uninstall -y apache-airflow || true
-    find /root/airflow/ -type f -print0 | xargs rm -f --
+    find /root/airflow/ -type f -print0 | xargs -0 rm -f --
     if [[ ${1} == "1.10.2" || ${1} == "1.10.1" ]]; then
         export SLUGIFY_USES_TEXT_UNIDECODE=yes
     fi


[airflow] 12/36: Add ADDITIONAL_PYTHON_DEPS (#9031)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 51a215c331e41e704534284f54ef32a6939a9471
Author: Fabian <fa...@t-online.de>
AuthorDate: Wed May 27 11:52:26 2020 +0200

    Add ADDITIONAL_PYTHON_DEPS (#9031)
    
    * add build-arg ADDITIONAL_PYTHON_DEPS
    
    * Add ADDITIONAL_PYTHON_DEPS example and description
    
    Co-authored-by: Fabian Witt <fa...@redheads.de>
    (cherry picked from commit 6fc555d0bc86fd752ac5162e4730d309c5828011)
---
 Dockerfile |  6 ++++++
 IMAGES.rst | 19 +++++++++++++++++++
 2 files changed, 25 insertions(+)

diff --git a/Dockerfile b/Dockerfile
index 429a723..2ca5046 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -35,6 +35,7 @@
 #
 ARG AIRFLOW_VERSION="2.0.0.dev0"
 ARG AIRFLOW_EXTRAS="async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv"
+ARG ADDITIONAL_PYTHON_DEPS=""
 
 ARG AIRFLOW_HOME=/opt/airflow
 ARG AIRFLOW_UID="50000"
@@ -169,6 +170,9 @@ ENV AIRFLOW_VERSION=${AIRFLOW_VERSION}
 ARG AIRFLOW_EXTRAS
 ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}
 
+ARG ADDITIONAL_PYTHON_DEPS
+ENV ADDITIONAL_PYTHON_DEPS=${ADDITIONAL_PYTHON_DEPS}
+
 ARG AIRFLOW_INSTALL_SOURCES="."
 ENV AIRFLOW_INSTALL_SOURCES=${AIRFLOW_INSTALL_SOURCES}
 
@@ -187,6 +191,8 @@ ENV PATH=${PATH}:/root/.local/bin
 
 RUN pip install --user "${AIRFLOW_INSTALL_SOURCES}[${AIRFLOW_EXTRAS}]${AIRFLOW_INSTALL_VERSION}" \
     --constraint /requirements.txt && \
+    if [ -n "${ADDITIONAL_PYTHON_DEPS}" ]; then pip install --user ${ADDITIONAL_PYTHON_DEPS} \
+    --constraint /requirements.txt; fi && \
     find /root/.local/ -name '*.pyc' -print0 | xargs -0 rm -r && \
     find /root/.local/ -type d -name '__pycache__' -print0 | xargs -0 rm -r
 
diff --git a/IMAGES.rst b/IMAGES.rst
index 545481b..5119a16 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -257,6 +257,9 @@ The following build arguments (``--build-arg`` in docker build command) can be u
 | ``AIRFLOW_EXTRAS``                       | (see Dockerfile)                         | Default extras with which airflow is     |
 |                                          |                                          | installed                                |
 +------------------------------------------+------------------------------------------+------------------------------------------+
+| ``ADDITIONAL_PYTHON_DEPS``               | (see Dockerfile)                         | Optional python packages to extend       |
+|                                          |                                          | the image with some extra dependencies   |
++------------------------------------------+------------------------------------------+------------------------------------------+
 | ``AIRFLOW_HOME``                         | ``/opt/airflow``                         | Airflow’s HOME (that’s where logs and    |
 |                                          |                                          | sqlite databases are stored)             |
 +------------------------------------------+------------------------------------------+------------------------------------------+
@@ -365,6 +368,22 @@ requirements taken from v1-10-test branch in Github.
     --build-arg AIRFLOW_SOURCES_FROM="entrypoint.sh" \
     --build-arg AIRFLOW_SOURCES_TO="/entrypoint"
 
+This builds the production image in version 3.7 with default airflow extras from 1.10.10 Pypi package and
+additional python dependencies.
+
+.. code-block::
+
+  docker build . \
+    --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
+    --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \
+    --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \
+    --build-arg AIRFLOW_INSTALL_VERSION="==1.10.10" \
+    --build-arg CONSTRAINT_REQUIREMENTS="https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt" \
+    --build-arg ENTRYPOINT_FILE="https://raw.githubusercontent.com/apache/airflow/1.10.10/entrypoint.sh" \
+    --build-arg AIRFLOW_SOURCES_FROM="entrypoint.sh" \
+    --build-arg AIRFLOW_SOURCES_TO="/entrypoint" \
+    --build-arg ADDITIONAL_PYTHON_DEPS="sshtunnel oauth2client"
+
 Image manifests
 ---------------
 


[airflow] 21/36: Produce less verbose output when building docker mount options (#9103)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit fbe1fe3d6fc9eff00ca39b804d03815d087d324f
Author: Ash Berlin-Taylor <as...@firemirror.com>
AuthorDate: Tue Jun 2 14:53:43 2020 +0100

    Produce less verbose output when building docker mount options (#9103)
    
    The previous method of generating this list had two "problems"/niggles
    that this PR solves, when running with VERBOSE=true
    
    - Firstly, LOCAL_MOUNTS was set at the top level, so running with
      `set -x` produced 30 extra lines of output.
    - Because of the `while read` used, it created 4 or 5 lines _per_ mount,
      resulting in a lot verbose output.
    
    Nothing I've changed here is "critical", it's just making it a bit
    easier to see with the debug output what is going on, by running fewer
    commands.
    
    I have also expanded the BATS test a little bit to check each pair (`-v`
    and its following option)
    
    (cherry picked from commit 9e0ccdea2050659958b722c6197b1e319d189c1d)
---
 scripts/ci/_utils.sh                      | 85 +++++++++++++++----------------
 scripts/ci/pre_commit_local_yml_mounts.sh | 26 ++++------
 tests/bats/test_local_mounts.bats         | 79 +++++-----------------------
 3 files changed, 63 insertions(+), 127 deletions(-)

diff --git a/scripts/ci/_utils.sh b/scripts/ci/_utils.sh
index 4ba7d83..6766a3a 100644
--- a/scripts/ci/_utils.sh
+++ b/scripts/ci/_utils.sh
@@ -145,11 +145,7 @@ function initialize_common_environment {
         print_info "Mounting necessary host volumes to Docker"
         print_info
 
-        EXTRA_DOCKER_FLAGS=()
-
-        while IFS= read -r LINE; do
-            EXTRA_DOCKER_FLAGS+=( "${LINE}")
-        done < <(convert_local_mounts_to_docker_params)
+        read -r -a EXTRA_DOCKER_FLAGS <<< "$(convert_local_mounts_to_docker_params)"
     else
         print_info
         print_info "Skip mounting host volumes to Docker"
@@ -238,51 +234,50 @@ function print_info() {
 # By default not the whole airflow sources directory is mounted because there are often
 # artifacts created there (for example .egg-info files) that are breaking the capability
 # of running different python versions in Breeze. So we only mount what is needed by default.
-LOCAL_MOUNTS="
-.bash_aliases /root/
-.bash_history /root/
-.coveragerc /opt/airflow/
-.dockerignore /opt/airflow/
-.flake8 /opt/airflow/
-.github /opt/airflow/
-.inputrc /root/
-.kube /root/
-.rat-excludes /opt/airflow/
-CHANGELOG.txt /opt/airflow/
-Dockerfile.ci /opt/airflow/
-LICENSE /opt/airflow/
-MANIFEST.in /opt/airflow/
-NOTICE /opt/airflow/
-airflow /opt/airflow/
-common /opt/airflow/
-dags /opt/airflow/
-dev /opt/airflow/
-docs /opt/airflow/
-files /
-dist /
-hooks /opt/airflow/
-logs /root/airflow/
-pytest.ini /opt/airflow/
-requirements /opt/airflow/
-scripts /opt/airflow/
-scripts/ci/in_container/entrypoint_ci.sh /
-setup.cfg /opt/airflow/
-setup.py /opt/airflow/
-tests /opt/airflow/
-tmp /opt/airflow/
-"
+function generate_local_mounts_list {
+    local prefix="$1"
+    LOCAL_MOUNTS=(
+        "$prefix".bash_aliases:/root/.bash_aliases:cached
+        "$prefix".bash_history:/root/.bash_history:cached
+        "$prefix".coveragerc:/opt/airflow/.coveragerc:cached
+        "$prefix".dockerignore:/opt/airflow/.dockerignore:cached
+        "$prefix".flake8:/opt/airflow/.flake8:cached
+        "$prefix".github:/opt/airflow/.github:cached
+        "$prefix".inputrc:/root/.inputrc:cached
+        "$prefix".kube:/root/.kube:cached
+        "$prefix".rat-excludes:/opt/airflow/.rat-excludes:cached
+        "$prefix"CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached
+        "$prefix"Dockerfile.ci:/opt/airflow/Dockerfile.ci:cached
+        "$prefix"LICENSE:/opt/airflow/LICENSE:cached
+        "$prefix"MANIFEST.in:/opt/airflow/MANIFEST.in:cached
+        "$prefix"NOTICE:/opt/airflow/NOTICE:cached
+        "$prefix"airflow:/opt/airflow/airflow:cached
+        "$prefix"common:/opt/airflow/common:cached
+        "$prefix"dags:/opt/airflow/dags:cached
+        "$prefix"dev:/opt/airflow/dev:cached
+        "$prefix"docs:/opt/airflow/docs:cached
+        "$prefix"files:/files:cached
+        "$prefix"dist:/dist:cached
+        "$prefix"hooks:/opt/airflow/hooks:cached
+        "$prefix"logs:/root/airflow/logs:cached
+        "$prefix"pytest.ini:/opt/airflow/pytest.ini:cached
+        "$prefix"requirements:/opt/airflow/requirements:cached
+        "$prefix"scripts:/opt/airflow/scripts:cached
+        "$prefix"scripts/ci/in_container/entrypoint_ci.sh:/entrypoint_ci.sh:cached
+        "$prefix"setup.cfg:/opt/airflow/setup.cfg:cached
+        "$prefix"setup.py:/opt/airflow/setup.py:cached
+        "$prefix"tests:/opt/airflow/tests:cached
+        "$prefix"tmp:/opt/airflow/tmp:cached
+    )
+}
 
 # Converts the local mounts that we defined above to the right set of -v
 # volume mappings in docker-compose file. This is needed so that we only
 # maintain the volumes in one place (above)
 function convert_local_mounts_to_docker_params() {
-    echo "${LOCAL_MOUNTS}" |sed '/^$/d' | awk -v AIRFLOW_SOURCES="${AIRFLOW_SOURCES}" \
-    '
-    function basename(file) {
-        sub(".*/", "", file)
-        return file
-    }
-    { print "-v"; print AIRFLOW_SOURCES "/" $1 ":" $2 basename($1) ":cached" }'
+    generate_local_mounts_list "${AIRFLOW_SOURCES}/"
+    # Bash can't "return" arrays, so we need to quote any special characters
+    printf -- '-v %q ' "${LOCAL_MOUNTS[@]}"
 }
 
 # Fixes a file that is expected to be a file. If - for whatever reason - the local file is not created
diff --git a/scripts/ci/pre_commit_local_yml_mounts.sh b/scripts/ci/pre_commit_local_yml_mounts.sh
index a614184..d77a528 100755
--- a/scripts/ci/pre_commit_local_yml_mounts.sh
+++ b/scripts/ci/pre_commit_local_yml_mounts.sh
@@ -23,29 +23,21 @@ MY_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
-TMP_FILE=$(mktemp)
 TMP_OUTPUT=$(mktemp)
 
+# Remove temp file if it's hanging around
+trap 'rm -rf -- "${TMP_OUTPUT}" 2>/dev/null' EXIT
+
 LOCAL_YML_FILE="${MY_DIR}/docker-compose/local.yml"
 
 LEAD='      # START automatically generated volumes from LOCAL_MOUNTS in _utils.sh'
 TAIL='      # END automatically generated volumes from LOCAL_MOUNTS in _utils.sh'
 
-echo "${LOCAL_MOUNTS}" |sed '/^$/d' | \
-    awk '
-    function basename(file) {
-        sub(".*/", "", file)
-        return file
-    }
-    { print "      - ../../../" $1 ":" $2 basename($1) ":cached"}
-    ' > "${TMP_FILE}"
-
-
-BEGIN_GEN=$(grep -n "${LEAD}" <"${LOCAL_YML_FILE}" | sed 's/\(.*\):.*/\1/g')
-END_GEN=$(grep -n "${TAIL}" <"${LOCAL_YML_FILE}" | sed 's/\(.*\):.*/\1/g')
-cat <(head -n "${BEGIN_GEN}" "${LOCAL_YML_FILE}") \
-    "${TMP_FILE}" \
-    <(tail -n +"${END_GEN}" "${LOCAL_YML_FILE}") \
-    >"${TMP_OUTPUT}"
+generate_local_mounts_list "      - ../../../"
+
+sed -ne "0,/$LEAD/ p" "${LOCAL_YML_FILE}" > "${TMP_OUTPUT}"
+
+printf '%s\n' "${LOCAL_MOUNTS[@]}" >> "${TMP_OUTPUT}"
+sed -ne "/$TAIL/,\$ p" "${LOCAL_YML_FILE}" >> "${TMP_OUTPUT}"
 
 mv "${TMP_OUTPUT}" "${LOCAL_YML_FILE}"
diff --git a/tests/bats/test_local_mounts.bats b/tests/bats/test_local_mounts.bats
index e8e9384..97cf984 100644
--- a/tests/bats/test_local_mounts.bats
+++ b/tests/bats/test_local_mounts.bats
@@ -23,69 +23,18 @@
 
   initialize_common_environment
 
-  run convert_local_mounts_to_docker_params
-  diff <(echo "${output}") - << EOF
--v
-${AIRFLOW_SOURCES}/.bash_aliases:/root/.bash_aliases:cached
--v
-${AIRFLOW_SOURCES}/.bash_history:/root/.bash_history:cached
--v
-${AIRFLOW_SOURCES}/.coveragerc:/opt/airflow/.coveragerc:cached
--v
-${AIRFLOW_SOURCES}/.dockerignore:/opt/airflow/.dockerignore:cached
--v
-${AIRFLOW_SOURCES}/.flake8:/opt/airflow/.flake8:cached
--v
-${AIRFLOW_SOURCES}/.github:/opt/airflow/.github:cached
--v
-${AIRFLOW_SOURCES}/.inputrc:/root/.inputrc:cached
--v
-${AIRFLOW_SOURCES}/.kube:/root/.kube:cached
--v
-${AIRFLOW_SOURCES}/.rat-excludes:/opt/airflow/.rat-excludes:cached
--v
-${AIRFLOW_SOURCES}/CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached
--v
-${AIRFLOW_SOURCES}/Dockerfile.ci:/opt/airflow/Dockerfile.ci:cached
--v
-${AIRFLOW_SOURCES}/LICENSE:/opt/airflow/LICENSE:cached
--v
-${AIRFLOW_SOURCES}/MANIFEST.in:/opt/airflow/MANIFEST.in:cached
--v
-${AIRFLOW_SOURCES}/NOTICE:/opt/airflow/NOTICE:cached
--v
-${AIRFLOW_SOURCES}/airflow:/opt/airflow/airflow:cached
--v
-${AIRFLOW_SOURCES}/common:/opt/airflow/common:cached
--v
-${AIRFLOW_SOURCES}/dags:/opt/airflow/dags:cached
--v
-${AIRFLOW_SOURCES}/dev:/opt/airflow/dev:cached
--v
-${AIRFLOW_SOURCES}/docs:/opt/airflow/docs:cached
--v
-${AIRFLOW_SOURCES}/files:/files:cached
--v
-${AIRFLOW_SOURCES}/dist:/dist:cached
--v
-${AIRFLOW_SOURCES}/hooks:/opt/airflow/hooks:cached
--v
-${AIRFLOW_SOURCES}/logs:/root/airflow/logs:cached
--v
-${AIRFLOW_SOURCES}/pytest.ini:/opt/airflow/pytest.ini:cached
--v
-${AIRFLOW_SOURCES}/requirements:/opt/airflow/requirements:cached
--v
-${AIRFLOW_SOURCES}/scripts:/opt/airflow/scripts:cached
--v
-${AIRFLOW_SOURCES}/scripts/ci/in_container/entrypoint_ci.sh:/entrypoint_ci.sh:cached
--v
-${AIRFLOW_SOURCES}/setup.cfg:/opt/airflow/setup.cfg:cached
--v
-${AIRFLOW_SOURCES}/setup.py:/opt/airflow/setup.py:cached
--v
-${AIRFLOW_SOURCES}/tests:/opt/airflow/tests:cached
--v
-${AIRFLOW_SOURCES}/tmp:/opt/airflow/tmp:cached
-EOF
+  read -r -a RES <<< "$(convert_local_mounts_to_docker_params)"
+
+  [[ ${#RES[@]} -gt 0 ]] # Array should be non-zero length
+  [[ $((${#RES[@]} % 2)) == 0 ]] # Array should be even length
+
+  for i in "${!RES[@]}"; do
+    if [[ $((i % 2)) == 0 ]]; then
+      # Every other value should be `-v`
+      [[ ${RES[$i]} == "-v" ]]
+    else
+      # And the options should be of form <src>:<dest>:cached
+      [[ ${RES[$i]} = *:*:cached ]]
+    fi
+  done
 }


[airflow] 30/36: Use static binary linked docker client in CI image (#9126)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 410bcd4cc5e86d1a993daed9ae4a1d5de5413989
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Thu Jun 4 15:24:51 2020 +0200

    Use static binary linked docker client in CI image (#9126)
    
    
    (cherry picked from commit 46fee7715635b0c0e754ee4eac3c969020cdfcc9)
---
 Dockerfile.ci                                  | 12 ++---
 scripts/ci/_utils.sh                           |  1 -
 scripts/ci/docker-compose/local.yml            |  1 -
 scripts/ci/in_container/_in_container_utils.sh | 43 -----------------
 scripts/ci/in_container/check_environment.sh   | 67 +-------------------------
 scripts/ci/in_container/run_ci_tests.sh        |  1 -
 scripts/ci/in_container/run_system_tests.sh    |  1 -
 7 files changed, 6 insertions(+), 120 deletions(-)

diff --git a/Dockerfile.ci b/Dockerfile.ci
index 1c93f27..d15cff0 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -146,13 +146,11 @@ RUN mkdir -pv /usr/share/man/man1 \
     && apt-get clean \
     && rm -rf /var/lib/apt/lists/*
 
-# Install Docker
-RUN curl --fail --location https://download.docker.com/linux/debian/gpg | apt-key add - \
-    && add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian stretch stable" \
-    && apt-get update \
-    && apt-get -y install --no-install-recommends docker-ce \
-    && apt-get autoremove -yqq --purge \
-    && apt-get clean && rm -rf /var/lib/apt/lists/*
+ARG DOCKER_CLI_VERSION=19.03.9
+ENV DOCKER_CLI_VERSION=${DOCKER_CLI_VERSION}
+
+RUN curl https://download.docker.com/linux/static/stable/x86_64/docker-${DOCKER_CLI_VERSION}.tgz \
+    |  tar -C /usr/bin --strip-components=1 -xvzf - docker/docker
 
 # Setup PIP
 # By default PIP install run without cache to make image smaller
diff --git a/scripts/ci/_utils.sh b/scripts/ci/_utils.sh
index 8ac47a0..d0b8251 100644
--- a/scripts/ci/_utils.sh
+++ b/scripts/ci/_utils.sh
@@ -260,7 +260,6 @@ function generate_local_mounts_list {
         "$prefix".kube:/root/.kube:cached
         "$prefix".rat-excludes:/opt/airflow/.rat-excludes:cached
         "$prefix"CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached
-        "$prefix"Dockerfile.ci:/opt/airflow/Dockerfile.ci:cached
         "$prefix"LICENSE:/opt/airflow/LICENSE:cached
         "$prefix"MANIFEST.in:/opt/airflow/MANIFEST.in:cached
         "$prefix"NOTICE:/opt/airflow/NOTICE:cached
diff --git a/scripts/ci/docker-compose/local.yml b/scripts/ci/docker-compose/local.yml
index 2991de1..42609e6 100644
--- a/scripts/ci/docker-compose/local.yml
+++ b/scripts/ci/docker-compose/local.yml
@@ -35,7 +35,6 @@ services:
       - ../../../.kube:/root/.kube:cached
       - ../../../.rat-excludes:/opt/airflow/.rat-excludes:cached
       - ../../../CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached
-      - ../../../Dockerfile.ci:/opt/airflow/Dockerfile.ci:cached
       - ../../../LICENSE:/opt/airflow/LICENSE:cached
       - ../../../MANIFEST.in:/opt/airflow/MANIFEST.in:cached
       - ../../../NOTICE:/opt/airflow/NOTICE:cached
diff --git a/scripts/ci/in_container/_in_container_utils.sh b/scripts/ci/in_container/_in_container_utils.sh
index 67cecfd..7e1fd4a 100644
--- a/scripts/ci/in_container/_in_container_utils.sh
+++ b/scripts/ci/in_container/_in_container_utils.sh
@@ -163,34 +163,6 @@ function setup_kerberos() {
     fi
 }
 
-
-function dump_container_logs() {
-    echo "###########################################################################################"
-    echo "                   Dumping logs from all the containers"
-    echo "###########################################################################################"
-    echo "  Docker processes:"
-    echo "###########################################################################################"
-    docker ps --no-trunc
-    echo "###########################################################################################"
-    for CONTAINER in $(docker ps -qa)
-    do
-        CONTAINER_NAME=$(docker inspect --format "{{.Name}}" "${CONTAINER}")
-        echo "-------------------------------------------------------------------------------------------"
-        echo " Docker inspect: ${CONTAINER_NAME}"
-        echo "-------------------------------------------------------------------------------------------"
-        echo
-        docker inspect "${CONTAINER}"
-        echo
-        echo "-------------------------------------------------------------------------------------------"
-        echo " Docker logs: ${CONTAINER_NAME}"
-        echo "-------------------------------------------------------------------------------------------"
-        echo
-        docker logs "${CONTAINER}"
-        echo
-        echo "###########################################################################################"
-    done
-}
-
 function dump_airflow_logs() {
     echo "###########################################################################################"
     echo "                   Dumping logs from all the airflow tasks"
@@ -202,21 +174,6 @@ function dump_airflow_logs() {
 }
 
 
-function send_docker_logs_to_file_io() {
-    echo "##############################################################################"
-    echo
-    echo "   DUMPING LOG FILES FROM CONTAINERS AND SENDING THEM TO file.io"
-    echo
-    echo "##############################################################################"
-    DUMP_FILE=/tmp/$(date "+%Y-%m-%d")_docker_${CI_BUILD_ID:="default"}_${CI_JOB_ID:="default"}.log.gz
-    dump_container_logs 2>&1 | gzip >"${DUMP_FILE}"
-    echo
-    echo "   Logs saved to ${DUMP_FILE}"
-    echo
-    echo "##############################################################################"
-    curl -F "file=@${DUMP_FILE}" https://file.io
-}
-
 function send_airflow_logs_to_file_io() {
     echo "##############################################################################"
     echo
diff --git a/scripts/ci/in_container/check_environment.sh b/scripts/ci/in_container/check_environment.sh
index a7ed4f8..d2ad571 100755
--- a/scripts/ci/in_container/check_environment.sh
+++ b/scripts/ci/in_container/check_environment.sh
@@ -120,61 +120,6 @@ function check_db_connection {
     echo "-----------------------------------------------------------------------------------------------"
 }
 
-function check_mysql_logs {
-    MAX_CHECK=${1:=60}
-    # Wait until mysql is ready!
-    MYSQL_CONTAINER=$(docker ps -q -f "name=mysql" -f "volume=/dev/urandom")
-    if [[ -z ${MYSQL_CONTAINER} ]]; then
-        echo
-        echo "ERROR! MYSQL container is not started. Exiting!"
-        echo
-        exit 1
-    elif [[ "${MYSQL_CONTAINER}" = *$'\n'* ]]; then
-        echo
-        echo "ERROR! pattern match multiple MYSQL containers. Exiting! Container ids as below:"
-        echo
-        echo "${MYSQL_CONTAINER}"
-        echo
-        exit 1
-    fi
-    echo
-    echo "Checking if MySQL is ready for connections (double restarts in the logs)"
-    echo
-    while true
-    do
-        CONNECTION_READY_MESSAGES=$(docker logs "${MYSQL_CONTAINER}" 2>&1 | \
-            grep -c "mysqld: ready for connections" )
-        # MySQL when starting from dockerfile starts a temporary server first because it
-        # starts with an empty database first and it will create the airflow database and then
-        # it will start a second server to serve this newly created database
-        # That's why we should wait until docker logs contain "ready for connections" twice
-        # more info: https://github.com/docker-library/mysql/issues/527
-        if [[ ${CONNECTION_READY_MESSAGES} -gt 1 ]];
-        then
-            echo
-            echo
-            echo "MySQL is ready for connections!"
-            echo
-            break
-        else
-            echo -n "."
-        fi
-        MAX_CHECK=$((MAX_CHECK-1))
-        if [[ ${MAX_CHECK} == 0 ]]; then
-            echo
-            echo "ERROR! Maximum number of retries while waiting for MySQL. Exiting"
-            echo
-            echo "Last check: ${CONNECTION_READY_MESSAGES} connection ready messages (expected >=2)"
-            echo
-            echo "==============================================================================================="
-            echo
-            exit 1
-        else
-            sleep 1
-        fi
-    done
-}
-
 function resetdb_if_requested() {
     if [[ ${DB_RESET:="false"} == "true" ]]; then
         if [[ ${RUN_AIRFLOW_1_10} == "true" ]]; then
@@ -186,23 +131,13 @@ function resetdb_if_requested() {
     return $?
 }
 
-function check_docker_client_server_version_compatibility() {
-    docker version || exit 1
-}
-
-check_docker_client_server_version_compatibility
-
 if [[ -n ${BACKEND:=} ]]; then
     echo "==============================================================================================="
     echo "             Checking backend: ${BACKEND}"
     echo "==============================================================================================="
 
     set +e
-    if [[ ${BACKEND} == "mysql" ]]; then
-        check_mysql_logs 60
-    fi
-
-    check_db_connection 5
+    check_db_connection 20
     set -e
 
     if [[ ${EXIT_CODE} == 0 ]]; then
diff --git a/scripts/ci/in_container/run_ci_tests.sh b/scripts/ci/in_container/run_ci_tests.sh
index f5f4340..fe5a574 100755
--- a/scripts/ci/in_container/run_ci_tests.sh
+++ b/scripts/ci/in_container/run_ci_tests.sh
@@ -37,7 +37,6 @@ if [[ "${RES}" == "0" && ${CI:="false"} == "true" ]]; then
 fi
 
 if [[ ${CI} == "true" ]]; then
-    send_docker_logs_to_file_io
     send_airflow_logs_to_file_io
 fi
 
diff --git a/scripts/ci/in_container/run_system_tests.sh b/scripts/ci/in_container/run_system_tests.sh
index 440532e..11dcf06 100755
--- a/scripts/ci/in_container/run_system_tests.sh
+++ b/scripts/ci/in_container/run_system_tests.sh
@@ -47,7 +47,6 @@ if [[ "${RES}" == "0" && ${CI} == "true" ]]; then
 fi
 
 if [[ ${CI} == "true" ]]; then
-    send_docker_logs_to_file_io
     send_airflow_logs_to_file_io
 fi
 


[airflow] 10/36: Hive/Hadoop minicluster needs JDK8 and JAVA_HOME to work (#8938)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 519401bfe07ded42db5207b99124995037e7ff88
Author: Ash Berlin-Taylor <as...@firemirror.com>
AuthorDate: Thu May 21 06:19:49 2020 +0100

    Hive/Hadoop minicluster needs JDK8 and JAVA_HOME to work (#8938)
    
    Debian Buster only ships with a JDK11, and Hive/Hadoop fails in odd,
    hard to debug ways (complains about metastore not being initalized,
    possibly related to the class loader issues.)
    
    Until we rip Hive out from the CI (replacing it with Hadoop in a seprate
    integration, only on for some builds) we'll have to stick with JRE8
    
    Our previous approach of installing openjdk-8 from Sid/Unstable started
    failing as Debian Sid has a new (and conflicting) version of GCC/libc.
    The adoptopenjdk package archive is designed for Buster so should be
    more resilient
    
    (cherry picked from commit 8476c1e387bb98db3feb9f26247bcef6e92948ec)
---
 Dockerfile.ci | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/Dockerfile.ci b/Dockerfile.ci
index 7a76295..15f3732 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -119,11 +119,14 @@ RUN adduser airflow \
 # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199
 RUN mkdir -pv /usr/share/man/man1 \
     && mkdir -pv /usr/share/man/man7 \
+    && curl -fsSL https://adoptopenjdk.jfrog.io/adoptopenjdk/api/gpg/key/public | apt-key add - \
+    && echo 'deb https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/ buster main' > \
+        /etc/apt/sources.list.d/adoptopenjdk.list \
     && apt-get update \
     && apt-get install --no-install-recommends -y \
       gnupg \
       libgcc-8-dev \
-      default-jre-headless \
+      adoptopenjdk-8-hotspot-jre \
       apt-transport-https \
       bash-completion \
       ca-certificates \
@@ -147,6 +150,8 @@ RUN mkdir -pv /usr/share/man/man1 \
     && apt-get clean \
     && rm -rf /var/lib/apt/lists/*
 
+ENV JAVA_HOME=/usr/lib/jvm/adoptopenjdk-8-hotspot-jre-amd64
+
 # Install Hadoop and Hive
 # It is done in one step to share variables.
 ENV HADOOP_HOME="/opt/hadoop-cdh" HIVE_HOME="/opt/hive"


[airflow] 35/36: Fixes failure of the build scripts when remote repo does not exist (#9188)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit e5162b0f6686e571c520a3e7f73fef8208b3ecf3
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Tue Jun 9 08:47:25 2020 +0200

    Fixes failure of the build scripts when remote repo does not exist (#9188)
    
    
    (cherry picked from commit 337a2dc9bce689061c822f9b1ad172d4128de153)
---
 scripts/ci/libraries/_build_images.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/ci/libraries/_build_images.sh b/scripts/ci/libraries/_build_images.sh
index 123665b..6561184 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -355,7 +355,7 @@ function get_ci_environment() {
                 export CI_SOURCE_REPO="${GITHUB_ACTOR}/${SOURCE_AIRFLOW_REPO}"
                 export CI_SOURCE_BRANCH="${GITHUB_HEAD_REF}"
                 BRANCH_EXISTS=$(git ls-remote --heads \
-                    "https://github.com/${CI_SOURCE_REPO}.git" "${CI_SOURCE_BRANCH}")
+                    "https://github.com/${CI_SOURCE_REPO}.git" "${CI_SOURCE_BRANCH}" || true)
                 if [[ ${BRANCH_EXISTS} == "" ]]; then
                     echo
                     echo "https://github.com/${CI_SOURCE_REPO}.git Branch ${CI_SOURCE_BRANCH} does not exist"


[airflow] 33/36: Split utils sh (#9132)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit a4dfb98d846698261a740aa4c44a32bf08cd630a
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Thu Jun 4 22:12:56 2020 +0200

    Split utils sh (#9132)
    
    * Split _utils.sh into separate files
    
    (cherry picked from commit 92d68807f34bf208d7c75472163d0028874772e7)
---
 .pre-commit-config.yaml                            |    2 +-
 CI.rst                                             |    6 +-
 breeze                                             |   19 +-
 scripts/ci/_all_libs.sh                            |   49 +
 scripts/ci/_script_init.sh                         |    9 +-
 scripts/ci/_utils.sh                               | 2155 --------------------
 scripts/ci/ci_build_dockerhub.sh                   |   33 +-
 scripts/ci/docker-compose/local.yml                |    4 +-
 scripts/ci/in_container/entrypoint_ci.sh           |    7 +-
 scripts/ci/libraries/_build_images.sh              |  771 +++++++
 scripts/ci/libraries/_initialization.sh            |  228 +++
 scripts/ci/libraries/_kind.sh                      |  450 ++++
 scripts/ci/libraries/_local_mounts.sh              |   66 +
 scripts/ci/libraries/_md5sum.sh                    |  137 ++
 scripts/ci/libraries/_parameters.sh                |   58 +
 scripts/ci/libraries/_permissions.sh               |   59 +
 scripts/ci/libraries/_push_pull_remove_images.sh   |  183 ++
 scripts/ci/libraries/_runs.sh                      |   50 +
 scripts/ci/libraries/_sanity_checks.sh             |  166 ++
 scripts/ci/libraries/_spinner.sh                   |   53 +
 scripts/ci/libraries/_start_end.sh                 |   79 +
 .../{_script_init.sh => libraries/_verbosity.sh}   |   37 +-
 scripts/ci/pre_commit_local_yml_mounts.sh          |    4 +-
 tests/bats/bats_utils.bash                         |    4 +-
 24 files changed, 2406 insertions(+), 2223 deletions(-)

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index fe543d1..4979f24 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -218,7 +218,7 @@ repos:
         name: Update mounts in the local yml file
         entry: "./scripts/ci/pre_commit_local_yml_mounts.sh"
         language: system
-        files: ^scripts/ci/_utils.sh$|s^scripts/ci/docker_compose/local.yml"
+        files: ^scripts/ci/libraries/_local_mounts.sh$|s^scripts/ci/docker_compose/local.yml"
         pass_filenames: false
         require_serial: true
       - id: update-extras
diff --git a/CI.rst b/CI.rst
index 6985f06..145a3ef 100644
--- a/CI.rst
+++ b/CI.rst
@@ -34,9 +34,9 @@ the CI jobs and we are mapping all the CI-specific environment variables to gene
 The only two places where CI-specific code might be are:
 
 - CI-specific declaration file (for example it is `<.github/workflow/ci.yml>`_ for GitHub Actions
-- The ``get_ci_environment`` function in `<scripts/ci/_utils.sh>`_ where mapping is performed from
-  the CI-environment specific to generic values. Example for that is CI_EVENT_TYPE variable which
-  determines whether we are running a ``push``. ``schedule`` or ``pull_request`` kind of CI job. For
+- The ``get_ci_environment`` function in `<scripts/ci/libraries/_build_images.sh>`_ where mapping is
+  performed from the CI-environment specific to generic values. Example for that is CI_EVENT_TYPE variable
+  which determines whether we are running a ``push``. ``schedule`` or ``pull_request`` kind of CI job. For
   GitHub Action those are values mapped from ``GITHUB_EVENT_NAME`` variable, for Travis several other
   variables were used to determine it (``TRAVIS_PULL_REQUEST``, ``TRAVIS_EVENT_TYPE``).
 
diff --git a/breeze b/breeze
index 8df8a41..d9cfd92 100755
--- a/breeze
+++ b/breeze
@@ -43,17 +43,17 @@ function setup_default_breeze_variables() {
     export MAX_SCREEN_WIDTH=100
 
     # Directory where all CI scripts are located
-    SCRIPTS_CI_DIR="${AIRFLOW_SOURCES}/scripts/ci"
+    export SCRIPTS_CI_DIR="${AIRFLOW_SOURCES}/scripts/ci"
 
     # Directory where all the build cache is stored - we keep there status of all the docker images
     # As well as hashes of the important files, but also we generate build scripts there that are
     # Used to execute the commands for breeze
-    BUILD_CACHE_DIR="${AIRFLOW_SOURCES}/.build"
+    export BUILD_CACHE_DIR="${AIRFLOW_SOURCES}/.build"
     # This folder is mounted to inside the container in /files folder. This is the way how
     # We can exchange DAGs, scripts, packages etc with the container environment
-    FILES_DIR="${AIRFLOW_SOURCES}/files"
+    export FILES_DIR="${AIRFLOW_SOURCES}/files"
     # Temporary dir used well ... temporarily
-    TMP_DIR="${AIRFLOW_SOURCES}/tmp"
+    export TMP_DIR="${AIRFLOW_SOURCES}/tmp"
 
     # Create those folders above in case they do not exist
     mkdir -pv "${BUILD_CACHE_DIR}"
@@ -62,8 +62,8 @@ function setup_default_breeze_variables() {
 
     # load all the common functions here - those are the functions that are shared between Breeze
     # and CI scripts (CI scripts do not use Breeze as execution environment)
-    # shellcheck source=scripts/ci/_utils.sh
-    . "${SCRIPTS_CI_DIR}/_utils.sh"
+    # shellcheck source=scripts/ci/_all_libs.sh
+    . "${SCRIPTS_CI_DIR}/_all_libs.sh"
 
     # We have different versions of images depending on the python version used. We keep up with the
     # Latest patch-level changes in Python (this is done automatically during CI builds) so we have
@@ -1965,12 +1965,7 @@ function run_breeze_command {
             fi
             AIRFLOW_TESTING_CONTAINER=$("${DC_FILE}" ps | \
                     grep airflow | awk '{print $1}' 2>/dev/null)
-            if [[ -z ${AIRFLOW_TESTING_CONTAINER} ]]; then
-                echo
-                echo "ERROR! Breeze must be running in order to exec into running container"
-                echo
-                exit 1
-            fi
+              : "${AIRFLOW_TESTING_CONTAINER:?"ERROR! Breeze must be running in order to exec into running container"}"
             set -e
             docker exec -it "${AIRFLOW_TESTING_CONTAINER}" \
                 "/opt/airflow/scripts/ci/in_container/entrypoint_exec.sh"
diff --git a/scripts/ci/_all_libs.sh b/scripts/ci/_all_libs.sh
new file mode 100755
index 0000000..9869598
--- /dev/null
+++ b/scripts/ci/_all_libs.sh
@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+SCRIPTS_CI_DIR=$(dirname "${BASH_SOURCE[0]}")
+
+# must be first to initialize arrays TODO: For sure?
+# shellcheck source=scripts/ci/libraries/_initialization.sh
+. "${SCRIPTS_CI_DIR}"/libraries/_initialization.sh
+
+
+# shellcheck source=scripts/ci/libraries/_sanity_checks.sh
+. "${SCRIPTS_CI_DIR}"/libraries/_sanity_checks.sh
+# shellcheck source=scripts/ci/libraries/_build_images.sh
+. "${SCRIPTS_CI_DIR}"/libraries/_build_images.sh
+# shellcheck source=scripts/ci/libraries/_kind.sh
+. "${SCRIPTS_CI_DIR}"/libraries/_kind.sh
+# shellcheck source=scripts/ci/libraries/_local_mounts.sh
+. "${SCRIPTS_CI_DIR}"/libraries/_local_mounts.sh
+# shellcheck source=scripts/ci/libraries/_md5sum.sh
+. "${SCRIPTS_CI_DIR}"/libraries/_md5sum.sh
+# shellcheck source=scripts/ci/libraries/_parameters.sh
+. "${SCRIPTS_CI_DIR}"/libraries/_parameters.sh
+# shellcheck source=scripts/ci/libraries/_permissions.sh
+. "${SCRIPTS_CI_DIR}"/libraries/_permissions.sh
+# shellcheck source=scripts/ci/libraries/_push_pull_remove_images.sh
+. "${SCRIPTS_CI_DIR}"/libraries/_push_pull_remove_images.sh
+# shellcheck source=scripts/ci/libraries/_runs.sh
+. "${SCRIPTS_CI_DIR}"/libraries/_runs.sh
+# shellcheck source=scripts/ci/libraries/_spinner.sh
+. "${SCRIPTS_CI_DIR}"/libraries/_spinner.sh
+# shellcheck source=scripts/ci/libraries/_start_end.sh
+. "${SCRIPTS_CI_DIR}"/libraries/_start_end.sh
+# shellcheck source=scripts/ci/libraries/_verbosity.sh
+. "${SCRIPTS_CI_DIR}"/libraries/_verbosity.sh
diff --git a/scripts/ci/_script_init.sh b/scripts/ci/_script_init.sh
index 514be24..2acbf10 100755
--- a/scripts/ci/_script_init.sh
+++ b/scripts/ci/_script_init.sh
@@ -19,8 +19,13 @@
 set -euo pipefail
 
 # This should only be sourced from CI directory!
-# shellcheck source=scripts/ci/_utils.sh
-. "$( dirname "${BASH_SOURCE[0]}" )/_utils.sh"
+
+SCRIPTS_CI_DIR="$( dirname "${BASH_SOURCE[0]}" )"
+export SCRIPTS_CI_DIR
+
+# shellcheck source=scripts/ci/_all_libs.sh
+. "${SCRIPTS_CI_DIR}"/_all_libs.sh
+
 
 MY_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
 export MY_DIR
diff --git a/scripts/ci/_utils.sh b/scripts/ci/_utils.sh
deleted file mode 100644
index 3bb6ee2..0000000
--- a/scripts/ci/_utils.sh
+++ /dev/null
@@ -1,2155 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#   http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied.  See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-# Assume all the scripts are sourcing the _utils.sh from the scripts/ci directory
-# and MY_DIR variable is set to this directory. It can be overridden however
-
-# extra flags passed to the docker run for CI image commands (as Bash array)
-declare -a EXTRA_DOCKER_FLAGS
-# extra flags passed to the docker run for PROD image commands(as Bash array)
-declare -a EXTRA_DOCKER_PROD_BUILD_FLAGS
-export EXTRA_DOCKER_FLAGS
-export EXTRA_DOCKER_PROD_BUILD_FLAGS
-
-
-# In case "VERBOSE_COMMANDS" is set to "true" set -x is used to enable debugging
-function check_verbose_setup {
-    if [[ ${VERBOSE_COMMANDS:="false"} == "true" ]]; then
-        set -x
-    else
-        set +x
-    fi
-}
-
-
-# In case "VERBOSE" is set to "true" (--verbose flag in Breeze) all docker commands run will be
-# printed before execution
-function verbose_docker {
-    if [[ ${VERBOSE:="false"} == "true" && ${VERBOSE_COMMANDS:=} != "true" ]]; then
-       # do not print echo if VERSBOSE_COMMAND is set (set -x does it already)
-        echo "docker" "${@}"
-    fi
-    docker "${@}"
-}
-
-# Common environment that is initialized by both Breeze and CI scripts
-function initialize_common_environment {
-    # default python Major/Minor version
-    PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:="3.5"}
-
-    FILES_TO_CLEANUP_ON_EXIT=()
-
-    # Sets to where airflow sources are located
-    AIRFLOW_SOURCES=${AIRFLOW_SOURCES:=$(cd "${MY_DIR}/../../" && pwd)}
-    export AIRFLOW_SOURCES
-
-    # Sets to the build cache directory - status of build and convenience scripts are stored there
-    BUILD_CACHE_DIR="${AIRFLOW_SOURCES}/.build"
-    export BUILD_CACHE_DIR
-
-    # File to keep the last forced answer. This is useful for pre-commits where you need to
-    # only answer once if the image should be rebuilt or not and your answer is used for
-    # All the subsequent questions
-    LAST_FORCE_ANSWER_FILE="${BUILD_CACHE_DIR}/last_force_answer.sh"
-
-    # Create useful directories if not yet created
-    mkdir -p "${AIRFLOW_SOURCES}/.mypy_cache"
-    mkdir -p "${AIRFLOW_SOURCES}/logs"
-    mkdir -p "${AIRFLOW_SOURCES}/tmp"
-    mkdir -p "${AIRFLOW_SOURCES}/files"
-    mkdir -p "${AIRFLOW_SOURCES}/dist"
-
-    # Read common values used across Breeze and CI scripts
-    # shellcheck source=common/_common_values.sh
-    . "${AIRFLOW_SOURCES}/common/_common_values.sh"
-    # Read image-specific values used across Breeze and CI scripts
-    # shellcheck source=common/_image_variables.sh
-    . "${AIRFLOW_SOURCES}/common/_image_variables.sh"
-    # Read information about files that are checked if image should be rebuilt
-    # shellcheck source=common/_files_for_rebuild_check.sh
-    . "${AIRFLOW_SOURCES}/common/_files_for_rebuild_check.sh"
-
-    # Default branch name for triggered builds is the one configured in default branch
-    export BRANCH_NAME=${BRANCH_NAME:=${DEFAULT_BRANCH}}
-
-    export GITHUB_ORGANISATION=${GITHUB_ORGANISATION:="apache"}
-    export GITHUB_REPO=${GITHUB_REPO:="airflow"}
-    export CACHE_REGISTRY=${CACHE_REGISTRY:="docker.pkg.github.com"}
-    export ENABLE_REGISTRY_CACHE=${ENABLE_REGISTRY_CACHE:="false"}
-
-    # Default port numbers for forwarded ports
-    export WEBSERVER_HOST_PORT=${WEBSERVER_HOST_PORT:="28080"}
-    export POSTGRES_HOST_PORT=${POSTGRES_HOST_PORT:="25433"}
-    export MYSQL_HOST_PORT=${MYSQL_HOST_PORT:="23306"}
-
-    # Default MySQL/Postgres versions
-    export POSTGRES_VERSION=${POSTGRES_VERSION:="9.6"}
-    export MYSQL_VERSION=${MYSQL_VERSION:="5.6"}
-
-    # Whether base python images should be pulled from cache
-    export PULL_PYTHON_BASE_IMAGES_FROM_CACHE=${PULL_PYTHON_BASE_IMAGES_FROM_CACHE:="true"}
-
-    # Disable writing .pyc files - slightly slower imports but not messing around when switching
-    # Python version and avoids problems with root-owned .pyc files in host
-    export PYTHONDONTWRITEBYTECODE=${PYTHONDONTWRITEBYTECODE:="true"}
-
-    # By default we assume the kubernetes cluster is not being started
-    export ENABLE_KIND_CLUSTER=${ENABLE_KIND_CLUSTER:="false"}
-    #
-    # Sets mounting of host volumes to container for static checks
-    # unless MOUNT_HOST_AIRFLOW_VOLUME is not true
-    #
-    MOUNT_HOST_AIRFLOW_VOLUME=${MOUNT_HOST_AIRFLOW_VOLUME:="true"}
-    export MOUNT_HOST_AIRFLOW_VOLUME
-
-    # If this variable is set, we mount the whole sources directory to the host rather than
-    # selected volumes. This is needed to check ALL source files during licence check
-    # for example
-    MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS=${MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS="false"}
-    export MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS
-
-    # Set host user id to current user. This is used to set the ownership properly when exiting
-    # The container on Linux - all files created inside docker are created with root user
-    # but they should be restored back to the host user
-    HOST_USER_ID="$(id -ur)"
-    export HOST_USER_ID
-
-    # Set host group id to current group This is used to set the ownership properly when exiting
-    # The container on Linux - all files created inside docker are created with root user
-    # but they should be restored back to the host user
-    HOST_GROUP_ID="$(id -gr)"
-    export HOST_GROUP_ID
-
-    # Add the right volume mount for sources, depending which mount strategy is used
-    if [[ ${MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS} == "true" ]]; then
-        print_info
-        print_info "Mount whole airflow source directory for static checks"
-        print_info
-        EXTRA_DOCKER_FLAGS=( \
-          "-v" "${AIRFLOW_SOURCES}:/opt/airflow" \
-          "--env" "PYTHONDONTWRITEBYTECODE" \
-        )
-    elif [[ ${MOUNT_HOST_AIRFLOW_VOLUME} == "true" ]]; then
-        print_info
-        print_info "Mounting necessary host volumes to Docker"
-        print_info
-
-        read -r -a EXTRA_DOCKER_FLAGS <<< "$(convert_local_mounts_to_docker_params)"
-    else
-        print_info
-        print_info "Skip mounting host volumes to Docker"
-        print_info
-        EXTRA_DOCKER_FLAGS=( \
-            "--env" "PYTHONDONTWRITEBYTECODE" \
-        )
-    fi
-
-    # In case of the CI build get environment variables from codecov.io and
-    # set it as the extra docker flags. As described in https://docs.codecov.io/docs/testing-with-docker
-    if [[ ${CI:=} == "true" ]]; then
-        CI_CODECOV_ENV="$(bash <(curl -s https://codecov.io/env))"
-        for ENV_PARAM in ${CI_CODECOV_ENV}
-        do
-            EXTRA_DOCKER_FLAGS+=("${ENV_PARAM}")
-        done
-    fi
-    EXTRA_DOCKER_PROD_BUILD_FLAGS=()
-
-    # We use pulled docker image cache by default to speed up the builds
-    # but we can also set different docker caching strategy (for example we can use local cache
-    # to build the images in case we iterate with the image
-    export DOCKER_CACHE=${DOCKER_CACHE:="pulled"}
-
-    # By default we are not upgrading to latest requirements when building Docker CI image
-    # This will only be done in cron jobs
-    export UPGRADE_TO_LATEST_REQUIREMENTS=${UPGRADE_TO_LATEST_REQUIREMENTS:="false"}
-
-    # In case of MacOS we need to use gstat - gnu version of the stats
-    STAT_BIN=stat
-    if [[ "${OSTYPE}" == "darwin"* ]]; then
-        STAT_BIN=gstat
-    fi
-
-    # Read airflow version from the version.py
-    AIRFLOW_VERSION=$(grep version "airflow/version.py" | awk '{print $3}' | sed "s/['+]//g")
-    export AIRFLOW_VERSION
-
-    # default version of python used to tag the "master" and "latest" images in DockerHub
-    export DEFAULT_PYTHON_MAJOR_MINOR_VERSION=3.6
-
-    # In case we are not in CI - we assume we run locally. There are subtle changes if you run
-    # CI scripts locally - for example requirements are eagerly updated if you do local run
-    # in generate requirements
-    if [[ ${CI:="false"} == "true" ]]; then
-        export LOCAL_RUN="false"
-    else
-        export LOCAL_RUN="true"
-    fi
-
-    # eager upgrade while generating requirements should only happen in locally run
-    # pre-commits or in cron job
-    if [[ ${LOCAL_RUN} == "true" ]]; then
-        export UPGRADE_WHILE_GENERATING_REQUIREMENTS="true"
-    else
-        export UPGRADE_WHILE_GENERATING_REQUIREMENTS=${UPGRADE_WHILE_GENERATING_REQUIREMENTS:="false"}
-    fi
-
-    # Default extras used for building CI image
-    export DEFAULT_CI_EXTRAS="devel_ci"
-
-    # Default extras used for building Production image. The master of this information is in the Dockerfile
-    DEFAULT_PROD_EXTRAS=$(grep "ARG AIRFLOW_EXTRAS=" "${AIRFLOW_SOURCES}/Dockerfile"|
-            awk 'BEGIN { FS="=" } { print $2 }' | tr -d '"')
-
-    # By default we build CI images  but when we specify `--production-image` we switch to production image
-    export PRODUCTION_IMAGE="false"
-
-    # The SQLlite URL used for sqlite runs
-    export SQLITE_URL="sqlite:////root/airflow/airflow.db"
-
-    # Determines if airflow should be installed from a specified reference in GitHub
-    export INSTALL_AIRFLOW_REFERENCE=""
-
-    # Version of Kubernetes to run
-    export KUBERNETES_VERSION="${KUBERNETES_VERSION:="v1.15.3"}"
-
-    # Name of the KinD cluster to connect to
-    export KIND_CLUSTER_NAME=${KIND_CLUSTER_NAME:="airflow-python-${PYTHON_MAJOR_MINOR_VERSION}-${KUBERNETES_VERSION}"}
-
-    # Name of the KinD cluster to connect to when referred to via kubectl
-    export KUBECTL_CLUSTER_NAME=kind-${KIND_CLUSTER_NAME}
-
-}
-
-# Prints verbose information in case VERBOSE variable is set
-function print_info() {
-    if [[ ${VERBOSE:="false"} == "true" ]]; then
-        echo "$@"
-    fi
-}
-
-# Those are files that are mounted locally when mounting local sources is requested
-# By default not the whole airflow sources directory is mounted because there are often
-# artifacts created there (for example .egg-info files) that are breaking the capability
-# of running different python versions in Breeze. So we only mount what is needed by default.
-function generate_local_mounts_list {
-    local prefix="$1"
-    LOCAL_MOUNTS=(
-        "$prefix".bash_aliases:/root/.bash_aliases:cached
-        "$prefix".bash_history:/root/.bash_history:cached
-        "$prefix".coveragerc:/opt/airflow/.coveragerc:cached
-        "$prefix".dockerignore:/opt/airflow/.dockerignore:cached
-        "$prefix".flake8:/opt/airflow/.flake8:cached
-        "$prefix".github:/opt/airflow/.github:cached
-        "$prefix".inputrc:/root/.inputrc:cached
-        "$prefix".rat-excludes:/opt/airflow/.rat-excludes:cached
-        "$prefix"CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached
-        "$prefix"LICENSE:/opt/airflow/LICENSE:cached
-        "$prefix"MANIFEST.in:/opt/airflow/MANIFEST.in:cached
-        "$prefix"NOTICE:/opt/airflow/NOTICE:cached
-        "$prefix"airflow:/opt/airflow/airflow:cached
-        "$prefix"common:/opt/airflow/common:cached
-        "$prefix"dags:/opt/airflow/dags:cached
-        "$prefix"dev:/opt/airflow/dev:cached
-        "$prefix"docs:/opt/airflow/docs:cached
-        "$prefix"files:/files:cached
-        "$prefix"dist:/dist:cached
-        "$prefix"hooks:/opt/airflow/hooks:cached
-        "$prefix"logs:/root/airflow/logs:cached
-        "$prefix"pytest.ini:/opt/airflow/pytest.ini:cached
-        "$prefix"requirements:/opt/airflow/requirements:cached
-        "$prefix"scripts:/opt/airflow/scripts:cached
-        "$prefix"scripts/ci/in_container/entrypoint_ci.sh:/entrypoint_ci.sh:cached
-        "$prefix"setup.cfg:/opt/airflow/setup.cfg:cached
-        "$prefix"setup.py:/opt/airflow/setup.py:cached
-        "$prefix"tests:/opt/airflow/tests:cached
-        "$prefix"kubernetes_tests:/opt/airflow/kubernetes_tests:cached
-        "$prefix"tmp:/opt/airflow/tmp:cached
-    )
-}
-
-# Converts the local mounts that we defined above to the right set of -v
-# volume mappings in docker-compose file. This is needed so that we only
-# maintain the volumes in one place (above)
-function convert_local_mounts_to_docker_params() {
-    generate_local_mounts_list "${AIRFLOW_SOURCES}/"
-    # Bash can't "return" arrays, so we need to quote any special characters
-    printf -- '-v %q ' "${LOCAL_MOUNTS[@]}"
-}
-
-# Fixes a file that is expected to be a file. If - for whatever reason - the local file is not created
-# When mounting it to container, docker assumes it is a missing directory and creates it. Such mistakenly
-# Created directories should be removed and replaced with files
-function sanitize_file() {
-    if [[ -d "${1}" ]]; then
-        rm -rf "${1}"
-    fi
-    touch "${1}"
-
-}
-
-# Those files are mounted into container when run locally
-# .bash_history is preserved and you can modify .bash_aliases and .inputrc
-# according to your liking
-function sanitize_mounted_files() {
-    sanitize_file "${AIRFLOW_SOURCES}/.bash_history"
-    sanitize_file "${AIRFLOW_SOURCES}/.bash_aliases"
-    sanitize_file "${AIRFLOW_SOURCES}/.inputrc"
-
-    # When KinD cluster is created, the folder keeps authentication information
-    # across sessions
-    mkdir -p "${AIRFLOW_SOURCES}/.kube" >/dev/null 2>&1
-}
-
-#
-# Creates cache directory where we will keep temporary files needed for the docker build
-#
-# This directory will be automatically deleted when the script is killed or exists (via trap)
-# Unless SKIP_CACHE_DELETION variable is set. You can set this variable and then see
-# the output/files generated by the scripts in this directory.
-#
-# Most useful is out.log file in this directory storing verbose output of the scripts.
-#
-function create_cache_directory() {
-    CACHE_TMP_FILE_DIR=$(mktemp -d)
-    export CACHE_TMP_FILE_DIR
-
-    if [[ ${SKIP_CACHE_DELETION:=} != "true" ]]; then
-        trap 'rm -rf -- "${CACHE_TMP_FILE_DIR}"' INT TERM HUP
-    fi
-
-    OUTPUT_LOG="${CACHE_TMP_FILE_DIR}/out.log"
-    export OUTPUT_LOG
-}
-
-# Removes the cache temporary directory
-function remove_cache_directory() {
-    if [[ -d ${CACHE_TMP_FILE_DIR} ]]; then
-        rm -rf "${CACHE_TMP_FILE_DIR}"
-    fi
-}
-
-#
-# Verifies if stored md5sum of the file changed since the last tme ot was checked
-# The md5sum files are stored in .build directory - you can delete this directory
-# If you want to rebuild everything from the scratch
-#
-function calculate_file_md5sum {
-    local FILE="${1}"
-    local MD5SUM
-    local MD5SUM_CACHE_DIR="${BUILD_CACHE_DIR}/${DEFAULT_BRANCH}/${PYTHON_MAJOR_MINOR_VERSION}/${THE_IMAGE_TYPE}"
-    mkdir -pv "${MD5SUM_CACHE_DIR}"
-    MD5SUM=$(md5sum "${FILE}")
-    local MD5SUM_FILE
-    MD5SUM_FILE="${MD5SUM_CACHE_DIR}"/$(basename "${FILE}").md5sum
-    local MD5SUM_FILE_NEW
-    MD5SUM_FILE_NEW=${CACHE_TMP_FILE_DIR}/$(basename "${FILE}").md5sum.new
-    echo "${MD5SUM}" > "${MD5SUM_FILE_NEW}"
-    local RET_CODE=0
-    if [[ ! -f "${MD5SUM_FILE}" ]]; then
-        print_info "Missing md5sum for ${FILE#${AIRFLOW_SOURCES}} (${MD5SUM_FILE#${AIRFLOW_SOURCES}})"
-        RET_CODE=1
-    else
-        diff "${MD5SUM_FILE_NEW}" "${MD5SUM_FILE}" >/dev/null
-        RES=$?
-        if [[ "${RES}" != "0" ]]; then
-            print_info "The md5sum changed for ${FILE}"
-            RET_CODE=1
-        fi
-    fi
-    return ${RET_CODE}
-}
-
-#
-# Moves md5sum file from it's temporary location in CACHE_TMP_FILE_DIR to
-# BUILD_CACHE_DIR - thus updating stored MD5 sum fo the file
-#
-function move_file_md5sum {
-    local FILE="${1}"
-    local MD5SUM_FILE
-    local MD5SUM_CACHE_DIR="${BUILD_CACHE_DIR}/${DEFAULT_BRANCH}/${PYTHON_MAJOR_MINOR_VERSION}/${THE_IMAGE_TYPE}"
-    mkdir -pv "${MD5SUM_CACHE_DIR}"
-    MD5SUM_FILE="${MD5SUM_CACHE_DIR}"/$(basename "${FILE}").md5sum
-    local MD5SUM_FILE_NEW
-    MD5SUM_FILE_NEW=${CACHE_TMP_FILE_DIR}/$(basename "${FILE}").md5sum.new
-    if [[ -f "${MD5SUM_FILE_NEW}" ]]; then
-        mv "${MD5SUM_FILE_NEW}" "${MD5SUM_FILE}"
-        print_info "Updated md5sum file ${MD5SUM_FILE} for ${FILE}."
-    fi
-}
-
-#
-# Stores md5sum files for all important files and
-# records that we built the images locally so that next time we use
-# it from the local docker cache rather than pull (unless forced)
-#
-function update_all_md5_files() {
-    print_info
-    print_info "Updating md5sum files"
-    print_info
-    for FILE in "${FILES_FOR_REBUILD_CHECK[@]}"
-    do
-        move_file_md5sum "${AIRFLOW_SOURCES}/${FILE}"
-    done
-    mkdir -pv "${BUILD_CACHE_DIR}/${BRANCH_NAME}"
-    touch "${BUILT_IMAGE_FLAG_FILE}"
-}
-
-
-function calculate_md5sum_for_all_files() {
-    FILES_MODIFIED="false"
-    set +e
-    for FILE in "${FILES_FOR_REBUILD_CHECK[@]}"
-    do
-        if ! calculate_file_md5sum "${AIRFLOW_SOURCES}/${FILE}"; then
-            FILES_MODIFIED="true"
-        fi
-    done
-    set +e
-}
-
-#
-# Checks md5sum of all important files in order to optimise speed of running various operations
-# That mount sources of Airflow to container and require docker image built with latest dependencies.
-# the Docker image will only be marked for rebuilding only in case any of the important files change:
-# * setup.py
-# * setup.cfg
-# * Dockerfile.ci
-# * airflow/version.py
-#
-# This is needed because we want to skip rebuilding of the image when only airflow sources change but
-# Trigger rebuild in case we need to change dependencies (setup.py, setup.cfg, change version of Airflow
-# or the Dockerfile.ci itself changes.
-#
-# Another reason to skip rebuilding Docker is thar currently it takes a bit longer time than simple Docker
-# We need to fix group permissions of files in Docker because different linux build services have
-# different default umask and Docker uses group permissions in checking for cache invalidation.
-#
-# As result of this check - most of the static checks will start pretty much immediately.
-#
-function check_if_docker_build_is_needed() {
-    print_info
-    print_info "Checking if image build is needed for ${THE_IMAGE_TYPE} image."
-    print_info
-    if [[ ${FORCE_BUILD_IMAGES:=""} == "true" ]]; then
-        echo "Docker image build is forced for ${THE_IMAGE_TYPE} image"
-        calculate_md5sum_for_all_files
-        export NEEDS_DOCKER_BUILD="true"
-    else
-        calculate_md5sum_for_all_files
-        if [[ ${FILES_MODIFIED} == "true" ]]; then
-            export NEEDS_DOCKER_BUILD="true"
-        fi
-        if [[ ${NEEDS_DOCKER_BUILD} == "true" ]]; then
-            echo "Docker image build is needed for ${THE_IMAGE_TYPE} image!"
-        else
-            print_info "Docker image build is not needed for ${THE_IMAGE_TYPE} image!"
-        fi
-    fi
-    print_info
-}
-
-#
-# Checks if core utils required in the host system are installed and explain what needs to be done if not
-#
-function check_if_coreutils_installed() {
-    set +e
-    getopt -T >/dev/null
-    GETOPT_RETVAL=$?
-
-    if [[ $(uname -s) == 'Darwin' ]] ; then
-        command -v gstat >/dev/null
-        STAT_PRESENT=$?
-    else
-        command -v stat >/dev/null
-        STAT_PRESENT=$?
-    fi
-
-    command -v md5sum >/dev/null
-    MD5SUM_PRESENT=$?
-
-    set -e
-
-    CMDNAME="$(basename -- "$0")"
-
-    ####################  Parsing options/arguments
-    if [[ ${GETOPT_RETVAL} != 4 || "${STAT_PRESENT}" != "0" || "${MD5SUM_PRESENT}" != "0" ]]; then
-        print_info
-        if [[ $(uname -s) == 'Darwin' ]] ; then
-            echo >&2 "You are running ${CMDNAME} in OSX environment"
-            echo >&2 "And you need to install gnu commands"
-            echo >&2
-            echo >&2 "Run 'brew install gnu-getopt coreutils'"
-            echo >&2
-            echo >&2 "Then link the gnu-getopt to become default as suggested by brew."
-            echo >&2
-            echo >&2 "If you use bash, you should run this command:"
-            echo >&2
-            echo >&2 "echo 'export PATH=\"/usr/local/opt/gnu-getopt/bin:\$PATH\"' >> ~/.bash_profile"
-            echo >&2 ". ~/.bash_profile"
-            echo >&2
-            echo >&2 "If you use zsh, you should run this command:"
-            echo >&2
-            echo >&2 "echo 'export PATH=\"/usr/local/opt/gnu-getopt/bin:\$PATH\"' >> ~/.zprofile"
-            echo >&2 ". ~/.zprofile"
-            echo >&2
-            echo >&2 "Login and logout afterwards !!"
-            echo >&2
-            echo >&2 "After re-login, your PATH variable should start with \"/usr/local/opt/gnu-getopt/bin\""
-            echo >&2 "Your current path is ${PATH}"
-            echo >&2
-        else
-            echo >&2 "You do not have necessary tools in your path (getopt, stat, md5sum)."
-            echo >&2 "Please install latest/GNU version of getopt and coreutils."
-            echo >&2 "This can usually be done with 'apt install util-linux coreutils'"
-        fi
-        print_info
-        exit 1
-    fi
-}
-
-#
-# Asserts that we are not inside of the container
-#
-function assert_not_in_container() {
-    if [[ ${SKIP_IN_CONTAINER_CHECK:=} == "true" ]]; then
-        return
-    fi
-    if [[ -f /.dockerenv ]]; then
-        echo >&2
-        echo >&2 "You are inside the Airflow docker container!"
-        echo >&2 "You should only run this script from the host."
-        echo >&2 "Learn more about how we develop and test airflow in:"
-        echo >&2 "https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst"
-        echo >&2
-        exit 1
-    fi
-}
-
-# Removes the "Forced answer" (yes/no/quit) given previously, unless you specifically want to remember it.
-#
-# This is the default behaviour of all rebuild scripts to ask independently whether you want to
-# rebuild the image or not. Sometimes however we want to remember answer previously given. For
-# example if you answered "no" to rebuild the image, the assumption is that you do not
-# want to rebuild image also for other rebuilds in the same pre-commit execution.
-#
-# All the pre-commit checks therefore have `export REMEMBER_LAST_ANSWER="true"` set
-# So that in case they are run in a sequence of commits they will not rebuild. Similarly if your most
-# recent answer was "no" and you run `pre-commit run mypy` (for example) it will also reuse the
-# "no" answer given previously. This happens until you run any of the breeze commands or run all
-# precommits `pre-commit run` - then the "LAST_FORCE_ANSWER_FILE" will be removed and you will
-# be asked again.
-function forget_last_answer() {
-    if [[ ${REMEMBER_LAST_ANSWER:="false"} != "true" ]]; then
-        print_info
-        print_info "Forgetting last answer from ${LAST_FORCE_ANSWER_FILE}:"
-        print_info
-        rm -f "${LAST_FORCE_ANSWER_FILE}"
-    else
-        if [[ -f "${LAST_FORCE_ANSWER_FILE}" ]]; then
-            print_info
-            print_info "Still remember last answer from ${LAST_FORCE_ANSWER_FILE}:"
-            print_info "$(cat "${LAST_FORCE_ANSWER_FILE}")"
-            print_info
-        fi
-    fi
-}
-
-# Confirms if hte image should be rebuild and interactively checks it with the user.
-# In case iit needs to be rebuild. It only ask the user if it determines that the rebuild
-# is needed and that the rebuild is not already forced. It asks the user using available terminals
-# So that the script works also from within pre-commit run via git hooks - where stdin is not
-# available - it tries to find usable terminal and ask the user via this terminal.
-function confirm_image_rebuild() {
-    ACTION="rebuild"
-    if [[ ${FORCE_PULL_IMAGES:=} == "true" ]]; then
-        ACTION="pull and rebuild"
-    fi
-    if [[ -f "${LAST_FORCE_ANSWER_FILE}" ]]; then
-        # set variable from last answered response given in the same pre-commit run - so that it can be
-        # answered in teh first pre-commit check (build) and then used in another (pylint/mypy/flake8 etc).
-        # shellcheck disable=SC1090
-        source "${LAST_FORCE_ANSWER_FILE}"
-    fi
-    set +e
-    local RES
-    if [[ ${CI:="false"} == "true" ]]; then
-        print_info
-        print_info "CI environment - forcing rebuild for image ${THE_IMAGE_TYPE}."
-        print_info
-        RES="0"
-    elif [[ -n "${FORCE_ANSWER_TO_QUESTIONS:=""}" ]]; then
-        print_info
-        print_info "Forcing answer '${FORCE_ANSWER_TO_QUESTIONS}'"
-        print_info
-        case "${FORCE_ANSWER_TO_QUESTIONS}" in
-            [yY][eE][sS]|[yY])
-                RES="0" ;;
-            [qQ][uU][iI][tT]|[qQ])
-                RES="2" ;;
-            *)
-                RES="1" ;;
-        esac
-    elif [[ -t 0 ]]; then
-        # Check if this script is run interactively with stdin open and terminal attached
-        "${AIRFLOW_SOURCES}/confirm" "${ACTION} image ${THE_IMAGE_TYPE}-python${PYTHON_MAJOR_MINOR_VERSION}"
-        RES=$?
-    elif [[ ${DETECTED_TERMINAL:=$(tty)} != "not a tty" ]]; then
-        # Make sure to use output of tty rather than stdin/stdout when available - this way confirm
-        # will works also in case of pre-commits (git does not pass stdin/stdout to pre-commit hooks)
-        # shellcheck disable=SC2094
-        "${AIRFLOW_SOURCES}/confirm" "${ACTION} image ${THE_IMAGE_TYPE}-python${PYTHON_MAJOR_MINOR_VERSION}" \
-            <"${DETECTED_TERMINAL}" >"${DETECTED_TERMINAL}"
-        RES=$?
-        export DETECTED_TERMINAL
-    elif [[ -c /dev/tty ]]; then
-        export DETECTED_TERMINAL=/dev/tty
-        # Make sure to use /dev/tty first rather than stdin/stdout when available - this way confirm
-        # will works also in case of pre-commits (git does not pass stdin/stdout to pre-commit hooks)
-        # shellcheck disable=SC2094
-        "${AIRFLOW_SOURCES}/confirm" "${ACTION} image ${THE_IMAGE_TYPE}-python${PYTHON_MAJOR_MINOR_VERSION}" \
-            <"${DETECTED_TERMINAL}" >"${DETECTED_TERMINAL}"
-        RES=$?
-    else
-        print_info
-        print_info "No terminal, no stdin - quitting"
-        print_info
-        # No terminal, no stdin, no force answer - quitting!
-        RES="2"
-    fi
-    set -e
-    if [[ ${RES} == "1" ]]; then
-        print_info
-        print_info "Skipping rebuilding the image ${THE_IMAGE_TYPE}-python${PYTHON_MAJOR_MINOR_VERSION}"
-        print_info
-        SKIP_REBUILD="true"
-        # Force "no" also to subsequent questions so that if you answer it once, you are not asked
-        # For all other pre-commits and you will continue using the images you already have
-        export FORCE_ANSWER_TO_QUESTIONS="no"
-        echo 'export FORCE_ANSWER_TO_QUESTIONS="no"' > "${LAST_FORCE_ANSWER_FILE}"
-    elif [[ ${RES} == "2" ]]; then
-        echo >&2
-        echo >&2 "ERROR: The ${THE_IMAGE_TYPE} needs to be rebuilt - it is outdated. "
-        echo >&2 "   Make sure you build the images bu running"
-        echo >&2
-        echo >&2 "      ./breeze --python ${PYTHON_MAJOR_MINOR_VERSION}" build-image
-        echo >&2
-        echo >&2 "   If you run it via pre-commit as individual hook, you can run 'pre-commit run build'."
-        echo >&2
-        exit 1
-    else
-        # Force "yes" also to subsequent questions
-        export FORCE_ANSWER_TO_QUESTIONS="yes"
-    fi
-}
-
-# Builds local image manifest
-# It contains only one .json file - result of docker inspect - describing the image
-# We cannot use docker registry APIs as they are available only with authorisation
-# But this image can be pulled without authentication
-function build_ci_image_manifest() {
-    verbose_docker inspect "${AIRFLOW_CI_IMAGE}" > "manifests/${AIRFLOW_CI_BASE_TAG}.json"
-    verbose_docker build \
-    --build-arg AIRFLOW_CI_BASE_TAG="${AIRFLOW_CI_BASE_TAG}" \
-    --tag="${AIRFLOW_CI_LOCAL_MANIFEST_IMAGE}" \
-    -f- . <<EOF
-ARG AIRFLOW_CI_BASE_TAG
-FROM scratch
-
-COPY "manifests/${AIRFLOW_CI_BASE_TAG}.json" .
-
-CMD ""
-EOF
-}
-
-#
-# Retrieves information about layers in the local IMAGE
-# it stores list of SHAs of image layers in the file pointed at by TMP_MANIFEST_LOCAL_SHA
-#
-function get_local_image_info() {
-    TMP_MANIFEST_LOCAL_JSON=$(mktemp)
-    TMP_MANIFEST_LOCAL_SHA=$(mktemp)
-    set +e
-    # Remove the container just in case
-    verbose_docker rm --force "local-airflow-manifest"  >/dev/null 2>&1
-    # Create manifest from the local manifest image
-    if ! verbose_docker create --name "local-airflow-manifest" \
-        "${AIRFLOW_CI_LOCAL_MANIFEST_IMAGE}"  >/dev/null 2>&1 ; then
-        echo
-        echo "Local manifest image not available"
-        echo
-        LOCAL_MANIFEST_IMAGE_UNAVAILABLE="true"
-        return
-    fi
-    set -e
-     # Create manifest from the local manifest image
-    verbose_docker cp "local-airflow-manifest:${AIRFLOW_CI_BASE_TAG}.json" "${TMP_MANIFEST_LOCAL_JSON}" >/dev/null 2>&1
-    sed 's/ *//g' "${TMP_MANIFEST_LOCAL_JSON}" | grep '^"sha256:' >"${TMP_MANIFEST_LOCAL_SHA}"
-    verbose_docker rm --force "local-airflow-manifest" >/dev/null 2>&1
-}
-
-#
-# Retrieves information about layers in the remote IMAGE
-# it stores list of SHAs of image layers in the file pointed at by TMP_MANIFEST_REMOTE_SHA
-# This cannot be done easily with existing APIs of Dockerhub because they require additional authentication
-# even for public images. Therefore instead we are downloading a specially prepared manifest image
-# which is built together with the main image. This special manifest image is prepared during
-# building of the main image and contains single JSON file being result of docker inspect on that image
-# This image is from scratch so it is very tiny
-function get_remote_image_info() {
-    set +e
-    # Pull remote manifest image
-    if ! verbose_docker pull "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}" >/dev/null; then
-        echo
-        echo "Remote docker registry unreachable"
-        echo
-        REMOTE_DOCKER_REGISTRY_UNREACHABLE="true"
-        return
-    fi
-    set -e
-
-    # Docker needs the file passed to --cidfile to not exist, so we can't use mktemp
-    TMP_CONTAINER_ID="remote-airflow-manifest-$$.container_id"
-    FILES_TO_CLEANUP_ON_EXIT+=("$TMP_CONTAINER_ID")
-
-    TMP_MANIFEST_REMOTE_JSON=$(mktemp)
-    TMP_MANIFEST_REMOTE_SHA=$(mktemp)
-    # Create container out of the manifest image without running it
-    verbose_docker create --cidfile "${TMP_CONTAINER_ID}" "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}"
-    # Extract manifest and store it in local file
-    verbose_docker cp "$(cat "${TMP_CONTAINER_ID}"):${AIRFLOW_CI_BASE_TAG}.json" "${TMP_MANIFEST_REMOTE_JSON}"
-    # Filter everything except SHAs of image layers
-    sed 's/ *//g' "${TMP_MANIFEST_REMOTE_JSON}" | grep '^"sha256:' >"${TMP_MANIFEST_REMOTE_SHA}"
-    verbose_docker rm --force "$( cat "${TMP_CONTAINER_ID}")"
-}
-
-# The Number determines the cut-off between local building time and pull + build time.
-# It is a bit experimental and it will have to be kept
-# updated as we keep on changing layers. The cut-off point is at the moment when we do first
-# pip install "https://github.com/apache/airflow/archive/${AIRFLOW_BRANCH}.tar...
-# you can get it via this command:
-# docker history --no-trunc  apache/airflow:master-python3.6-ci | \
-#      grep ^sha256 | grep -n "pip uninstall" | awk 'BEGIN {FS=":"} {print $1 }'
-#
-# This command returns the number of layer in docker history where pip uninstall is called. This is the
-# line that will take a lot of time to run and at this point it's worth to pull the image from repo
-# if there are at least NN changed layers in your docker file, you should pull the image.
-#
-# Note that this only matters if you have any of the important files changed since the last build
-# of your image such as Dockerfile.ci, setup.py etc.
-#
-MAGIC_CUT_OFF_NUMBER_OF_LAYERS=41
-
-# Compares layers from both remote and local image and set FORCE_PULL_IMAGES to true in case
-# More than the last NN layers are different.
-function compare_layers() {
-    NUM_DIFF=$(diff  -y --suppress-common-lines "${TMP_MANIFEST_REMOTE_SHA}" "${TMP_MANIFEST_LOCAL_SHA}" | \
-        wc -l || true)
-    rm -f "${TMP_MANIFEST_REMOTE_JSON}" "${TMP_MANIFEST_REMOTE_SHA}" "${TMP_MANIFEST_LOCAL_JSON}" "${TMP_MANIFEST_LOCAL_SHA}"
-    echo
-    echo "Number of layers different between the local and remote image: ${NUM_DIFF}"
-    echo
-    # This is where setup py is rebuilt - it will usually take a looooot of time to build it, so it is
-    # Better to pull here
-    if (( NUM_DIFF >= MAGIC_CUT_OFF_NUMBER_OF_LAYERS )); then
-        echo
-        echo
-        echo "WARNING! Your image and the dockerhub image differ significantly"
-        echo
-        echo "Forcing pulling the images. It will be faster than rebuilding usually."
-        echo "You can avoid it by setting SKIP_CHECK_REMOTE_IMAGE to true"
-        echo
-        export FORCE_PULL_IMAGES="true"
-    else
-        echo
-        echo "No need to pull the image. Local rebuild will be faster"
-        echo
-    fi
-}
-
-# Only rebuilds CI image if needed. It checks if the docker image build is needed
-# because any of the important source files (from common/_files_for_rebuild_check.sh) has
-# changed or in any of the edge cases (docker image removed, .build cache removed etc.
-# In case rebuild is needed, it determines (by comparing layers in local and remote image)
-# Whether pull is needed before rebuild.
-function rebuild_ci_image_if_needed() {
-    if [[ ${SKIP_CI_IMAGE_CHECK:="false"} == "true" ]]; then
-        echo
-        echo "Skip checking CI image"
-        echo
-        return
-    fi
-    if [[ -f "${BUILT_IMAGE_FLAG_FILE}" ]]; then
-        print_info
-        print_info "${THE_IMAGE_TYPE} image already built locally."
-        print_info
-    else
-        print_info
-        print_info "${THE_IMAGE_TYPE} image not built locally: pulling and building"
-        print_info
-        export FORCE_PULL_IMAGES="true"
-        export FORCE_BUILD_IMAGES="true"
-    fi
-
-    NEEDS_DOCKER_BUILD="false"
-    check_if_docker_build_is_needed
-    if [[ ${NEEDS_DOCKER_BUILD} == "true" ]]; then
-        if [[ ${SKIP_CHECK_REMOTE_IMAGE:=} != "true" && ${DOCKER_CACHE} == "pulled" ]]; then
-            # Check if remote image is different enough to force pull
-            # This is an optimisation pull vs. build time. When there
-            # are enough changes (specifically after setup.py changes) it is faster to pull
-            # and build the image rather than just build it
-            echo
-            echo "Checking if the remote image needs to be pulled"
-            echo
-            get_remote_image_info
-            if [[ ${REMOTE_DOCKER_REGISTRY_UNREACHABLE:=} != "true" ]]; then
-                get_local_image_info
-                if [[ ${LOCAL_MANIFEST_IMAGE_UNAVAILABLE:=} != "true" ]]; then
-                    compare_layers
-                else
-                    FORCE_PULL_IMAGES="true"
-                fi
-            fi
-        fi
-        SKIP_REBUILD="false"
-        if [[ ${CI:=} != "true" && "${FORCE_BUILD:=}" != "true" ]]; then
-            confirm_image_rebuild
-        fi
-        if [[ ${SKIP_REBUILD} != "true" ]]; then
-            SYSTEM=$(uname -s)
-            if [[ ${SYSTEM} != "Darwin" ]]; then
-                ROOT_FILES_COUNT=$(find "airflow" "tests" -user root | wc -l | xargs)
-                if [[ ${ROOT_FILES_COUNT} != "0" ]]; then
-                    ./scripts/ci/ci_fix_ownership.sh
-                fi
-            fi
-            print_info
-            print_info "Build start: ${THE_IMAGE_TYPE} image."
-            print_info
-            build_ci_image
-            update_all_md5_files
-            build_ci_image_manifest
-            print_info
-            print_info "Build completed: ${THE_IMAGE_TYPE} image."
-            print_info
-        fi
-    else
-        print_info
-        print_info "No need to build - none of the important files changed: ${FILES_FOR_REBUILD_CHECK[*]}"
-        print_info
-    fi
-}
-
-#
-# Starts the script.
-# If VERBOSE_COMMANDS variable is set to true, it enables verbose output of commands executed
-# Also prints some useful diagnostics information at start of the script if VERBOSE is set to true
-#
-function script_start {
-    print_info
-    print_info "Running $(basename "$0")"
-    print_info
-    print_info "Log is redirected to ${OUTPUT_LOG}"
-    print_info
-    if [[ ${VERBOSE_COMMANDS:="false"} == "true" ]]; then
-        print_info
-        print_info "Variable VERBOSE_COMMANDS Set to \"true\""
-        print_info "You will see a lot of output"
-        print_info
-        set -x
-    else
-        print_info "You can increase verbosity by running 'export VERBOSE_COMMANDS=\"true\""
-        if [[ ${SKIP_CACHE_DELETION:=} != "true" ]]; then
-            print_info "And skip deleting the output file with 'export SKIP_CACHE_DELETION=\"true\""
-        fi
-        print_info
-        set +x
-    fi
-    START_SCRIPT_TIME=$(date +%s)
-}
-
-#
-# Trap function executed always at the end of the script. In case of verbose output it also
-# Prints the exit code that the script exits with. Removes verbosity of commands in case it was run with
-# command verbosity and in case the script was not run from Breeze (so via ci scripts) it displays
-# total time spent in the script so that we can easily see it.
-#
-function script_end {
-    #shellcheck disable=2181
-    EXIT_CODE=$?
-    if [[ ${EXIT_CODE} != 0 ]]; then
-        print_info "###########################################################################################"
-        print_info "                   EXITING WITH STATUS CODE ${EXIT_CODE}"
-        print_info "###########################################################################################"
-    fi
-    if [[ ${VERBOSE_COMMANDS:="false"} == "true" ]]; then
-        set +x
-    fi
-
-    if [[ ${#FILES_TO_CLEANUP_ON_EXIT[@]} -gt 0 ]]; then
-      rm -rf -- "${FILES_TO_CLEANUP_ON_EXIT[@]}"
-    fi
-
-    END_SCRIPT_TIME=$(date +%s)
-    RUN_SCRIPT_TIME=$((END_SCRIPT_TIME-START_SCRIPT_TIME))
-    if [[ ${BREEZE:=} != "true" ]]; then
-        print_info
-        print_info "Finished the script $(basename "$0")"
-        print_info "Elapsed time spent in the script: ${RUN_SCRIPT_TIME} seconds"
-        print_info "Exit code ${EXIT_CODE}"
-        print_info
-    fi
-    remove_cache_directory
-}
-
-# Changes directory to local sources
-function go_to_airflow_sources {
-    print_info
-    pushd "${AIRFLOW_SOURCES}" &>/dev/null || exit 1
-    print_info
-    print_info "Running in host in $(pwd)"
-    print_info
-}
-
-#
-# Performs basic sanity checks common for most of the scripts in this directory
-#
-function basic_sanity_checks() {
-    assert_not_in_container
-    go_to_airflow_sources
-    check_if_coreutils_installed
-    create_cache_directory
-    sanitize_mounted_files
-}
-
-
-# Interactive version of confirming the ci image that is used in pre-commits
-# it displays additional information - what the user should do in order to bring the local images
-# back to state that pre-commit will be happy with
-function rebuild_ci_image_if_needed_and_confirmed() {
-    NEEDS_DOCKER_BUILD="false"
-    THE_IMAGE_TYPE="CI"
-
-    check_if_docker_build_is_needed
-
-    if [[ ${NEEDS_DOCKER_BUILD} == "true" ]]; then
-        print_info
-        print_info "Docker image build is needed!"
-        print_info
-    else
-        print_info
-        print_info "Docker image build is not needed!"
-        print_info
-    fi
-
-    if [[ "${NEEDS_DOCKER_BUILD}" == "true" ]]; then
-        echo
-        echo "Some of your images need to be rebuild because important files (like package list) has changed."
-        echo
-        echo "You have those options:"
-        echo "   * Rebuild the images now by answering 'y' (this might take some time!)"
-        echo "   * Skip rebuilding the images and hope changes are not big (you will be asked again)"
-        echo "   * Quit and manually rebuild the images using one of the following commands"
-        echo "        * ./breeze build-image"
-        echo "        * ./breeze build-image --force-pull-images"
-        echo
-        echo "   The first command works incrementally from your last local build."
-        echo "   The second command you use if you want to completely refresh your images from dockerhub."
-        echo
-        SKIP_REBUILD="false"
-        confirm_image_rebuild
-
-        if [[ ${SKIP_REBUILD} != "true" ]]; then
-            rebuild_ci_image_if_needed
-        fi
-    fi
-}
-
-# Checks if any of the files match the regexp specified the parameters here should be
-# match_files_regexp REGEXP FILE FILE ...
-function match_files_regexp() {
-    FILE_MATCHES="false"
-    REGEXP=${1}
-    while (($#))
-    do
-        REGEXP=${1}
-        for FILE in ${CHANGED_FILE_NAMES}
-        do
-          if  [[ ${FILE} =~ ${REGEXP} ]]; then
-             FILE_MATCHES="true"
-          fi
-        done
-        shift
-    done
-    export FILE_MATCHES
-}
-
-# Retrieves CI environment variables needed - depending on the CI system we run it in.
-# We try to be CI - agnostic and our scripts should run the same way on different CI systems
-# (This makes it easy to move between different CI systems)
-# This function maps CI-specific variables into a generic ones (prefixed with CI_) that
-# we used in other scripts
-function get_ci_environment() {
-    export CI_EVENT_TYPE="manual"
-    export CI_TARGET_REPO="apache/airflow"
-    export CI_TARGET_BRANCH="master"
-    export CI_SOURCE_REPO="apache/airflow"
-    export CI_SOURCE_BRANCH="master"
-    export CI_BUILD_ID="default-build-id"
-    export CI_JOB_ID="default-job-id"
-    if [[ ${CI:=} != "true" ]]; then
-        echo
-        echo "This is not a CI environment!. Staying with the defaults"
-        echo
-    else
-        if [[ ${TRAVIS:=} == "true" ]]; then
-            export CI_TARGET_REPO="${TRAVIS_REPO_SLUG}"
-            export CI_TARGET_BRANCH="${TRAVIS_BRANCH}"
-            export CI_BUILD_ID="${TRAVIS_BUILD_ID}"
-            export CI_JOB_ID="${TRAVIS_JOB_ID}"
-            if [[ "${TRAVIS_PULL_REQUEST:=}" == "true" ]]; then
-                export CI_EVENT_TYPE="pull_request"
-                export CI_SOURCE_REPO="${TRAVIS_PULL_REQUEST_SLUG}"
-                export CI_SOURCE_BRANCH="${TRAVIS_PULL_REQUEST_BRANCH}"
-            elif [[ "${TRAVIS_EVENT_TYPE:=}" == "cron" ]]; then
-                export CI_EVENT_TYPE="schedule"
-            else
-                export CI_EVENT_TYPE="push"
-            fi
-        elif [[ ${GITHUB_ACTIONS:=} == "true" ]]; then
-            export CI_TARGET_REPO="${GITHUB_REPOSITORY}"
-            export CI_TARGET_BRANCH="${GITHUB_BASE_REF}"
-            export CI_BUILD_ID="${GITHUB_RUN_ID}"
-            export CI_JOB_ID="${GITHUB_JOB}"
-            if [[ ${GITHUB_EVENT_NAME:=} == "pull_request" ]]; then
-                export CI_EVENT_TYPE="pull_request"
-                # default name of the source repo (assuming it's forked without rename)
-                export SOURCE_AIRFLOW_REPO=${SOURCE_AIRFLOW_REPO:="airflow"}
-                # For Pull Requests it's ambiguous to find the PR and we need to
-                # assume that name of repo is airflow but it could be overridden in case it's not
-                export CI_SOURCE_REPO="${GITHUB_ACTOR}/${SOURCE_AIRFLOW_REPO}"
-                export CI_SOURCE_BRANCH="${GITHUB_HEAD_REF}"
-                BRANCH_EXISTS=$(git ls-remote --heads \
-                    "https://github.com/${CI_SOURCE_REPO}.git" "${CI_SOURCE_BRANCH}")
-                if [[ ${BRANCH_EXISTS} == "" ]]; then
-                    echo
-                    echo "https://github.com/${CI_SOURCE_REPO}.git Branch ${CI_SOURCE_BRANCH} does not exist"
-                    echo
-                    echo
-                    echo "Fallback to https://github.com/${CI_TARGET_REPO}.git Branch ${CI_TARGET_BRANCH}"
-                    echo
-                    # Fallback to the target repository if the repo does not exist
-                    export CI_SOURCE_REPO="${CI_TARGET_REPO}"
-                    export CI_SOURCE_BRANCH="${CI_TARGET_BRANCH}"
-                fi
-            elif [[ ${GITHUB_EVENT_TYPE:=} == "schedule" ]]; then
-                export CI_EVENT_TYPE="schedule"
-            else
-                export CI_EVENT_TYPE="push"
-            fi
-        else
-            echo
-            echo "ERROR! Unknown CI environment. Exiting"
-            exit 1
-        fi
-    fi
-    echo
-    echo "Detected CI build environment"
-    echo
-    echo "CI_EVENT_TYPE=${CI_EVENT_TYPE}"
-    echo "CI_TARGET_REPO=${CI_TARGET_REPO}"
-    echo "CI_TARGET_BRANCH=${CI_TARGET_BRANCH}"
-    echo "CI_SOURCE_REPO=${CI_SOURCE_REPO}"
-    echo "CI_SOURCE_BRANCH=${CI_SOURCE_BRANCH}"
-    echo "CI_BUILD_ID=${CI_BUILD_ID}"
-    echo "CI_JOB_ID=${CI_JOB_ID}"
-    echo
-}
-
-# Builds the CI image in the CI environment.
-# Depending on the type of build (push/pr/scheduled) it will either build it incrementally or
-# from the scratch without cache (the latter for scheduled builds only)
-function build_ci_image_on_ci() {
-    get_ci_environment
-
-    # In case of CRON jobs we run builds without cache and upgrade to latest requirements
-    if [[ "${CI_EVENT_TYPE:=}" == "schedule" ]]; then
-        echo
-        echo "Disabling cache for scheduled jobs"
-        echo
-        export DOCKER_CACHE="no-cache"
-        echo
-        echo "Requirements are upgraded to latest while running Docker build"
-        echo
-        export UPGRADE_TO_LATEST_REQUIREMENTS="true"
-    fi
-
-    prepare_ci_build
-
-    rm -rf "${BUILD_CACHE_DIR}"
-    mkdir -pv "${BUILD_CACHE_DIR}"
-
-    rebuild_ci_image_if_needed
-
-    # Disable force pulling forced above this is needed for the subsequent scripts so that
-    # They do not try to pull/build images again
-    unset FORCE_PULL_IMAGES
-    unset FORCE_BUILD
-}
-
-# Reads environment variable passed as first parameter from the .build cache file
-function read_from_file {
-    cat "${BUILD_CACHE_DIR}/.$1" 2>/dev/null || true
-}
-
-# Saves environment variable passed as first parameter to the .build cache file
-function save_to_file {
-    # shellcheck disable=SC2005
-    echo "$(eval echo "\$$1")" > "${BUILD_CACHE_DIR}/.$1"
-}
-
-# check if parameter set for the variable is allowed (should be on the _BREEZE_ALLOWED list)
-# and if it is, it saves it to .build cache file. In case the parameter is wrong, the
-# saved variable is removed (so that bad value is not used again in case it comes from there)
-# and exits with an error
-function check_and_save_allowed_param {
-    _VARIABLE_NAME="${1}"
-    _VARIABLE_DESCRIPTIVE_NAME="${2}"
-    _FLAG="${3}"
-    _ALLOWED_VALUES_ENV_NAME="_BREEZE_ALLOWED_${_VARIABLE_NAME}S"
-    _ALLOWED_VALUES=" ${!_ALLOWED_VALUES_ENV_NAME//$'\n'/ } "
-    _VALUE=${!_VARIABLE_NAME}
-    if [[ ${_ALLOWED_VALUES:=} != *" ${_VALUE} "* ]]; then
-        echo >&2
-        echo >&2 "ERROR:  Allowed ${_VARIABLE_DESCRIPTIVE_NAME}: [${_ALLOWED_VALUES}]. Is: '${!_VARIABLE_NAME}'."
-        echo >&2
-        echo >&2 "Switch to supported value with ${_FLAG} flag."
-
-        if [[ -n ${!_VARIABLE_NAME} && \
-            -f "${BUILD_CACHE_DIR}/.${_VARIABLE_NAME}" && \
-            ${!_VARIABLE_NAME} == $(cat "${BUILD_CACHE_DIR}/.${_VARIABLE_NAME}" ) ]]; then
-            echo >&2
-            echo >&2 "Removing ${BUILD_CACHE_DIR}/.${_VARIABLE_NAME}. Next time you run it, it should be OK."
-            echo >&2
-            rm -f "${BUILD_CACHE_DIR}/.${_VARIABLE_NAME}"
-        fi
-        exit 1
-    fi
-    save_to_file "${_VARIABLE_NAME}"
-}
-
-# Docker command to build documentation
-function run_docs() {
-    verbose_docker run "${EXTRA_DOCKER_FLAGS[@]}" -t \
-            --entrypoint "/usr/local/bin/dumb-init"  \
-            --env PYTHONDONTWRITEBYTECODE \
-            --env VERBOSE \
-            --env VERBOSE_COMMANDS \
-            --env HOST_USER_ID="$(id -ur)" \
-            --env HOST_GROUP_ID="$(id -gr)" \
-            --rm \
-            "${AIRFLOW_CI_IMAGE}" \
-            "--" "/opt/airflow/docs/build" \
-            | tee -a "${OUTPUT_LOG}"
-}
-
-# Pulls image in case it is needed (either has never been pulled or pulling was forced
-# Should be run with set +e
-# Parameters:
-#   $1 -> image to pull
-function pull_image_if_needed() {
-    local IMAGE_TO_PULL="${1}"
-    local IMAGE_HASH
-    IMAGE_HASH=$(docker images -q "${IMAGE_TO_PULL}" 2> /dev/null || true)
-    local PULL_IMAGE=${FORCE_PULL_IMAGES}
-
-    if [[ "${IMAGE_HASH}" == "" ]]; then
-        PULL_IMAGE="true"
-    fi
-    if [[ "${PULL_IMAGE}" == "true" ]]; then
-        echo
-        echo "Pulling the image ${IMAGE_TO_PULL}"
-        echo
-        verbose_docker pull "${IMAGE_TO_PULL}" | tee -a "${OUTPUT_LOG}"
-        EXIT_VALUE="$?"
-        echo
-        return ${EXIT_VALUE}
-    fi
-}
-
-# Pulls image if needed but tries to pull it from cache (for example GitHub registry) before
-# It attempts to pull it from the main repository. This is used to speed up the builds
-# In GitHub Actions.
-# Parameters:
-#   $1 -> image to pull
-#   $2 -> cache image to pull first
-function pull_image_possibly_from_cache() {
-    local IMAGE="${1}"
-    local CACHED_IMAGE="${2}"
-    local IMAGE_PULL_RETURN_VALUE=-1
-
-    set +e
-    if [[ ${CACHED_IMAGE:=} != "" ]]; then
-        pull_image_if_needed "${CACHED_IMAGE}"
-        IMAGE_PULL_RETURN_VALUE="$?"
-        if [[ ${IMAGE_PULL_RETURN_VALUE} == "0" ]]; then
-            # Tag the image to be the target one
-            verbose_docker tag "${CACHED_IMAGE}" "${IMAGE}"
-        fi
-    fi
-    if [[ ${IMAGE_PULL_RETURN_VALUE} != "0" ]]; then
-        pull_image_if_needed "${IMAGE}"
-    fi
-    set -e
-}
-
-# Pulls CI image in case caching strategy is "pulled" and the image needs to be pulled
-function pull_ci_image_if_needed() {
-    # Whether to force pull images to populate cache
-    export FORCE_PULL_IMAGES=${FORCE_PULL_IMAGES:="false"}
-
-    if [[ "${DOCKER_CACHE}" == "pulled" ]]; then
-        if [[ "${FORCE_PULL_IMAGES}" == "true" ]]; then
-            echo
-            echo "Force pull base image ${PYTHON_BASE_IMAGE}"
-            echo
-            if [[ -n ${DETECTED_TERMINAL:=""} ]]; then
-                echo -n "
-Docker pulling ${PYTHON_BASE_IMAGE}.
-                    " > "${DETECTED_TERMINAL}"
-            fi
-            if [[ ${PULL_PYTHON_BASE_IMAGES_FROM_CACHE:="true"} == "true" ]]; then
-                pull_image_possibly_from_cache "${PYTHON_BASE_IMAGE}" "${CACHED_PYTHON_BASE_IMAGE}"
-            else
-                verbose_docker pull "${PYTHON_BASE_IMAGE}" | tee -a "${OUTPUT_LOG}"
-            fi
-            echo
-        fi
-        pull_image_possibly_from_cache "${AIRFLOW_CI_IMAGE}" "${CACHED_AIRFLOW_CI_IMAGE}"
-    fi
-}
-
-
-# Pulls PROD image in case caching strategy is "pulled" and the image needs to be pulled
-function pull_prod_images_if_needed() {
-    # Whether to force pull images to populate cache
-    export FORCE_PULL_IMAGES=${FORCE_PULL_IMAGES:="false"}
-
-    if [[ "${DOCKER_CACHE}" == "pulled" ]]; then
-        if [[ "${FORCE_PULL_IMAGES}" == "true" ]]; then
-            echo
-            echo "Force pull base image ${PYTHON_BASE_IMAGE}"
-            echo
-            if [[ ${PULL_PYTHON_BASE_IMAGES_FROM_CACHE:="true"} == "true" ]]; then
-                pull_image_possibly_from_cache "${PYTHON_BASE_IMAGE}" "${CACHED_PYTHON_BASE_IMAGE}"
-            else
-                verbose_docker pull "${PYTHON_BASE_IMAGE}" | tee -a "${OUTPUT_LOG}"
-            fi
-            echo
-        fi
-        # "Build" segment of production image
-        pull_image_possibly_from_cache "${AIRFLOW_PROD_BUILD_IMAGE}" "${CACHED_AIRFLOW_PROD_BUILD_IMAGE}"
-        # Main segment of production image
-        pull_image_possibly_from_cache "${AIRFLOW_PROD_IMAGE}" "${CACHED_AIRFLOW_PROD_IMAGE}"
-    fi
-}
-
-# Prints summary of the build parameters
-function print_build_info() {
-    print_info
-    print_info "Airflow ${AIRFLOW_VERSION} Python: ${PYTHON_MAJOR_MINOR_VERSION}. Image description: ${IMAGE_DESCRIPTION}"
-    print_info
-}
-
-# Function to spin ASCII spinner during pull and build in pre-commits to give the user indication that
-# Pull/Build is happening. It only spins if the output log changes, so if pull/build is stalled
-# The spinner will not move.
-function spin() {
-    local FILE_TO_MONITOR=${1}
-    local SPIN=("-" "\\" "|" "/")
-    echo -n "
-Build log: ${FILE_TO_MONITOR}
-" > "${DETECTED_TERMINAL}"
-
-    LAST_STEP=""
-    while "true"
-    do
-      for i in "${SPIN[@]}"
-      do
-            echo -ne "\r${LAST_STEP}$i" > "${DETECTED_TERMINAL}"
-            local LAST_FILE_SIZE
-            local FILE_SIZE
-            LAST_FILE_SIZE=$(set +e; wc -c "${FILE_TO_MONITOR}" 2>/dev/null | awk '{print $1}' || true)
-            FILE_SIZE=${LAST_FILE_SIZE}
-            while [[ "${LAST_FILE_SIZE}" == "${FILE_SIZE}" ]];
-            do
-                FILE_SIZE=$(set +e; wc -c "${FILE_TO_MONITOR}" 2>/dev/null | awk '{print $1}' || true)
-                sleep 0.2
-            done
-            LAST_FILE_SIZE=FILE_SIZE
-            sleep 0.2
-            if [[ ! -f "${FILE_TO_MONITOR}" ]]; then
-                exit
-            fi
-            LAST_LINE=$(set +e; grep "Step" <"${FILE_TO_MONITOR}" | tail -1 || true)
-            [[ ${LAST_LINE} =~ ^(Step [0-9/]*)\ : ]] && LAST_STEP="${BASH_REMATCH[1]} :"
-      done
-    done
-}
-
-# Builds CI image - depending on the caching strategy (pulled, local, no-cache) it
-# passes the necessary docker build flags via DOCKER_CACHE_CI_DIRECTIVE array
-# it also passes the right Build args depending on the configuration of the build
-# selected by Breeze flags or environment variables.
-function build_ci_image() {
-    print_build_info
-    if [[ -n ${DETECTED_TERMINAL:=""} ]]; then
-        echo -n "Preparing ${AIRFLOW_CI_IMAGE}.
-        " > "${DETECTED_TERMINAL}"
-        spin "${OUTPUT_LOG}" &
-        SPIN_PID=$!
-        # shellcheck disable=SC2064
-        trap "kill ${SPIN_PID}" SIGINT SIGTERM
-    fi
-    pull_ci_image_if_needed
-
-    if [[ "${DOCKER_CACHE}" == "no-cache" ]]; then
-        export DOCKER_CACHE_CI_DIRECTIVE=("--no-cache")
-    elif [[ "${DOCKER_CACHE}" == "local" ]]; then
-        export DOCKER_CACHE_CI_DIRECTIVE=()
-    elif [[ "${DOCKER_CACHE}" == "pulled" ]]; then
-        export DOCKER_CACHE_CI_DIRECTIVE=(
-            "--cache-from" "${AIRFLOW_CI_IMAGE}"
-        )
-    else
-        echo 2>&1
-        echo 2>&1 "Error - thee ${DOCKER_CACHE} cache is unknown!"
-        echo 2>&1
-        exit 1
-    fi
-    if [[ -n ${DETECTED_TERMINAL:=""} ]]; then
-        echo -n "
-Docker building ${AIRFLOW_CI_IMAGE}.
-" > "${DETECTED_TERMINAL}"
-    fi
-    set +u
-    verbose_docker build \
-        --build-arg PYTHON_BASE_IMAGE="${PYTHON_BASE_IMAGE}" \
-        --build-arg PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION}" \
-            --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
-        --build-arg AIRFLOW_BRANCH="${BRANCH_NAME}" \
-        --build-arg AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS}" \
-        --build-arg ADDITIONAL_AIRFLOW_EXTRAS="${ADDITIONAL_AIRFLOW_EXTRAS}" \
-        --build-arg ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS}" \
-        --build-arg AIRFLOW_CONTAINER_CI_OPTIMISED_BUILD="${AIRFLOW_CONTAINER_CI_OPTIMISED_BUILD}" \
-        --build-arg UPGRADE_TO_LATEST_REQUIREMENTS="${UPGRADE_TO_LATEST_REQUIREMENTS}" \
-        "${DOCKER_CACHE_CI_DIRECTIVE[@]}" \
-        -t "${AIRFLOW_CI_IMAGE}" \
-        --target "main" \
-        . -f Dockerfile.ci | tee -a "${OUTPUT_LOG}"
-    set -u
-    if [[ -n "${DEFAULT_IMAGE:=}" ]]; then
-        verbose_docker tag "${AIRFLOW_CI_IMAGE}" "${DEFAULT_IMAGE}" | tee -a "${OUTPUT_LOG}"
-    fi
-    if [[ -n ${SPIN_PID:=""} ]]; then
-        kill "${SPIN_PID}" || true
-        wait "${SPIN_PID}" || true
-        echo > "${DETECTED_TERMINAL}"
-    fi
-}
-
-# Builds PROD image - depending on the caching strategy (pulled, local, no-cache) it
-# passes the necessary docker build flags via DOCKER_CACHE_PROD_DIRECTIVE and
-# DOCKER_CACHE_PROD_BUILD_DIRECTIVE (separate caching options are needed for "build" segment of the image)
-# it also passes the right Build args depending on the configuration of the build
-# selected by Breeze flags or environment variables.
-function build_prod_image() {
-    print_build_info
-    pull_prod_images_if_needed
-
-    if [[ "${DOCKER_CACHE}" == "no-cache" ]]; then
-        export DOCKER_CACHE_PROD_DIRECTIVE=("--cache-from" "${AIRFLOW_PROD_BUILD_IMAGE}")
-        export DOCKER_CACHE_PROD_BUILD_DIRECTIVE=("--no-cache")
-    elif [[ "${DOCKER_CACHE}" == "local" ]]; then
-        export DOCKER_CACHE_PROD_DIRECTIVE=()
-        export DOCKER_CACHE_PROD_BUILD_DIRECTIVE=()
-    elif [[ "${DOCKER_CACHE}" == "pulled" ]]; then
-        export DOCKER_CACHE_PROD_DIRECTIVE=(
-            "--cache-from" "${AIRFLOW_PROD_BUILD_IMAGE}"
-            "--cache-from" "${AIRFLOW_PROD_IMAGE}"
-        )
-        export DOCKER_CACHE_PROD_BUILD_DIRECTIVE=(
-            "--cache-from" "${AIRFLOW_PROD_BUILD_IMAGE}"
-        )
-    else
-        echo 2>&1
-        echo 2>&1 "Error - thee ${DOCKER_CACHE} cache is unknown!"
-        echo 2>&1
-        exit 1
-    fi
-    set +u
-    verbose_docker build \
-        "${EXTRA_DOCKER_PROD_BUILD_FLAGS[@]}" \
-        --build-arg PYTHON_BASE_IMAGE="${PYTHON_BASE_IMAGE}" \
-        --build-arg PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION}" \
-        --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
-        --build-arg AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS}" \
-        --build-arg ADDITIONAL_AIRFLOW_EXTRAS="${ADDITIONAL_AIRFLOW_EXTRAS}" \
-        --build-arg ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS}" \
-        "${DOCKER_CACHE_PROD_BUILD_DIRECTIVE[@]}" \
-        -t "${AIRFLOW_PROD_BUILD_IMAGE}" \
-        --target "airflow-build-image" \
-        . -f Dockerfile | tee -a "${OUTPUT_LOG}"
-    verbose_docker build \
-        "${EXTRA_DOCKER_PROD_BUILD_FLAGS[@]}" \
-        --build-arg PYTHON_BASE_IMAGE="${PYTHON_BASE_IMAGE}" \
-        --build-arg PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION}" \
-        --build-arg ADDITIONAL_AIRFLOW_EXTRAS="${ADDITIONAL_AIRFLOW_EXTRAS}" \
-        --build-arg ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS}" \
-        --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
-        --build-arg AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS}" \
-        "${DOCKER_CACHE_PROD_DIRECTIVE[@]}" \
-        -t "${AIRFLOW_PROD_IMAGE}" \
-        --target "main" \
-        . -f Dockerfile | tee -a "${OUTPUT_LOG}"
-    set -u
-    if [[ -n "${DEFAULT_IMAGE:=}" ]]; then
-        verbose_docker tag "${AIRFLOW_PROD_IMAGE}" "${DEFAULT_IMAGE}" | tee -a "${OUTPUT_LOG}"
-    fi
-}
-
-
-# Removes airflow CI and base images
-function remove_all_images() {
-    echo
-    "${AIRFLOW_SOURCES}/confirm" "Removing all local images ."
-    echo
-    verbose_docker rmi "${PYTHON_BASE_IMAGE}" || true
-    verbose_docker rmi "${AIRFLOW_CI_IMAGE}" || true
-    echo
-    echo "###################################################################"
-    echo "NOTE!! Removed Airflow images for Python version ${PYTHON_MAJOR_MINOR_VERSION}."
-    echo "       But the disk space in docker will be reclaimed only after"
-    echo "       running 'docker system prune' command."
-    echo "###################################################################"
-    echo
-}
-
-# Fixing permissions for all important files that are going to be added to Docker context
-# This is necessary, because there are different default umask settings on different *NIX
-# In case of some systems (especially in the CI environments) there is default +w group permission
-# set automatically via UMASK when git checkout is performed.
-#    https://unix.stackexchange.com/questions/315121/why-is-the-default-umask-002-or-022-in-many-unix-systems-seems-insecure-by-defa
-# Unfortunately default setting in git is to use UMASK by default:
-#    https://git-scm.com/docs/git-config/1.6.3.1#git-config-coresharedRepository
-# This messes around with Docker context invalidation because the same files have different permissions
-# and effectively different hash used for context validation calculation.
-#
-# We fix it by removing write permissions for other/group for all files that are in the Docker context.
-#
-# Since we can't (easily) tell what dockerignore would restrict, we'll just to
-# it to "all" files in the git repo, making sure to exclude the www/static/docs
-# symlink which is broken until the docs are built.
-function filterout_deleted_files {
-  # Take NUL-separated stdin, return only files that exist on stdout NUL-separated
-  # This is to cope with users deleting files or folders locally but not doing `git rm`
-  xargs -0 "$STAT_BIN" --printf '%n\0' 2>/dev/null || true;
-}
-
-# Fixes permissions for groups for all the files that are quickly filtered using the filterout_deleted_files
-function fix_group_permissions() {
-    if [[ ${PERMISSIONS_FIXED:=} == "true" ]]; then
-        echo
-        echo "Permissions already fixed"
-        echo
-        return
-    fi
-    echo
-    echo "Fixing group permissions"
-    pushd "${AIRFLOW_SOURCES}" >/dev/null
-    # This deals with files
-    git ls-files -z -- ./ | filterout_deleted_files | xargs -0 chmod og-w
-    # and this deals with directories
-    git ls-tree -z -r -d --name-only HEAD | filterout_deleted_files | xargs -0 chmod og-w,og+x
-    popd >/dev/null
-    echo "Fixed group permissions"
-    echo
-    export PERMISSIONS_FIXED="true"
-}
-
-# Prepares all variables needed by the CI build. Depending on the configuration used (python version
-# DockerHub user etc. the variables are set so that other functions can use those variables.
-function prepare_ci_build() {
-    export AIRFLOW_CI_BASE_TAG="${DEFAULT_BRANCH}-python${PYTHON_MAJOR_MINOR_VERSION}-ci"
-    export AIRFLOW_CI_LOCAL_MANIFEST_IMAGE="local/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}-manifest"
-    export AIRFLOW_CI_REMOTE_MANIFEST_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}-manifest"
-    export AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}"
-    if [[ ${ENABLE_REGISTRY_CACHE="false"} == "true" ]]; then
-        if [[ ${CACHE_REGISTRY_PASSWORD:=} != "" ]]; then
-            echo "${CACHE_REGISTRY_PASSWORD}" | docker login \
-                --username "${CACHE_REGISTRY_USERNAME}" \
-                --password-stdin \
-                "${CACHE_REGISTRY}"
-        fi
-        export CACHE_IMAGE_PREFIX=${CACHE_IMAGE_PREFX:=${GITHUB_ORGANISATION}/${GITHUB_REPO}}
-        export CACHED_AIRFLOW_CI_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_CI_BASE_TAG}"
-        export CACHED_PYTHON_BASE_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
-    else
-        export CACHED_AIRFLOW_CI_IMAGE=""
-        export CACHED_PYTHON_BASE_IMAGE=""
-    fi
-    export AIRFLOW_BUILD_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}/${AIRFLOW_CI_BASE_TAG}"
-    export AIRFLOW_CI_IMAGE_DEFAULT="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DEFAULT_BRANCH}-ci"
-    export PYTHON_BASE_IMAGE="python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
-    export BUILT_IMAGE_FLAG_FILE="${BUILD_CACHE_DIR}/${BRANCH_NAME}/.built_${PYTHON_MAJOR_MINOR_VERSION}"
-    if [[ "${DEFAULT_PYTHON_MAJOR_MINOR_VERSION}" == "${PYTHON_MAJOR_MINOR_VERSION}" ]]; then
-        export DEFAULT_IMAGE="${AIRFLOW_CI_IMAGE_DEFAULT}"
-    else
-        export DEFAULT_IMAGE=""
-    fi
-    export THE_IMAGE_TYPE="CI"
-    export IMAGE_DESCRIPTION="Airflow CI"
-    export AIRFLOW_CONTAINER_CI_OPTIMISED_BUILD="true"
-    export AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS:="${DEFAULT_CI_EXTRAS}"}"
-    export ADDITIONAL_AIRFLOW_EXTRAS="${ADDITIONAL_AIRFLOW_EXTRAS:=""}"
-    export ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS:=""}"
-    export AIRFLOW_IMAGE="${AIRFLOW_CI_IMAGE}"
-    go_to_airflow_sources
-    fix_group_permissions
-}
-
-# For remote installation of airflow (from GitHub or Pypi) when building the image, you need to
-# pass build flags depending on the version and method of the installation (for example to
-# get proper requirement constraint files)
-function add_build_args_for_remote_install() {
-    # entrypoint is used as AIRFLOW_SOURCES_FROM/TO in order to avoid costly copying of all sources of
-    # Airflow - those are not needed for remote install at all. Entrypoint is later overwritten by
-    # ENTRYPOINT_FILE - downloaded entrypoint.sh so this is only for the purpose of iteration on Dockerfile
-    EXTRA_DOCKER_PROD_BUILD_FLAGS+=(
-        "--build-arg" "AIRFLOW_SOURCES_FROM=entrypoint.sh"
-        "--build-arg" "AIRFLOW_SOURCES_TO=/entrypoint"
-    )
-    if [[ ${AIRFLOW_VERSION} =~ [^0-9]*1[^0-9]*10[^0-9]([0-9]*) ]]; then
-        # All types of references/versions match this regexp for 1.10 series
-        # for example v1_10_test, 1.10.10, 1.10.9 etc. ${BASH_REMATCH[1]} is the () group matches last
-        # minor digit of version and it's length is 0 for v1_10_test, 1 for 1.10.9 and 2 for 1.10.10+
-        if [[ ${#BASH_REMATCH[1]} == "1" ]]; then
-            # This is only for 1.10.0 - 1.10.9
-            EXTRA_DOCKER_PROD_BUILD_FLAGS+=(
-                "--build-arg" "CONSTRAINT_REQUIREMENTS=https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"
-                "--build-arg" "ENTRYPOINT_FILE=https://raw.githubusercontent.com/apache/airflow/1.10.10/entrypoint.sh"
-            )
-        else
-            EXTRA_DOCKER_PROD_BUILD_FLAGS+=(
-                # For 1.10.10+ and v1-10-test it's ok to use AIRFLOW_VERSION as reference
-                "--build-arg" "CONSTRAINT_REQUIREMENTS=https://raw.githubusercontent.com/apache/airflow/${AIRFLOW_VERSION}/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"
-                "--build-arg" "ENTRYPOINT_FILE=https://raw.githubusercontent.com/apache/airflow/${AIRFLOW_VERSION}/entrypoint.sh"
-            )
-        fi
-    else
-        # For all other (master, 2.0+) we just match ${AIRFLOW_VERSION}
-        EXTRA_DOCKER_PROD_BUILD_FLAGS+=(
-            "--build-arg" "CONSTRAINT_REQUIREMENTS=https://raw.githubusercontent.com/apache/airflow/${AIRFLOW_VERSION}/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"
-            "--build-arg" "ENTRYPOINT_FILE=https://raw.githubusercontent.com/apache/airflow/${AIRFLOW_VERSION}/entrypoint.sh"
-        )
-    fi
-}
-
-# Prepares all variables needed by the CI build. Depending on the configuration used (python version
-# DockerHub user etc. the variables are set so that other functions can use those variables.
-function prepare_prod_build() {
-    export AIRFLOW_PROD_BASE_TAG="${DEFAULT_BRANCH}-python${PYTHON_MAJOR_MINOR_VERSION}"
-    export AIRFLOW_PROD_BUILD_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_PROD_BASE_TAG}-build"
-    export AIRFLOW_PROD_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_PROD_BASE_TAG}"
-    export AIRFLOW_PROD_IMAGE_DEFAULT="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DEFAULT_BRANCH}"
-    export PYTHON_BASE_IMAGE="python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
-    if [[ "${DEFAULT_PYTHON_MAJOR_MINOR_VERSION}" == "${PYTHON_MAJOR_MINOR_VERSION}" ]]; then
-        export DEFAULT_IMAGE="${AIRFLOW_PROD_IMAGE_DEFAULT}"
-    else
-        export DEFAULT_IMAGE=""
-    fi
-    export THE_IMAGE_TYPE="PROD"
-    export IMAGE_DESCRIPTION="Airflow production"
-    export AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS:="${DEFAULT_PROD_EXTRAS}"}"
-    export ADDITIONAL_AIRFLOW_EXTRAS="${ADDITIONAL_AIRFLOW_EXTRAS:=""}"
-    export ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS:=""}"
-    export AIRFLOW_IMAGE="${AIRFLOW_PROD_IMAGE}"
-
-    if [[ ${ENABLE_REGISTRY_CACHE="false"} == "true" ]]; then
-        if [[ ${CACHE_REGISTRY_PASSWORD:=} != "" ]]; then
-            echo "${CACHE_REGISTRY_PASSWORD}" | docker login \
-                --username "${CACHE_REGISTRY_USERNAME}" \
-                --password-stdin \
-                "${CACHE_REGISTRY}"
-        fi
-        export CACHE_IMAGE_PREFIX=${CACHE_IMAGE_PREFX:=${GITHUB_ORGANISATION}/${GITHUB_REPO}}
-        export CACHED_AIRFLOW_PROD_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_PROD_BASE_TAG}"
-        export CACHED_AIRFLOW_PROD_BUILD_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_PROD_BASE_TAG}-build"
-        export CACHED_PYTHON_BASE_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
-    else
-        export CACHED_AIRFLOW_PROD_IMAGE=""
-        export CACHED_AIRFLOW_PROD_BUILD_IMAGE=""
-        export CACHED_PYTHON_BASE_IMAGE=""
-    fi
-    export AIRFLOW_KUBERNETES_IMAGE=${AIRFLOW_PROD_IMAGE}-kubernetes
-    AIRFLOW_KUBERNETES_IMAGE_NAME=$(echo "${AIRFLOW_KUBERNETES_IMAGE}" | cut -f 1 -d ":")
-    export AIRFLOW_KUBERNETES_IMAGE_NAME
-    AIRFLOW_KUBERNETES_IMAGE_TAG=$(echo "${AIRFLOW_KUBERNETES_IMAGE}" | cut -f 2 -d ":")
-    export AIRFLOW_KUBERNETES_IMAGE_TAG
-
-    if [[ "${INSTALL_AIRFLOW_REFERENCE:=}" != "" ]]; then
-        # When --install-airflow-reference is used then the image is build from github tag
-        EXTRA_DOCKER_PROD_BUILD_FLAGS=(
-            "--build-arg" "AIRFLOW_INSTALL_SOURCES=https://github.com/apache/airflow/archive/${INSTALL_AIRFLOW_REFERENCE}.tar.gz#egg=apache-airflow"
-        )
-        export AIRFLOW_VERSION="${INSTALL_AIRFLOW_REFERENCE}"
-        add_build_args_for_remote_install
-    elif [[ "${INSTALL_AIRFLOW_VERSION:=}" != "" ]]; then
-        # When --install-airflow-version is used then the image is build from PIP package
-        EXTRA_DOCKER_PROD_BUILD_FLAGS=(
-            "--build-arg" "AIRFLOW_INSTALL_SOURCES=apache-airflow"
-            "--build-arg" "AIRFLOW_INSTALL_VERSION===${INSTALL_AIRFLOW_VERSION}"
-        )
-        export AIRFLOW_VERSION="${INSTALL_AIRFLOW_VERSION}"
-        add_build_args_for_remote_install
-    else
-        # When no airflow version/reference is specified, production image is built from local sources
-        EXTRA_DOCKER_PROD_BUILD_FLAGS=(
-        )
-    fi
-    go_to_airflow_sources
-}
-
-# Pushes Ci image and it's manifest to the registry. In case the image was taken from cache registry
-# it is pushed to the cache, not to the main registry. Manifest is only pushed to the main registry
-function push_ci_image() {
-    if [[ ${CACHED_AIRFLOW_CI_IMAGE:=} != "" ]]; then
-        verbose_docker tag "${AIRFLOW_CI_IMAGE}" "${CACHED_AIRFLOW_CI_IMAGE}"
-        IMAGE_TO_PUSH="${CACHED_AIRFLOW_CI_IMAGE}"
-    else
-        IMAGE_TO_PUSH="${AIRFLOW_CI_IMAGE}"
-    fi
-    verbose_docker push "${IMAGE_TO_PUSH}"
-    if [[ ${CACHED_AIRFLOW_CI_IMAGE} == "" ]]; then
-        # Only push manifest image for builds that are not using CI cache
-        verbose_docker tag "${AIRFLOW_CI_LOCAL_MANIFEST_IMAGE}" "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}"
-        verbose_docker push "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}"
-        if [[ -n ${DEFAULT_IMAGE:=""} ]]; then
-            verbose_docker push "${DEFAULT_IMAGE}"
-        fi
-    fi
-    if [[ ${CACHED_PYTHON_BASE_IMAGE} != "" ]]; then
-        verbose_docker tag "${PYTHON_BASE_IMAGE}" "${CACHED_PYTHON_BASE_IMAGE}"
-        verbose_docker push "${CACHED_PYTHON_BASE_IMAGE}"
-    fi
-
-}
-
-# Pushes PROD image to the registry. In case the image was taken from cache registry
-# it is also pushed to the cache, not to the main registry
-function push_prod_images() {
-    if [[ ${CACHED_AIRFLOW_PROD_IMAGE:=} != "" ]]; then
-        verbose_docker tag "${AIRFLOW_PROD_IMAGE}" "${CACHED_AIRFLOW_PROD_IMAGE}"
-        IMAGE_TO_PUSH="${CACHED_AIRFLOW_PROD_IMAGE}"
-    else
-        IMAGE_TO_PUSH="${AIRFLOW_PROD_IMAGE}"
-    fi
-    if [[ ${CACHED_AIRFLOW_PROD_BUILD_IMAGE:=} != "" ]]; then
-        verbose_docker tag "${AIRFLOW_PROD_BUILD_IMAGE}" "${CACHED_AIRFLOW_PROD_BUILD_IMAGE}"
-        IMAGE_TO_PUSH_BUILD="${CACHED_AIRFLOW_PROD_BUILD_IMAGE}"
-    else
-        IMAGE_TO_PUSH_BUILD="${AIRFLOW_PROD_BUILD_IMAGE}"
-    fi
-    verbose_docker push "${IMAGE_TO_PUSH}"
-    verbose_docker push "${IMAGE_TO_PUSH_BUILD}"
-    if [[ -n ${DEFAULT_IMAGE:=""} && ${CACHED_AIRFLOW_PROD_IMAGE} == "" ]]; then
-        verbose_docker push "${DEFAULT_IMAGE}"
-    fi
-
-    # we do not need to push PYTHON base image here - they are already pushed in the CI push
-}
-
-# Docker command to generate constraint requirement files.
-function run_generate_requirements() {
-    docker run "${EXTRA_DOCKER_FLAGS[@]}" \
-        --entrypoint "/usr/local/bin/dumb-init"  \
-        --env PYTHONDONTWRITEBYTECODE \
-        --env VERBOSE \
-        --env VERBOSE_COMMANDS \
-        --env HOST_USER_ID="$(id -ur)" \
-        --env HOST_GROUP_ID="$(id -gr)" \
-        --env UPGRADE_WHILE_GENERATING_REQUIREMENTS \
-        --env PYTHON_MAJOR_MINOR_VERSION \
-        --env CHECK_REQUIREMENTS_ONLY \
-        --rm \
-        "${AIRFLOW_CI_IMAGE}" \
-        "--" "/opt/airflow/scripts/ci/in_container/run_generate_requirements.sh" \
-        | tee -a "${OUTPUT_LOG}"
-}
-
-function set_mysql_encoding() {
-    export MYSQL_ENCODING="utf8mb4"
-
-    if [[ ${MYSQL_VERSION:="5.7"} == "5.6" ]]; then
-        # In MySQL 5.6 maximum size of the index is 1536 bytes and it is too small for
-        # Airflow for utf8mb4 encoding (it takes 4 bytes per character and some indexes
-        # Are bigger than 1536 in this case. So for MySQL 5.6 we need to change the encoding
-        # to utf8 (which is an alias to utf8mb3)
-        export MYSQL_ENCODING=utf8
-    fi
-}
-
-# Retrieves version of airflow stored in the production image (used to display the actual
-# Version we use if it was build from PyPI or GitHub
-function get_airflow_version_from_production_image() {
-     docker run --entrypoint /bin/bash "${AIRFLOW_PROD_IMAGE}" -c 'echo "${AIRFLOW_VERSION}"'
-}
-
-function dump_kind_logs() {
-    echo "###########################################################################################"
-    echo "                   Dumping logs from KIND"
-    echo "###########################################################################################"
-
-    FILE_NAME="${1}"
-    kind --name "${KIND_CLUSTER_NAME}" export logs "${FILE_NAME}"
-}
-
-
-function send_kubernetes_logs_to_file_io() {
-    echo "##############################################################################"
-    echo
-    echo "   DUMPING LOG FILES FROM KIND AND SENDING THEM TO file.io"
-    echo
-    echo "##############################################################################"
-    DUMP_DIR_NAME=$(date "+%Y-%m-%d")_kind_${CI_BUILD_ID:="default"}_${CI_JOB_ID:="default"}
-    DUMP_DIR=/tmp/${DUMP_DIR_NAME}
-    dump_kind_logs "${DUMP_DIR}"
-    tar -cvzf "${DUMP_DIR}.tar.gz" -C /tmp "${DUMP_DIR_NAME}"
-    echo
-    echo "   Logs saved to ${DUMP_DIR}.tar.gz"
-    echo
-    echo "##############################################################################"
-    curl -F "file=@${DUMP_DIR}.tar.gz" https://file.io
-}
-
-function check_kind_and_kubectl_are_installed() {
-    SYSTEM=$(uname -s| tr '[:upper:]' '[:lower:]')
-    KIND_VERSION="v0.7.0"
-    KIND_URL="https://github.com/kubernetes-sigs/kind/releases/download/${KIND_VERSION}/kind-${SYSTEM}-amd64"
-    KIND_PATH="${BUILD_CACHE_DIR}/bin/kind"
-    KUBECTL_VERSION="v1.15.3"
-    KUBECTL_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/${SYSTEM}/amd64/kubectl"
-    KUBECTL_PATH="${BUILD_CACHE_DIR}/bin/kubectl"
-    mkdir -pv "${BUILD_CACHE_DIR}/bin"
-    if [[ ! -f "${KIND_PATH}" ]]; then
-        echo
-        echo "Downloading Kind version ${KIND_VERSION}"
-        echo
-        curl --fail --location "${KIND_URL}" --output "${KIND_PATH}"
-        chmod +x "${KIND_PATH}"
-    fi
-    if [[ ! -f "${KUBECTL_PATH}" ]]; then
-        echo
-        echo "Downloading Kubectl version ${KUBECTL_VERSION}"
-        echo
-        curl --fail --location "${KUBECTL_URL}" --output "${KUBECTL_PATH}"
-        chmod +x "${KUBECTL_PATH}"
-    fi
-    PATH=${PATH}:${BUILD_CACHE_DIR}/bin
-}
-
-function create_cluster() {
-    if [[ "${TRAVIS:="false"}" == "true" ]]; then
-        # Travis CI does not handle the nice output of Kind well, so we need to capture it
-        # And display only if kind fails to start
-        start_output_heartbeat "Creating kubernetes cluster" 10
-        set +e
-        if ! OUTPUT=$(kind create cluster \
-                        --name "${KIND_CLUSTER_NAME}" \
-                        --config "${AIRFLOW_SOURCES}/scripts/ci/kubernetes/kind-cluster-conf.yaml" \
-                        --image "kindest/node:${KUBERNETES_VERSION}" 2>&1); then
-            echo "${OUTPUT}"
-        fi
-        stop_output_heartbeat
-    else
-        kind create cluster \
-            --name "${KIND_CLUSTER_NAME}" \
-            --config "${AIRFLOW_SOURCES}/scripts/ci/kubernetes/kind-cluster-conf.yaml" \
-            --image "kindest/node:${KUBERNETES_VERSION}"
-    fi
-    echo
-    echo "Created cluster ${KIND_CLUSTER_NAME}"
-    echo
-
-}
-
-function delete_cluster() {
-    kind delete cluster --name "${KIND_CLUSTER_NAME}"
-    echo
-    echo "Deleted cluster ${KIND_CLUSTER_NAME}"
-    echo
-}
-
-function perform_kind_cluster_operation() {
-    OPERATION="${1}"
-    ALL_CLUSTERS=$(kind get clusters || true)
-
-    echo
-    echo "Kubernetes mode: ${KUBERNETES_MODE}"
-    echo
-
-    if [[ ${OPERATION} == "status" ]]; then
-        if [[ ${ALL_CLUSTERS} == *"${KIND_CLUSTER_NAME}"* ]]; then
-            echo
-            echo "Cluster name: ${KIND_CLUSTER_NAME}"
-            echo
-            kind get nodes --name "${KIND_CLUSTER_NAME}"
-            echo
-            exit
-        else
-            echo
-            echo "Cluster ${KIND_CLUSTER_NAME} is not running"
-            echo
-            exit
-        fi
-    fi
-    if [[ ${ALL_CLUSTERS} == *"${KIND_CLUSTER_NAME}"* ]]; then
-        if [[ ${OPERATION} == "start" ]]; then
-            echo
-            echo "Cluster ${KIND_CLUSTER_NAME} is already created"
-            echo "Reusing previously created cluster"
-            echo
-        elif [[ ${OPERATION} == "restart" ]]; then
-            echo
-            echo "Recreating cluster"
-            echo
-            delete_cluster
-            create_cluster
-        elif [[ ${OPERATION} == "stop" ]]; then
-            echo
-            echo "Deleting cluster"
-            echo
-            delete_cluster
-            exit
-        elif [[ ${OPERATION} == "deploy" ]]; then
-            echo
-            echo "Deploying Airflow to KinD"
-            echo
-            get_ci_environment
-            check_kind_and_kubectl_are_installed
-            build_kubernetes_image
-            load_image_to_kind_cluster
-            prepare_kubernetes_app_variables
-            prepare_kubernetes_resources
-            apply_kubernetes_resources
-            wait_for_airflow_pods_up_and_running
-            wait_for_airflow_webserver_up_and_running
-        elif [[ ${OPERATION} == "test" ]]; then
-            echo
-            echo "Testing with kind to KinD"
-            echo
-            "${AIRFLOW_SOURCES}/scripts/ci/ci_run_kubernetes_tests.sh"
-        else
-            echo
-            echo "Wrong cluster operation: ${OPERATION}. Should be one of:"
-            echo "${FORMATTED_KIND_OPERATIONS}"
-            echo
-            exit 1
-        fi
-    else
-        if [[ ${OPERATION} == "start" ]]; then
-            echo
-            echo "Creating cluster"
-            echo
-            create_cluster
-        elif [[ ${OPERATION} == "recreate" ]]; then
-            echo
-            echo "Cluster ${KIND_CLUSTER_NAME} does not exist. Creating rather than recreating"
-            echo "Creating cluster"
-            echo
-            create_cluster
-        elif [[ ${OPERATION} == "stop" || ${OEPRATON} == "deploy" || ${OPERATION} == "test" ]]; then
-            echo
-            echo "Cluster ${KIND_CLUSTER_NAME} does not exist. It should exist for ${OPERATION} operation"
-            echo
-            exit 1
-        else
-            echo
-            echo "Wrong cluster operation: ${OPERATION}. Should be one of:"
-            echo "${FORMATTED_KIND_OPERATIONS}"
-            echo
-            exit 1
-        fi
-    fi
-}
-
-function check_cluster_ready_for_airflow() {
-    kubectl cluster-info --cluster "${KUBECTL_CLUSTER_NAME}"
-    kubectl get nodes --cluster "${KUBECTL_CLUSTER_NAME}"
-    echo
-    echo "Showing storageClass"
-    echo
-    kubectl get storageclass --cluster "${KUBECTL_CLUSTER_NAME}"
-    echo
-    echo "Showing kube-system pods"
-    echo
-    kubectl get -n kube-system pods --cluster "${KUBECTL_CLUSTER_NAME}"
-    echo
-    echo "Airflow environment on kubernetes is good to go!"
-    echo
-    kubectl create namespace test-namespace --cluster "${KUBECTL_CLUSTER_NAME}"
-}
-
-
-function build_kubernetes_image() {
-    cd "${AIRFLOW_SOURCES}" || exit 1
-    prepare_prod_build
-    if [[ $(docker images -q "${AIRFLOW_PROD_IMAGE}") == "" ||
-            ${FORCE_BUILD_IMAGES:="false"} == "true" ]]; then
-        build_prod_image
-    else
-        echo
-        echo "Skip rebuilding prod image. Use --force-build-images to rebuild prod image."
-        echo
-    fi
-    echo
-    echo "Adding kubernetes-specific scripts to prod image."
-    echo "Building ${AIRFLOW_KUBERNETES_IMAGE} from ${AIRFLOW_PROD_IMAGE} with latest sources."
-    echo
-    docker build \
-        --build-arg AIRFLOW_PROD_IMAGE="${AIRFLOW_PROD_IMAGE}" \
-        --cache-from "${AIRFLOW_PROD_IMAGE}" \
-        --tag="${AIRFLOW_KUBERNETES_IMAGE}" \
-        -f- . << 'EOF'
-    ARG AIRFLOW_PROD_IMAGE
-    FROM ${AIRFLOW_PROD_IMAGE}
-
-    ARG AIRFLOW_SOURCES=/home/airflow/airflow_sources/
-    ENV AIRFLOW_SOURCES=${AIRFLOW_SOURCES}
-
-    USER root
-
-    COPY --chown=airflow:airflow . ${AIRFLOW_SOURCES}
-
-    COPY scripts/ci/kubernetes/docker/airflow-test-env-init-db.sh /tmp/airflow-test-env-init-db.sh
-    COPY scripts/ci/kubernetes/docker/airflow-test-env-init-dags.sh /tmp/airflow-test-env-init-dags.sh
-    COPY scripts/ci/kubernetes/docker/bootstrap.sh /bootstrap.sh
-
-    RUN chmod +x /bootstrap.sh
-
-
-    USER airflow
-
-
-    ENTRYPOINT ["/bootstrap.sh"]
-EOF
-
-    echo "The ${AIRFLOW_KUBERNETES_IMAGE} is prepared for deployment."
-}
-
-function load_image_to_kind_cluster() {
-    echo
-    echo "Loading ${AIRFLOW_KUBERNETES_IMAGE} to ${KIND_CLUSTER_NAME}"
-    echo
-    kind load docker-image --name "${KIND_CLUSTER_NAME}" "${AIRFLOW_KUBERNETES_IMAGE}"
-}
-
-function prepare_kubernetes_app_variables() {
-    echo
-    echo "Preparing kubernetes variables"
-    echo
-    KUBERNETES_APP_DIR="${AIRFLOW_SOURCES}/scripts/ci/kubernetes/app"
-    TEMPLATE_DIRNAME="${KUBERNETES_APP_DIR}/templates"
-    BUILD_DIRNAME="${KUBERNETES_APP_DIR}/build"
-
-    # shellcheck source=common/_image_variables.sh
-    . "${AIRFLOW_SOURCES}/common/_image_variables.sh"
-
-    # Source branch will be set in DockerHub
-    SOURCE_BRANCH=${SOURCE_BRANCH:=${DEFAULT_BRANCH}}
-    BRANCH_NAME=${BRANCH_NAME:=${SOURCE_BRANCH}}
-
-    if [[ ! -d "${BUILD_DIRNAME}" ]]; then
-        mkdir -p "${BUILD_DIRNAME}"
-    fi
-
-    rm -f "${BUILD_DIRNAME}"/*
-    rm -f "${BUILD_DIRNAME}"/*
-
-    if [[ "${KUBERNETES_MODE}" == "image" ]]; then
-        INIT_DAGS_VOLUME_NAME=airflow-dags
-        POD_AIRFLOW_DAGS_VOLUME_NAME=airflow-dags
-        CONFIGMAP_DAGS_FOLDER=/opt/airflow/dags
-        CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT=
-        CONFIGMAP_DAGS_VOLUME_CLAIM=airflow-dags
-    else
-        INIT_DAGS_VOLUME_NAME=airflow-dags-fake
-        POD_AIRFLOW_DAGS_VOLUME_NAME=airflow-dags-git
-        CONFIGMAP_DAGS_FOLDER=/opt/airflow/dags/repo/airflow/example_dags
-        CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT=/opt/airflow/dags
-        CONFIGMAP_DAGS_VOLUME_CLAIM=
-    fi
-
-
-    CONFIGMAP_GIT_REPO=${CI_SOURCE_REPO}
-    CONFIGMAP_BRANCH=${CI_SOURCE_BRANCH}
-}
-
-function prepare_kubernetes_resources() {
-    echo
-    echo "Preparing kubernetes resources"
-    echo
-    if [[ "${KUBERNETES_MODE}" == "image" ]]; then
-        sed -e "s/{{INIT_GIT_SYNC}}//g" \
-            "${TEMPLATE_DIRNAME}/airflow.template.yaml" >"${BUILD_DIRNAME}/airflow.yaml"
-    else
-        sed -e "/{{INIT_GIT_SYNC}}/{r ${TEMPLATE_DIRNAME}/init_git_sync.template.yaml" -e 'd}' \
-            "${TEMPLATE_DIRNAME}/airflow.template.yaml" >"${BUILD_DIRNAME}/airflow.yaml"
-    fi
-    sed -i "s|{{AIRFLOW_KUBERNETES_IMAGE}}|${AIRFLOW_KUBERNETES_IMAGE}|g" "${BUILD_DIRNAME}/airflow.yaml"
-
-    sed -i "s|{{CONFIGMAP_GIT_REPO}}|${CONFIGMAP_GIT_REPO}|g" "${BUILD_DIRNAME}/airflow.yaml"
-    sed -i "s|{{CONFIGMAP_BRANCH}}|${CONFIGMAP_BRANCH}|g" "${BUILD_DIRNAME}/airflow.yaml"
-    sed -i "s|{{INIT_DAGS_VOLUME_NAME}}|${INIT_DAGS_VOLUME_NAME}|g" "${BUILD_DIRNAME}/airflow.yaml"
-    sed -i "s|{{POD_AIRFLOW_DAGS_VOLUME_NAME}}|${POD_AIRFLOW_DAGS_VOLUME_NAME}|g" \
-        "${BUILD_DIRNAME}/airflow.yaml"
-
-    sed "s|{{CONFIGMAP_DAGS_FOLDER}}|${CONFIGMAP_DAGS_FOLDER}|g" \
-        "${TEMPLATE_DIRNAME}/configmaps.template.yaml" >"${BUILD_DIRNAME}/configmaps.yaml"
-    sed -i "s|{{CONFIGMAP_GIT_REPO}}|${CONFIGMAP_GIT_REPO}|g" "${BUILD_DIRNAME}/configmaps.yaml"
-    sed -i "s|{{CONFIGMAP_BRANCH}}|${CONFIGMAP_BRANCH}|g" "${BUILD_DIRNAME}/configmaps.yaml"
-    sed -i "s|{{CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT}}|${CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT}|g" \
-        "${BUILD_DIRNAME}/configmaps.yaml"
-    sed -i "s|{{CONFIGMAP_DAGS_VOLUME_CLAIM}}|${CONFIGMAP_DAGS_VOLUME_CLAIM}|g" \
-        "${BUILD_DIRNAME}/configmaps.yaml"
-    sed -i "s|{{AIRFLOW_KUBERNETES_IMAGE_NAME}}|${AIRFLOW_KUBERNETES_IMAGE_NAME}|g" \
-        "${BUILD_DIRNAME}/configmaps.yaml"
-    sed -i "s|{{AIRFLOW_KUBERNETES_IMAGE_TAG}}|${AIRFLOW_KUBERNETES_IMAGE_TAG}|g" \
-        "${BUILD_DIRNAME}/configmaps.yaml"
-}
-
-function apply_kubernetes_resources() {
-    echo
-    echo "Apply kubernetes resources."
-    echo
-
-
-    kubectl delete -f "${KUBERNETES_APP_DIR}/postgres.yaml" --cluster "${KUBECTL_CLUSTER_NAME}" \
-        2>&1 | grep -v "NotFound" || true
-    kubectl delete -f "${BUILD_DIRNAME}/airflow.yaml" --cluster "${KUBECTL_CLUSTER_NAME}" \
-        2>&1 | grep -v "NotFound" || true
-    kubectl delete -f "${KUBERNETES_APP_DIR}/secrets.yaml" --cluster "${KUBECTL_CLUSTER_NAME}" \
-        2>&1 | grep -v "NotFound" || true
-
-    set -e
-
-    kubectl apply -f "${KUBERNETES_APP_DIR}/secrets.yaml" --cluster "${KUBECTL_CLUSTER_NAME}"
-    kubectl apply -f "${BUILD_DIRNAME}/configmaps.yaml" --cluster "${KUBECTL_CLUSTER_NAME}"
-    kubectl apply -f "${KUBERNETES_APP_DIR}/postgres.yaml" --cluster "${KUBECTL_CLUSTER_NAME}"
-    kubectl apply -f "${KUBERNETES_APP_DIR}/volumes.yaml" --cluster "${KUBECTL_CLUSTER_NAME}"
-    kubectl apply -f "${BUILD_DIRNAME}/airflow.yaml" --cluster "${KUBECTL_CLUSTER_NAME}"
-}
-
-
-function dump_kubernetes_logs() {
-    POD=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' \
-        --cluster "${KUBECTL_CLUSTER_NAME}" | grep airflow | head -1)
-    echo "------- pod description -------"
-    kubectl describe pod "${POD}" --cluster "${KUBECTL_CLUSTER_NAME}"
-    echo "------- webserver init container logs - init -------"
-    kubectl logs "${POD}" -c init --cluster "${KUBECTL_CLUSTER_NAME}" || true
-    if [[ "${KUBERNETES_MODE}" == "git" ]]; then
-        echo "------- webserver init container logs - git-sync-clone -------"
-        kubectl logs "${POD}" -c git-sync-clone --cluster "${KUBECTL_CLUSTER_NAME}" || true
-    fi
-    echo "------- webserver logs -------"
-    kubectl logs "${POD}" -c webserver --cluster "${KUBECTL_CLUSTER_NAME}" || true
-    echo "------- scheduler logs -------"
-    kubectl logs "${POD}" -c scheduler --cluster "${KUBECTL_CLUSTER_NAME}" || true
-    echo "--------------"
-}
-
-function wait_for_airflow_pods_up_and_running() {
-    set +o pipefail
-    # wait for up to 10 minutes for everything to be deployed
-    PODS_ARE_READY="0"
-    for i in {1..150}; do
-        echo "------- Running kubectl get pods: $i -------"
-        PODS=$(kubectl get pods --cluster "${KUBECTL_CLUSTER_NAME}" | awk 'NR>1 {print $0}')
-        echo "$PODS"
-        NUM_AIRFLOW_READY=$(echo "${PODS}" | grep airflow | awk '{print $2}' | grep -cE '([0-9])\/(\1)' \
-            | xargs)
-        NUM_POSTGRES_READY=$(echo "${PODS}" | grep postgres | awk '{print $2}' | grep -cE '([0-9])\/(\1)' \
-            | xargs)
-        if [[ "${NUM_AIRFLOW_READY}" == "1" && "${NUM_POSTGRES_READY}" == "1" ]]; then
-            PODS_ARE_READY="1"
-            break
-        fi
-        sleep 4
-    done
-    POD=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' \
-        --cluster "${KUBECTL_CLUSTER_NAME}" | grep airflow | head -1)
-
-    if [[ "${PODS_ARE_READY}" == "1" ]]; then
-        echo "PODS are ready."
-        set -o pipefail
-    else
-        echo >&2 "PODS are not ready after waiting for a long time. Exiting..."
-        dump_kubernetes_logs
-        exit 1
-    fi
-}
-
-
-function wait_for_airflow_webserver_up_and_running() {
-    set +o pipefail
-    # Wait until Airflow webserver is up
-    KUBERNETES_HOST=localhost
-    AIRFLOW_WEBSERVER_IS_READY="0"
-    CONSECUTIVE_SUCCESS_CALLS=0
-    for i in {1..30}; do
-        echo "------- Wait until webserver is up: $i -------"
-        PODS=$(kubectl get pods --cluster "${KUBECTL_CLUSTER_NAME}" | awk 'NR>1 {print $0}')
-        echo "$PODS"
-        HTTP_CODE=$(curl -LI "http://${KUBERNETES_HOST}:30809/health" -o /dev/null -w '%{http_code}\n' -sS) \
-            || true
-        if [[ "${HTTP_CODE}" == 200 ]]; then
-            ((CONSECUTIVE_SUCCESS_CALLS += 1))
-        else
-            CONSECUTIVE_SUCCESS_CALLS="0"
-        fi
-        if [[ "${CONSECUTIVE_SUCCESS_CALLS}" == 3 ]]; then
-            AIRFLOW_WEBSERVER_IS_READY="1"
-            break
-        fi
-        sleep 10
-    done
-    set -o pipefail
-    if [[ "${AIRFLOW_WEBSERVER_IS_READY}" == "1" ]]; then
-        echo
-        echo "Airflow webserver is ready."
-        echo
-    else
-        echo >&2
-        echo >&2 "Airflow webserver is not ready after waiting for a long time. Exiting..."
-        echo >&2
-        dump_kubernetes_logs
-        exit 1
-    fi
-}
diff --git a/scripts/ci/ci_build_dockerhub.sh b/scripts/ci/ci_build_dockerhub.sh
index 6c68d7e..97f9b8c 100755
--- a/scripts/ci/ci_build_dockerhub.sh
+++ b/scripts/ci/ci_build_dockerhub.sh
@@ -22,15 +22,9 @@
 export FORCE_ANSWER_TO_QUESTIONS="yes"
 export VERBOSE_COMMANDS="true"
 
-if [[ -z ${DOCKER_REPO} ]]; then
-   echo
-   echo "Error! Missing DOCKER_REPO environment variable"
-   echo "Please specify DOCKER_REPO variable following the pattern HOST/DOCKERHUB_USER/DOCKERHUB_REPO"
-   echo
-   exit 1
-else
-   echo "DOCKER_REPO=${DOCKER_REPO}"
-fi
+: "${DOCKER_REPO:?"ERROR: Please specify DOCKER_REPO variable following the pattern HOST/DOCKERHUB_USER/DOCKERHUB_REPO"}"
+
+echo "DOCKER_REPO=${DOCKER_REPO}"
 
 [[ ${DOCKER_REPO:=} =~ [^/]*/([^/]*)/([^/]*) ]] && \
     export DOCKERHUB_USER=${BASH_REMATCH[1]} &&
@@ -41,25 +35,14 @@ echo "DOCKERHUB_USER=${DOCKERHUB_USER}"
 echo "DOCKERHUB_REPO=${DOCKERHUB_REPO}"
 echo
 
-if [[ -z ${DOCKER_TAG:=} ]]; then
-   echo
-   echo "Error! Missing DOCKER_TAG environment variable"
-   echo "Please specify DOCKER_TAG variable following the pattern BRANCH-pythonX.Y[-ci]"
-   echo
-   exit 1
-else
-   echo "DOCKER_TAG=${DOCKER_TAG}"
-fi
+: "${DOCKER_TAG:?"ERROR: Please specify DOCKER_TAG variable following the pattern BRANCH-pythonX.Y[-ci]"}"
+
+echo "DOCKER_TAG=${DOCKER_TAG}"
+
 
 [[ ${DOCKER_TAG:=} =~ ${DEFAULT_BRANCH}-python([0-9.]*)(.*) ]] && export PYTHON_MAJOR_MINOR_VERSION=${BASH_REMATCH[1]}
 
-if [[ -z ${PYTHON_MAJOR_MINOR_VERSION:=} ]]; then
-    echo
-    echo "Error! Wrong DOCKER_TAG"
-    echo "The tag '${DOCKER_TAG}' should follow the pattern ${DEFAULT_BRANCH}-pythonX.Y[-ci]"
-    echo
-    exit 1
-fi
+: "${PYTHON_MAJOR_MINOR_VERSION:?"The tag '${DOCKER_TAG}' should follow the pattern .*-pythonX.Y[-ci]"}"
 
 echo "Detected PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION}"
 echo
diff --git a/scripts/ci/docker-compose/local.yml b/scripts/ci/docker-compose/local.yml
index 793a322..9603417 100644
--- a/scripts/ci/docker-compose/local.yml
+++ b/scripts/ci/docker-compose/local.yml
@@ -24,7 +24,7 @@ services:
     # or those that might be useful to see in the host as output of the
     # tests (such as logs)
     volumes:
-      # START automatically generated volumes from LOCAL_MOUNTS in _utils.sh
+      # START automatically generated volumes from LOCAL_MOUNTS in _local_mounts.sh
       - ../../../.bash_aliases:/root/.bash_aliases:cached
       - ../../../.bash_history:/root/.bash_history:cached
       - ../../../.coveragerc:/opt/airflow/.coveragerc:cached
@@ -55,7 +55,7 @@ services:
       - ../../../tests:/opt/airflow/tests:cached
       - ../../../kubernetes_tests:/opt/airflow/kubernetes_tests:cached
       - ../../../tmp:/opt/airflow/tmp:cached
-      # END automatically generated volumes from LOCAL_MOUNTS in _utils.sh
+      # END automatically generated volumes from LOCAL_MOUNTS in _local_mounts.sh
     environment:
       - HOST_USER_ID
       - HOST_GROUP_ID
diff --git a/scripts/ci/in_container/entrypoint_ci.sh b/scripts/ci/in_container/entrypoint_ci.sh
index 60757fa..2a7535c 100755
--- a/scripts/ci/in_container/entrypoint_ci.sh
+++ b/scripts/ci/in_container/entrypoint_ci.sh
@@ -29,12 +29,7 @@ BACKEND=${BACKEND:=sqlite}
 
 export AIRFLOW_HOME=${AIRFLOW_HOME:=${HOME}}
 
-if [[ -z ${AIRFLOW_SOURCES:=} ]]; then
-    echo >&2
-    echo >&2 AIRFLOW_SOURCES not set !!!!
-    echo >&2
-    exit 1
-fi
+: "${AIRFLOW_SOURCES:?"ERROR: AIRFLOW_SOURCES not set !!!!"}"
 
 echo
 echo "Airflow home: ${AIRFLOW_HOME}"
diff --git a/scripts/ci/libraries/_build_images.sh b/scripts/ci/libraries/_build_images.sh
new file mode 100644
index 0000000..123665b
--- /dev/null
+++ b/scripts/ci/libraries/_build_images.sh
@@ -0,0 +1,771 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+# For remote installation of airflow (from GitHub or Pypi) when building the image, you need to
+# pass build flags depending on the version and method of the installation (for example to
+# get proper requirement constraint files)
+function add_build_args_for_remote_install() {
+    # entrypoint is used as AIRFLOW_SOURCES_FROM/TO in order to avoid costly copying of all sources of
+    # Airflow - those are not needed for remote install at all. Entrypoint is later overwritten by
+    # ENTRYPOINT_FILE - downloaded entrypoint.sh so this is only for the purpose of iteration on Dockerfile
+    EXTRA_DOCKER_PROD_BUILD_FLAGS+=(
+        "--build-arg" "AIRFLOW_SOURCES_FROM=entrypoint.sh"
+        "--build-arg" "AIRFLOW_SOURCES_TO=/entrypoint"
+    )
+    if [[ ${AIRFLOW_VERSION} =~ [^0-9]*1[^0-9]*10[^0-9]([0-9]*) ]]; then
+        # All types of references/versions match this regexp for 1.10 series
+        # for example v1_10_test, 1.10.10, 1.10.9 etc. ${BASH_REMATCH[1]} is the () group matches last
+        # minor digit of version and it's length is 0 for v1_10_test, 1 for 1.10.9 and 2 for 1.10.10+
+        if [[ ${#BASH_REMATCH[1]} == "1" ]]; then
+            # This is only for 1.10.0 - 1.10.9
+            EXTRA_DOCKER_PROD_BUILD_FLAGS+=(
+                "--build-arg" "CONSTRAINT_REQUIREMENTS=https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"
+                "--build-arg" "ENTRYPOINT_FILE=https://raw.githubusercontent.com/apache/airflow/1.10.10/entrypoint.sh"
+            )
+        else
+            EXTRA_DOCKER_PROD_BUILD_FLAGS+=(
+                # For 1.10.10+ and v1-10-test it's ok to use AIRFLOW_VERSION as reference
+                "--build-arg" "CONSTRAINT_REQUIREMENTS=https://raw.githubusercontent.com/apache/airflow/${AIRFLOW_VERSION}/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"
+                "--build-arg" "ENTRYPOINT_FILE=https://raw.githubusercontent.com/apache/airflow/${AIRFLOW_VERSION}/entrypoint.sh"
+            )
+        fi
+    else
+        # For all other (master, 2.0+) we just match ${AIRFLOW_VERSION}
+        EXTRA_DOCKER_PROD_BUILD_FLAGS+=(
+            "--build-arg" "CONSTRAINT_REQUIREMENTS=https://raw.githubusercontent.com/apache/airflow/${AIRFLOW_VERSION}/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"
+            "--build-arg" "ENTRYPOINT_FILE=https://raw.githubusercontent.com/apache/airflow/${AIRFLOW_VERSION}/entrypoint.sh"
+        )
+    fi
+}
+
+# Retrieves version of airflow stored in the production image (used to display the actual
+# Version we use if it was build from PyPI or GitHub
+function get_airflow_version_from_production_image() {
+     docker run --entrypoint /bin/bash "${AIRFLOW_PROD_IMAGE}" -c 'echo "${AIRFLOW_VERSION}"'
+}
+
+# Removes the "Forced answer" (yes/no/quit) given previously, unless you specifically want to remember it.
+#
+# This is the default behaviour of all rebuild scripts to ask independently whether you want to
+# rebuild the image or not. Sometimes however we want to remember answer previously given. For
+# example if you answered "no" to rebuild the image, the assumption is that you do not
+# want to rebuild image also for other rebuilds in the same pre-commit execution.
+#
+# All the pre-commit checks therefore have `export REMEMBER_LAST_ANSWER="true"` set
+# So that in case they are run in a sequence of commits they will not rebuild. Similarly if your most
+# recent answer was "no" and you run `pre-commit run mypy` (for example) it will also reuse the
+# "no" answer given previously. This happens until you run any of the breeze commands or run all
+# precommits `pre-commit run` - then the "LAST_FORCE_ANSWER_FILE" will be removed and you will
+# be asked again.
+function forget_last_answer() {
+    if [[ ${REMEMBER_LAST_ANSWER:="false"} != "true" ]]; then
+        print_info
+        print_info "Forgetting last answer from ${LAST_FORCE_ANSWER_FILE}:"
+        print_info
+        rm -f "${LAST_FORCE_ANSWER_FILE}"
+    else
+        if [[ -f "${LAST_FORCE_ANSWER_FILE}" ]]; then
+            print_info
+            print_info "Still remember last answer from ${LAST_FORCE_ANSWER_FILE}:"
+            print_info "$(cat "${LAST_FORCE_ANSWER_FILE}")"
+            print_info
+        fi
+    fi
+}
+
+# Confirms if hte image should be rebuild and interactively checks it with the user.
+# In case iit needs to be rebuild. It only ask the user if it determines that the rebuild
+# is needed and that the rebuild is not already forced. It asks the user using available terminals
+# So that the script works also from within pre-commit run via git hooks - where stdin is not
+# available - it tries to find usable terminal and ask the user via this terminal.
+function confirm_image_rebuild() {
+    ACTION="rebuild"
+    if [[ ${FORCE_PULL_IMAGES:=} == "true" ]]; then
+        ACTION="pull and rebuild"
+    fi
+    if [[ -f "${LAST_FORCE_ANSWER_FILE}" ]]; then
+        # set variable from last answered response given in the same pre-commit run - so that it can be
+        # answered in teh first pre-commit check (build) and then used in another (pylint/mypy/flake8 etc).
+        # shellcheck disable=SC1090
+        source "${LAST_FORCE_ANSWER_FILE}"
+    fi
+    set +e
+    local RES
+    if [[ ${CI:="false"} == "true" ]]; then
+        print_info
+        print_info "CI environment - forcing rebuild for image ${THE_IMAGE_TYPE}."
+        print_info
+        RES="0"
+    elif [[ -n "${FORCE_ANSWER_TO_QUESTIONS:=""}" ]]; then
+        print_info
+        print_info "Forcing answer '${FORCE_ANSWER_TO_QUESTIONS}'"
+        print_info
+        case "${FORCE_ANSWER_TO_QUESTIONS}" in
+            [yY][eE][sS]|[yY])
+                RES="0" ;;
+            [qQ][uU][iI][tT]|[qQ])
+                RES="2" ;;
+            *)
+                RES="1" ;;
+        esac
+    elif [[ -t 0 ]]; then
+        # Check if this script is run interactively with stdin open and terminal attached
+        "${AIRFLOW_SOURCES}/confirm" "${ACTION} image ${THE_IMAGE_TYPE}-python${PYTHON_MAJOR_MINOR_VERSION}"
+        RES=$?
+    elif [[ ${DETECTED_TERMINAL:=$(tty)} != "not a tty" ]]; then
+        # Make sure to use output of tty rather than stdin/stdout when available - this way confirm
+        # will works also in case of pre-commits (git does not pass stdin/stdout to pre-commit hooks)
+        # shellcheck disable=SC2094
+        "${AIRFLOW_SOURCES}/confirm" "${ACTION} image ${THE_IMAGE_TYPE}-python${PYTHON_MAJOR_MINOR_VERSION}" \
+            <"${DETECTED_TERMINAL}" >"${DETECTED_TERMINAL}"
+        RES=$?
+        export DETECTED_TERMINAL
+    elif [[ -c /dev/tty ]]; then
+        export DETECTED_TERMINAL=/dev/tty
+        # Make sure to use /dev/tty first rather than stdin/stdout when available - this way confirm
+        # will works also in case of pre-commits (git does not pass stdin/stdout to pre-commit hooks)
+        # shellcheck disable=SC2094
+        "${AIRFLOW_SOURCES}/confirm" "${ACTION} image ${THE_IMAGE_TYPE}-python${PYTHON_MAJOR_MINOR_VERSION}" \
+            <"${DETECTED_TERMINAL}" >"${DETECTED_TERMINAL}"
+        RES=$?
+    else
+        print_info
+        print_info "No terminal, no stdin - quitting"
+        print_info
+        # No terminal, no stdin, no force answer - quitting!
+        RES="2"
+    fi
+    set -e
+    if [[ ${RES} == "1" ]]; then
+        print_info
+        print_info "Skipping rebuilding the image ${THE_IMAGE_TYPE}-python${PYTHON_MAJOR_MINOR_VERSION}"
+        print_info
+        export SKIP_REBUILD="true"
+        # Force "no" also to subsequent questions so that if you answer it once, you are not asked
+        # For all other pre-commits and you will continue using the images you already have
+        export FORCE_ANSWER_TO_QUESTIONS="no"
+        echo 'export FORCE_ANSWER_TO_QUESTIONS="no"' > "${LAST_FORCE_ANSWER_FILE}"
+    elif [[ ${RES} == "2" ]]; then
+        echo >&2
+        echo >&2 "ERROR: The ${THE_IMAGE_TYPE} needs to be rebuilt - it is outdated. "
+        echo >&2 "   Make sure you build the images bu running"
+        echo >&2
+        echo >&2 "      ./breeze --python ${PYTHON_MAJOR_MINOR_VERSION}" build-image
+        echo >&2
+        echo >&2 "   If you run it via pre-commit as individual hook, you can run 'pre-commit run build'."
+        echo >&2
+        exit 1
+    else
+        # Force "yes" also to subsequent questions
+        export FORCE_ANSWER_TO_QUESTIONS="yes"
+    fi
+}
+
+# Builds local image manifest
+# It contains only one .json file - result of docker inspect - describing the image
+# We cannot use docker registry APIs as they are available only with authorisation
+# But this image can be pulled without authentication
+function build_ci_image_manifest() {
+    verbose_docker inspect "${AIRFLOW_CI_IMAGE}" > "manifests/${AIRFLOW_CI_BASE_TAG}.json"
+    verbose_docker build \
+    --build-arg AIRFLOW_CI_BASE_TAG="${AIRFLOW_CI_BASE_TAG}" \
+    --tag="${AIRFLOW_CI_LOCAL_MANIFEST_IMAGE}" \
+    -f- . <<EOF
+ARG AIRFLOW_CI_BASE_TAG
+FROM scratch
+
+COPY "manifests/${AIRFLOW_CI_BASE_TAG}.json" .
+
+CMD ""
+EOF
+}
+
+#
+# Retrieves information about layers in the local IMAGE
+# it stores list of SHAs of image layers in the file pointed at by TMP_MANIFEST_LOCAL_SHA
+#
+function get_local_image_info() {
+    TMP_MANIFEST_LOCAL_JSON=$(mktemp)
+    TMP_MANIFEST_LOCAL_SHA=$(mktemp)
+    set +e
+    # Remove the container just in case
+    verbose_docker rm --force "local-airflow-manifest"  >/dev/null 2>&1
+    # Create manifest from the local manifest image
+    if ! verbose_docker create --name "local-airflow-manifest" \
+        "${AIRFLOW_CI_LOCAL_MANIFEST_IMAGE}"  >/dev/null 2>&1 ; then
+        echo
+        echo "Local manifest image not available"
+        echo
+        LOCAL_MANIFEST_IMAGE_UNAVAILABLE="true"
+        return
+    fi
+    set -e
+     # Create manifest from the local manifest image
+    verbose_docker cp "local-airflow-manifest:${AIRFLOW_CI_BASE_TAG}.json" "${TMP_MANIFEST_LOCAL_JSON}" >/dev/null 2>&1
+    sed 's/ *//g' "${TMP_MANIFEST_LOCAL_JSON}" | grep '^"sha256:' >"${TMP_MANIFEST_LOCAL_SHA}"
+    verbose_docker rm --force "local-airflow-manifest" >/dev/null 2>&1
+}
+
+#
+# Retrieves information about layers in the remote IMAGE
+# it stores list of SHAs of image layers in the file pointed at by TMP_MANIFEST_REMOTE_SHA
+# This cannot be done easily with existing APIs of Dockerhub because they require additional authentication
+# even for public images. Therefore instead we are downloading a specially prepared manifest image
+# which is built together with the main image. This special manifest image is prepared during
+# building of the main image and contains single JSON file being result of docker inspect on that image
+# This image is from scratch so it is very tiny
+function get_remote_image_info() {
+    set +e
+    # Pull remote manifest image
+    if ! verbose_docker pull "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}" >/dev/null; then
+        echo
+        echo "Remote docker registry unreachable"
+        echo
+        REMOTE_DOCKER_REGISTRY_UNREACHABLE="true"
+        return
+    fi
+    set -e
+
+    # Docker needs the file passed to --cidfile to not exist, so we can't use mktemp
+    TMP_CONTAINER_ID="remote-airflow-manifest-$$.container_id"
+    FILES_TO_CLEANUP_ON_EXIT+=("$TMP_CONTAINER_ID")
+
+    TMP_MANIFEST_REMOTE_JSON=$(mktemp)
+    TMP_MANIFEST_REMOTE_SHA=$(mktemp)
+    # Create container out of the manifest image without running it
+    verbose_docker create --cidfile "${TMP_CONTAINER_ID}" "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}"
+    # Extract manifest and store it in local file
+    verbose_docker cp "$(cat "${TMP_CONTAINER_ID}"):${AIRFLOW_CI_BASE_TAG}.json" "${TMP_MANIFEST_REMOTE_JSON}"
+    # Filter everything except SHAs of image layers
+    sed 's/ *//g' "${TMP_MANIFEST_REMOTE_JSON}" | grep '^"sha256:' >"${TMP_MANIFEST_REMOTE_SHA}"
+    verbose_docker rm --force "$( cat "${TMP_CONTAINER_ID}")"
+}
+
+# The Number determines the cut-off between local building time and pull + build time.
+# It is a bit experimental and it will have to be kept
+# updated as we keep on changing layers. The cut-off point is at the moment when we do first
+# pip install "https://github.com/apache/airflow/archive/${AIRFLOW_BRANCH}.tar...
+# you can get it via this command:
+# docker history --no-trunc  apache/airflow:master-python3.6-ci | \
+#      grep ^sha256 | grep -n "pip uninstall" | awk 'BEGIN {FS=":"} {print $1 }'
+#
+# This command returns the number of layer in docker history where pip uninstall is called. This is the
+# line that will take a lot of time to run and at this point it's worth to pull the image from repo
+# if there are at least NN changed layers in your docker file, you should pull the image.
+#
+# Note that this only matters if you have any of the important files changed since the last build
+# of your image such as Dockerfile.ci, setup.py etc.
+#
+MAGIC_CUT_OFF_NUMBER_OF_LAYERS=34
+
+# Compares layers from both remote and local image and set FORCE_PULL_IMAGES to true in case
+# More than the last NN layers are different.
+function compare_layers() {
+    NUM_DIFF=$(diff  -y --suppress-common-lines "${TMP_MANIFEST_REMOTE_SHA}" "${TMP_MANIFEST_LOCAL_SHA}" | \
+        wc -l || true)
+    rm -f "${TMP_MANIFEST_REMOTE_JSON}" "${TMP_MANIFEST_REMOTE_SHA}" "${TMP_MANIFEST_LOCAL_JSON}" "${TMP_MANIFEST_LOCAL_SHA}"
+    echo
+    echo "Number of layers different between the local and remote image: ${NUM_DIFF}"
+    echo
+    # This is where setup py is rebuilt - it will usually take a looooot of time to build it, so it is
+    # Better to pull here
+    if (( NUM_DIFF >= MAGIC_CUT_OFF_NUMBER_OF_LAYERS )); then
+        echo
+        echo
+        echo "WARNING! Your image and the dockerhub image differ significantly"
+        echo
+        echo "Forcing pulling the images. It will be faster than rebuilding usually."
+        echo "You can avoid it by setting SKIP_CHECK_REMOTE_IMAGE to true"
+        echo
+        export FORCE_PULL_IMAGES="true"
+    else
+        echo
+        echo "No need to pull the image. Local rebuild will be faster"
+        echo
+    fi
+}
+
+
+# Prints summary of the build parameters
+function print_build_info() {
+    print_info
+    print_info "Airflow ${AIRFLOW_VERSION} Python: ${PYTHON_MAJOR_MINOR_VERSION}. Image description: ${IMAGE_DESCRIPTION}"
+    print_info
+}
+
+# Retrieves CI environment variables needed - depending on the CI system we run it in.
+# We try to be CI - agnostic and our scripts should run the same way on different CI systems
+# (This makes it easy to move between different CI systems)
+# This function maps CI-specific variables into a generic ones (prefixed with CI_) that
+# we used in other scripts
+function get_ci_environment() {
+    export CI_EVENT_TYPE="manual"
+    export CI_TARGET_REPO="apache/airflow"
+    export CI_TARGET_BRANCH="master"
+    export CI_SOURCE_REPO="apache/airflow"
+    export CI_SOURCE_BRANCH="master"
+    export CI_BUILD_ID="default-build-id"
+    export CI_JOB_ID="default-job-id"
+    if [[ ${CI:=} != "true" ]]; then
+        echo
+        echo "This is not a CI environment!. Staying with the defaults"
+        echo
+    else
+        if [[ ${TRAVIS:=} == "true" ]]; then
+            export CI_TARGET_REPO="${TRAVIS_REPO_SLUG}"
+            export CI_TARGET_BRANCH="${TRAVIS_BRANCH}"
+            export CI_BUILD_ID="${TRAVIS_BUILD_ID}"
+            export CI_JOB_ID="${TRAVIS_JOB_ID}"
+            if [[ "${TRAVIS_PULL_REQUEST:=}" == "true" ]]; then
+                export CI_EVENT_TYPE="pull_request"
+                export CI_SOURCE_REPO="${TRAVIS_PULL_REQUEST_SLUG}"
+                export CI_SOURCE_BRANCH="${TRAVIS_PULL_REQUEST_BRANCH}"
+            elif [[ "${TRAVIS_EVENT_TYPE:=}" == "cron" ]]; then
+                export CI_EVENT_TYPE="schedule"
+            else
+                export CI_EVENT_TYPE="push"
+            fi
+        elif [[ ${GITHUB_ACTIONS:=} == "true" ]]; then
+            export CI_TARGET_REPO="${GITHUB_REPOSITORY}"
+            export CI_TARGET_BRANCH="${GITHUB_BASE_REF}"
+            export CI_BUILD_ID="${GITHUB_RUN_ID}"
+            export CI_JOB_ID="${GITHUB_JOB}"
+            if [[ ${GITHUB_EVENT_NAME:=} == "pull_request" ]]; then
+                export CI_EVENT_TYPE="pull_request"
+                # default name of the source repo (assuming it's forked without rename)
+                export SOURCE_AIRFLOW_REPO=${SOURCE_AIRFLOW_REPO:="airflow"}
+                # For Pull Requests it's ambiguous to find the PR and we need to
+                # assume that name of repo is airflow but it could be overridden in case it's not
+                export CI_SOURCE_REPO="${GITHUB_ACTOR}/${SOURCE_AIRFLOW_REPO}"
+                export CI_SOURCE_BRANCH="${GITHUB_HEAD_REF}"
+                BRANCH_EXISTS=$(git ls-remote --heads \
+                    "https://github.com/${CI_SOURCE_REPO}.git" "${CI_SOURCE_BRANCH}")
+                if [[ ${BRANCH_EXISTS} == "" ]]; then
+                    echo
+                    echo "https://github.com/${CI_SOURCE_REPO}.git Branch ${CI_SOURCE_BRANCH} does not exist"
+                    echo
+                    echo
+                    echo "Fallback to https://github.com/${CI_TARGET_REPO}.git Branch ${CI_TARGET_BRANCH}"
+                    echo
+                    # Fallback to the target repository if the repo does not exist
+                    export CI_SOURCE_REPO="${CI_TARGET_REPO}"
+                    export CI_SOURCE_BRANCH="${CI_TARGET_BRANCH}"
+                fi
+            elif [[ ${GITHUB_EVENT_TYPE:=} == "schedule" ]]; then
+                export CI_EVENT_TYPE="schedule"
+            else
+                export CI_EVENT_TYPE="push"
+            fi
+        else
+            echo
+            echo "ERROR! Unknown CI environment. Exiting"
+            exit 1
+        fi
+    fi
+    echo
+    echo "Detected CI build environment"
+    echo
+    echo "CI_EVENT_TYPE=${CI_EVENT_TYPE}"
+    echo "CI_TARGET_REPO=${CI_TARGET_REPO}"
+    echo "CI_TARGET_BRANCH=${CI_TARGET_BRANCH}"
+    echo "CI_SOURCE_REPO=${CI_SOURCE_REPO}"
+    echo "CI_SOURCE_BRANCH=${CI_SOURCE_BRANCH}"
+    echo "CI_BUILD_ID=${CI_BUILD_ID}"
+    echo "CI_JOB_ID=${CI_JOB_ID}"
+    echo
+}
+
+
+# Prepares all variables needed by the CI build. Depending on the configuration used (python version
+# DockerHub user etc. the variables are set so that other functions can use those variables.
+function prepare_ci_build() {
+    export AIRFLOW_CI_BASE_TAG="${DEFAULT_BRANCH}-python${PYTHON_MAJOR_MINOR_VERSION}-ci"
+    export AIRFLOW_CI_LOCAL_MANIFEST_IMAGE="local/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}-manifest"
+    export AIRFLOW_CI_REMOTE_MANIFEST_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}-manifest"
+    export AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}"
+    if [[ ${ENABLE_REGISTRY_CACHE="false"} == "true" ]]; then
+        if [[ ${CACHE_REGISTRY_PASSWORD:=} != "" ]]; then
+            echo "${CACHE_REGISTRY_PASSWORD}" | docker login \
+                --username "${CACHE_REGISTRY_USERNAME}" \
+                --password-stdin \
+                "${CACHE_REGISTRY}"
+        fi
+        export CACHE_IMAGE_PREFIX=${CACHE_IMAGE_PREFX:=${GITHUB_ORGANISATION}/${GITHUB_REPO}}
+        export CACHED_AIRFLOW_CI_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_CI_BASE_TAG}"
+        export CACHED_PYTHON_BASE_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
+    else
+        export CACHED_AIRFLOW_CI_IMAGE=""
+        export CACHED_PYTHON_BASE_IMAGE=""
+    fi
+    export AIRFLOW_BUILD_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}/${AIRFLOW_CI_BASE_TAG}"
+    export AIRFLOW_CI_IMAGE_DEFAULT="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DEFAULT_BRANCH}-ci"
+    export PYTHON_BASE_IMAGE="python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
+    export BUILT_IMAGE_FLAG_FILE="${BUILD_CACHE_DIR}/${BRANCH_NAME}/.built_${PYTHON_MAJOR_MINOR_VERSION}"
+    if [[ "${DEFAULT_PYTHON_MAJOR_MINOR_VERSION}" == "${PYTHON_MAJOR_MINOR_VERSION}" ]]; then
+        export DEFAULT_IMAGE="${AIRFLOW_CI_IMAGE_DEFAULT}"
+    else
+        export DEFAULT_IMAGE=""
+    fi
+    export THE_IMAGE_TYPE="CI"
+    export IMAGE_DESCRIPTION="Airflow CI"
+    export AIRFLOW_CONTAINER_CI_OPTIMISED_BUILD="true"
+    export AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS:="${DEFAULT_CI_EXTRAS}"}"
+    export ADDITIONAL_AIRFLOW_EXTRAS="${ADDITIONAL_AIRFLOW_EXTRAS:=""}"
+    export ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS:=""}"
+    export AIRFLOW_IMAGE="${AIRFLOW_CI_IMAGE}"
+    go_to_airflow_sources
+    fix_group_permissions
+}
+
+
+# Only rebuilds CI image if needed. It checks if the docker image build is needed
+# because any of the important source files (from common/_files_for_rebuild_check.sh) has
+# changed or in any of the edge cases (docker image removed, .build cache removed etc.
+# In case rebuild is needed, it determines (by comparing layers in local and remote image)
+# Whether pull is needed before rebuild.
+function rebuild_ci_image_if_needed() {
+    if [[ ${SKIP_CI_IMAGE_CHECK:="false"} == "true" ]]; then
+        echo
+        echo "Skip checking CI image"
+        echo
+        return
+    fi
+    if [[ -f "${BUILT_IMAGE_FLAG_FILE}" ]]; then
+        print_info
+        print_info "${THE_IMAGE_TYPE} image already built locally."
+        print_info
+    else
+        print_info
+        print_info "${THE_IMAGE_TYPE} image not built locally: pulling and building"
+        print_info
+        export FORCE_PULL_IMAGES="true"
+        export FORCE_BUILD_IMAGES="true"
+    fi
+
+    NEEDS_DOCKER_BUILD="false"
+    check_if_docker_build_is_needed
+    if [[ ${NEEDS_DOCKER_BUILD} == "true" ]]; then
+        if [[ ${SKIP_CHECK_REMOTE_IMAGE:=} != "true" && ${DOCKER_CACHE} == "pulled" ]]; then
+            # Check if remote image is different enough to force pull
+            # This is an optimisation pull vs. build time. When there
+            # are enough changes (specifically after setup.py changes) it is faster to pull
+            # and build the image rather than just build it
+            echo
+            echo "Checking if the remote image needs to be pulled"
+            echo
+            get_remote_image_info
+            if [[ ${REMOTE_DOCKER_REGISTRY_UNREACHABLE:=} != "true" ]]; then
+                get_local_image_info
+                if [[ ${LOCAL_MANIFEST_IMAGE_UNAVAILABLE:=} != "true" ]]; then
+                    compare_layers
+                else
+                    FORCE_PULL_IMAGES="true"
+                fi
+            fi
+        fi
+        SKIP_REBUILD="false"
+        if [[ ${CI:=} != "true" && "${FORCE_BUILD:=}" != "true" ]]; then
+            confirm_image_rebuild
+        fi
+        if [[ ${SKIP_REBUILD} != "true" ]]; then
+            SYSTEM=$(uname -s)
+            if [[ ${SYSTEM} != "Darwin" ]]; then
+                ROOT_FILES_COUNT=$(find "airflow" "tests" -user root | wc -l | xargs)
+                if [[ ${ROOT_FILES_COUNT} != "0" ]]; then
+                    ./scripts/ci/ci_fix_ownership.sh
+                fi
+            fi
+            print_info
+            print_info "Build start: ${THE_IMAGE_TYPE} image."
+            print_info
+            build_ci_image
+            update_all_md5_files
+            build_ci_image_manifest
+            print_info
+            print_info "Build completed: ${THE_IMAGE_TYPE} image."
+            print_info
+        fi
+    else
+        print_info
+        print_info "No need to build - none of the important files changed: ${FILES_FOR_REBUILD_CHECK[*]}"
+        print_info
+    fi
+}
+
+
+# Interactive version of confirming the ci image that is used in pre-commits
+# it displays additional information - what the user should do in order to bring the local images
+# back to state that pre-commit will be happy with
+function rebuild_ci_image_if_needed_and_confirmed() {
+    NEEDS_DOCKER_BUILD="false"
+    THE_IMAGE_TYPE="CI"
+
+    check_if_docker_build_is_needed
+
+    if [[ ${NEEDS_DOCKER_BUILD} == "true" ]]; then
+        print_info
+        print_info "Docker image build is needed!"
+        print_info
+    else
+        print_info
+        print_info "Docker image build is not needed!"
+        print_info
+    fi
+
+    if [[ "${NEEDS_DOCKER_BUILD}" == "true" ]]; then
+        echo
+        echo "Some of your images need to be rebuild because important files (like package list) has changed."
+        echo
+        echo "You have those options:"
+        echo "   * Rebuild the images now by answering 'y' (this might take some time!)"
+        echo "   * Skip rebuilding the images and hope changes are not big (you will be asked again)"
+        echo "   * Quit and manually rebuild the images using one of the following commands"
+        echo "        * ./breeze build-image"
+        echo "        * ./breeze build-image --force-pull-images"
+        echo
+        echo "   The first command works incrementally from your last local build."
+        echo "   The second command you use if you want to completely refresh your images from dockerhub."
+        echo
+        SKIP_REBUILD="false"
+        confirm_image_rebuild
+
+        if [[ ${SKIP_REBUILD} != "true" ]]; then
+            rebuild_ci_image_if_needed
+        fi
+    fi
+}
+
+
+# Builds the CI image in the CI environment.
+# Depending on the type of build (push/pr/scheduled) it will either build it incrementally or
+# from the scratch without cache (the latter for scheduled builds only)
+function build_ci_image_on_ci() {
+    get_ci_environment
+
+    # In case of CRON jobs we run builds without cache and upgrade to latest requirements
+    if [[ "${CI_EVENT_TYPE:=}" == "schedule" ]]; then
+        echo
+        echo "Disabling cache for scheduled jobs"
+        echo
+        export DOCKER_CACHE="no-cache"
+        echo
+        echo "Requirements are upgraded to latest while running Docker build"
+        echo
+        export UPGRADE_TO_LATEST_REQUIREMENTS="true"
+    fi
+
+    prepare_ci_build
+
+    rm -rf "${BUILD_CACHE_DIR}"
+    mkdir -pv "${BUILD_CACHE_DIR}"
+
+    rebuild_ci_image_if_needed
+
+    # Disable force pulling forced above this is needed for the subsequent scripts so that
+    # They do not try to pull/build images again
+    unset FORCE_PULL_IMAGES
+    unset FORCE_BUILD
+}
+
+# Builds CI image - depending on the caching strategy (pulled, local, no-cache) it
+# passes the necessary docker build flags via DOCKER_CACHE_CI_DIRECTIVE array
+# it also passes the right Build args depending on the configuration of the build
+# selected by Breeze flags or environment variables.
+function build_ci_image() {
+    print_build_info
+    if [[ -n ${DETECTED_TERMINAL:=""} ]]; then
+        echo -n "Preparing ${AIRFLOW_CI_IMAGE}.
+        " > "${DETECTED_TERMINAL}"
+        spin "${OUTPUT_LOG}" &
+        SPIN_PID=$!
+        # shellcheck disable=SC2064
+        trap "kill ${SPIN_PID}" SIGINT SIGTERM
+    fi
+    pull_ci_image_if_needed
+
+    if [[ "${DOCKER_CACHE}" == "no-cache" ]]; then
+        export DOCKER_CACHE_CI_DIRECTIVE=("--no-cache")
+    elif [[ "${DOCKER_CACHE}" == "local" ]]; then
+        export DOCKER_CACHE_CI_DIRECTIVE=()
+    elif [[ "${DOCKER_CACHE}" == "pulled" ]]; then
+        export DOCKER_CACHE_CI_DIRECTIVE=(
+            "--cache-from" "${AIRFLOW_CI_IMAGE}"
+        )
+    else
+        echo >&2
+        echo >&2 "Error - thee ${DOCKER_CACHE} cache is unknown!"
+        echo >&2
+        exit 1
+    fi
+    if [[ -n ${DETECTED_TERMINAL:=""} ]]; then
+        echo -n "
+Docker building ${AIRFLOW_CI_IMAGE}.
+" > "${DETECTED_TERMINAL}"
+    fi
+    set +u
+    verbose_docker build \
+        --build-arg PYTHON_BASE_IMAGE="${PYTHON_BASE_IMAGE}" \
+        --build-arg PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION}" \
+            --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
+        --build-arg AIRFLOW_BRANCH="${BRANCH_NAME}" \
+        --build-arg AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS}" \
+        --build-arg ADDITIONAL_AIRFLOW_EXTRAS="${ADDITIONAL_AIRFLOW_EXTRAS}" \
+        --build-arg ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS}" \
+        --build-arg AIRFLOW_CONTAINER_CI_OPTIMISED_BUILD="${AIRFLOW_CONTAINER_CI_OPTIMISED_BUILD}" \
+        --build-arg UPGRADE_TO_LATEST_REQUIREMENTS="${UPGRADE_TO_LATEST_REQUIREMENTS}" \
+        "${DOCKER_CACHE_CI_DIRECTIVE[@]}" \
+        -t "${AIRFLOW_CI_IMAGE}" \
+        --target "main" \
+        . -f Dockerfile.ci | tee -a "${OUTPUT_LOG}"
+    set -u
+    if [[ -n "${DEFAULT_IMAGE:=}" ]]; then
+        verbose_docker tag "${AIRFLOW_CI_IMAGE}" "${DEFAULT_IMAGE}" | tee -a "${OUTPUT_LOG}"
+    fi
+    if [[ -n ${SPIN_PID:=""} ]]; then
+        kill "${SPIN_PID}" || true
+        wait "${SPIN_PID}" || true
+        echo > "${DETECTED_TERMINAL}"
+    fi
+}
+
+# Prepares all variables needed by the CI build. Depending on the configuration used (python version
+# DockerHub user etc. the variables are set so that other functions can use those variables.
+function prepare_prod_build() {
+    export AIRFLOW_PROD_BASE_TAG="${DEFAULT_BRANCH}-python${PYTHON_MAJOR_MINOR_VERSION}"
+    export AIRFLOW_PROD_BUILD_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_PROD_BASE_TAG}-build"
+    export AIRFLOW_PROD_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_PROD_BASE_TAG}"
+    export AIRFLOW_PROD_IMAGE_DEFAULT="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DEFAULT_BRANCH}"
+    export PYTHON_BASE_IMAGE="python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
+    if [[ "${DEFAULT_PYTHON_MAJOR_MINOR_VERSION}" == "${PYTHON_MAJOR_MINOR_VERSION}" ]]; then
+        export DEFAULT_IMAGE="${AIRFLOW_PROD_IMAGE_DEFAULT}"
+    else
+        export DEFAULT_IMAGE=""
+    fi
+    export THE_IMAGE_TYPE="PROD"
+    export IMAGE_DESCRIPTION="Airflow production"
+    export AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS:="${DEFAULT_PROD_EXTRAS}"}"
+    export ADDITIONAL_AIRFLOW_EXTRAS="${ADDITIONAL_AIRFLOW_EXTRAS:=""}"
+    export ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS:=""}"
+    export AIRFLOW_IMAGE="${AIRFLOW_PROD_IMAGE}"
+
+    if [[ ${ENABLE_REGISTRY_CACHE="false"} == "true" ]]; then
+        if [[ ${CACHE_REGISTRY_PASSWORD:=} != "" ]]; then
+            echo "${CACHE_REGISTRY_PASSWORD}" | docker login \
+                --username "${CACHE_REGISTRY_USERNAME}" \
+                --password-stdin \
+                "${CACHE_REGISTRY}"
+        fi
+        export CACHE_IMAGE_PREFIX=${CACHE_IMAGE_PREFX:=${GITHUB_ORGANISATION}/${GITHUB_REPO}}
+        export CACHED_AIRFLOW_PROD_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_PROD_BASE_TAG}"
+        export CACHED_AIRFLOW_PROD_BUILD_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_PROD_BASE_TAG}-build"
+        export CACHED_PYTHON_BASE_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
+    else
+        export CACHED_AIRFLOW_PROD_IMAGE=""
+        export CACHED_AIRFLOW_PROD_BUILD_IMAGE=""
+        export CACHED_PYTHON_BASE_IMAGE=""
+    fi
+    export AIRFLOW_KUBERNETES_IMAGE=${AIRFLOW_PROD_IMAGE}-kubernetes
+    AIRFLOW_KUBERNETES_IMAGE_NAME=$(echo "${AIRFLOW_KUBERNETES_IMAGE}" | cut -f 1 -d ":")
+    export AIRFLOW_KUBERNETES_IMAGE_NAME
+    AIRFLOW_KUBERNETES_IMAGE_TAG=$(echo "${AIRFLOW_KUBERNETES_IMAGE}" | cut -f 2 -d ":")
+    export AIRFLOW_KUBERNETES_IMAGE_TAG
+
+    if [[ "${INSTALL_AIRFLOW_REFERENCE:=}" != "" ]]; then
+        # When --install-airflow-reference is used then the image is build from github tag
+        EXTRA_DOCKER_PROD_BUILD_FLAGS=(
+            "--build-arg" "AIRFLOW_INSTALL_SOURCES=https://github.com/apache/airflow/archive/${INSTALL_AIRFLOW_REFERENCE}.tar.gz#egg=apache-airflow"
+        )
+        export AIRFLOW_VERSION="${INSTALL_AIRFLOW_REFERENCE}"
+        add_build_args_for_remote_install
+    elif [[ "${INSTALL_AIRFLOW_VERSION:=}" != "" ]]; then
+        # When --install-airflow-version is used then the image is build from PIP package
+        EXTRA_DOCKER_PROD_BUILD_FLAGS=(
+            "--build-arg" "AIRFLOW_INSTALL_SOURCES=apache-airflow"
+            "--build-arg" "AIRFLOW_INSTALL_VERSION===${INSTALL_AIRFLOW_VERSION}"
+        )
+        export AIRFLOW_VERSION="${INSTALL_AIRFLOW_VERSION}"
+        add_build_args_for_remote_install
+    else
+        # When no airflow version/reference is specified, production image is built from local sources
+        EXTRA_DOCKER_PROD_BUILD_FLAGS=(
+        )
+    fi
+    go_to_airflow_sources
+}
+
+
+# Builds PROD image - depending on the caching strategy (pulled, local, no-cache) it
+# passes the necessary docker build flags via DOCKER_CACHE_PROD_DIRECTIVE and
+# DOCKER_CACHE_PROD_BUILD_DIRECTIVE (separate caching options are needed for "build" segment of the image)
+# it also passes the right Build args depending on the configuration of the build
+# selected by Breeze flags or environment variables.
+function build_prod_image() {
+    print_build_info
+    pull_prod_images_if_needed
+
+    if [[ "${DOCKER_CACHE}" == "no-cache" ]]; then
+        export DOCKER_CACHE_PROD_DIRECTIVE=("--cache-from" "${AIRFLOW_PROD_BUILD_IMAGE}")
+        export DOCKER_CACHE_PROD_BUILD_DIRECTIVE=("--no-cache")
+    elif [[ "${DOCKER_CACHE}" == "local" ]]; then
+        export DOCKER_CACHE_PROD_DIRECTIVE=()
+        export DOCKER_CACHE_PROD_BUILD_DIRECTIVE=()
+    elif [[ "${DOCKER_CACHE}" == "pulled" ]]; then
+        export DOCKER_CACHE_PROD_DIRECTIVE=(
+            "--cache-from" "${AIRFLOW_PROD_BUILD_IMAGE}"
+            "--cache-from" "${AIRFLOW_PROD_IMAGE}"
+        )
+        export DOCKER_CACHE_PROD_BUILD_DIRECTIVE=(
+            "--cache-from" "${AIRFLOW_PROD_BUILD_IMAGE}"
+        )
+    else
+        echo >&2
+        echo >&2 "Error - thee ${DOCKER_CACHE} cache is unknown!"
+        echo >&2
+        exit 1
+    fi
+    set +u
+    verbose_docker build \
+        "${EXTRA_DOCKER_PROD_BUILD_FLAGS[@]}" \
+        --build-arg PYTHON_BASE_IMAGE="${PYTHON_BASE_IMAGE}" \
+        --build-arg PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION}" \
+        --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
+        --build-arg AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS}" \
+        --build-arg ADDITIONAL_AIRFLOW_EXTRAS="${ADDITIONAL_AIRFLOW_EXTRAS}" \
+        --build-arg ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS}" \
+        "${DOCKER_CACHE_PROD_BUILD_DIRECTIVE[@]}" \
+        -t "${AIRFLOW_PROD_BUILD_IMAGE}" \
+        --target "airflow-build-image" \
+        . -f Dockerfile | tee -a "${OUTPUT_LOG}"
+    verbose_docker build \
+        "${EXTRA_DOCKER_PROD_BUILD_FLAGS[@]}" \
+        --build-arg PYTHON_BASE_IMAGE="${PYTHON_BASE_IMAGE}" \
+        --build-arg PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION}" \
+        --build-arg ADDITIONAL_AIRFLOW_EXTRAS="${ADDITIONAL_AIRFLOW_EXTRAS}" \
+        --build-arg ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS}" \
+        --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
+        --build-arg AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS}" \
+        "${DOCKER_CACHE_PROD_DIRECTIVE[@]}" \
+        -t "${AIRFLOW_PROD_IMAGE}" \
+        --target "main" \
+        . -f Dockerfile | tee -a "${OUTPUT_LOG}"
+    set -u
+    if [[ -n "${DEFAULT_IMAGE:=}" ]]; then
+        verbose_docker tag "${AIRFLOW_PROD_IMAGE}" "${DEFAULT_IMAGE}" | tee -a "${OUTPUT_LOG}"
+    fi
+}
diff --git a/scripts/ci/libraries/_initialization.sh b/scripts/ci/libraries/_initialization.sh
new file mode 100644
index 0000000..9f86ba4
--- /dev/null
+++ b/scripts/ci/libraries/_initialization.sh
@@ -0,0 +1,228 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Common environment that is initialized by both Breeze and CI scripts
+function initialize_common_environment {
+    # default python Major/Minor version
+    PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:="3.6"}
+
+    # extra flags passed to docker run for CI image
+    # shellcheck disable=SC2034
+    EXTRA_DOCKER_FLAGS=()
+
+    # extra flags passed to docker run for PROD image
+    # shellcheck disable=SC2034
+    EXTRA_DOCKER_PROD_BUILD_FLAGS=()
+
+    # files that should be cleaned up when the script exits
+    # shellcheck disable=SC2034
+    FILES_TO_CLEANUP_ON_EXIT=()
+
+    # Sets to where airflow sources are located
+    AIRFLOW_SOURCES=${AIRFLOW_SOURCES:=$(cd "${MY_DIR}/../../" && pwd)}
+    export AIRFLOW_SOURCES
+
+    # Sets to the build cache directory - status of build and convenience scripts are stored there
+    BUILD_CACHE_DIR="${AIRFLOW_SOURCES}/.build"
+    export BUILD_CACHE_DIR
+
+    # File to keep the last forced answer. This is useful for pre-commits where you need to
+    # only answer once if the image should be rebuilt or not and your answer is used for
+    # All the subsequent questions
+    export LAST_FORCE_ANSWER_FILE="${BUILD_CACHE_DIR}/last_force_answer.sh"
+
+    # Create useful directories if not yet created
+    mkdir -p "${AIRFLOW_SOURCES}/.mypy_cache"
+    mkdir -p "${AIRFLOW_SOURCES}/logs"
+    mkdir -p "${AIRFLOW_SOURCES}/tmp"
+    mkdir -p "${AIRFLOW_SOURCES}/files"
+    mkdir -p "${AIRFLOW_SOURCES}/dist"
+
+    # Read common values used across Breeze and CI scripts
+    # shellcheck source=common/_common_values.sh
+    . "${AIRFLOW_SOURCES}/common/_common_values.sh"
+    # Read image-specific values used across Breeze and CI scripts
+    # shellcheck source=common/_image_variables.sh
+    . "${AIRFLOW_SOURCES}/common/_image_variables.sh"
+    # Read information about files that are checked if image should be rebuilt
+    # shellcheck source=common/_files_for_rebuild_check.sh
+    . "${AIRFLOW_SOURCES}/common/_files_for_rebuild_check.sh"
+
+    # Default branch name for triggered builds is the one configured in default branch
+    export BRANCH_NAME=${BRANCH_NAME:=${DEFAULT_BRANCH}}
+
+    export GITHUB_ORGANISATION=${GITHUB_ORGANISATION:="apache"}
+    export GITHUB_REPO=${GITHUB_REPO:="airflow"}
+    export CACHE_REGISTRY=${CACHE_REGISTRY:="docker.pkg.github.com"}
+    export ENABLE_REGISTRY_CACHE=${ENABLE_REGISTRY_CACHE:="false"}
+
+    # Default port numbers for forwarded ports
+    export WEBSERVER_HOST_PORT=${WEBSERVER_HOST_PORT:="28080"}
+    export POSTGRES_HOST_PORT=${POSTGRES_HOST_PORT:="25433"}
+    export MYSQL_HOST_PORT=${MYSQL_HOST_PORT:="23306"}
+
+    # Default MySQL/Postgres versions
+    export POSTGRES_VERSION=${POSTGRES_VERSION:="9.6"}
+    export MYSQL_VERSION=${MYSQL_VERSION:="5.7"}
+
+    # Whether base python images should be pulled from cache
+    export PULL_PYTHON_BASE_IMAGES_FROM_CACHE=${PULL_PYTHON_BASE_IMAGES_FROM_CACHE:="true"}
+
+    # Disable writing .pyc files - slightly slower imports but not messing around when switching
+    # Python version and avoids problems with root-owned .pyc files in host
+    export PYTHONDONTWRITEBYTECODE=${PYTHONDONTWRITEBYTECODE:="true"}
+
+    # By default we assume the kubernetes cluster is not being started
+    export ENABLE_KIND_CLUSTER=${ENABLE_KIND_CLUSTER:="false"}
+    #
+    # Sets mounting of host volumes to container for static checks
+    # unless MOUNT_HOST_AIRFLOW_VOLUME is not true
+    #
+    MOUNT_HOST_AIRFLOW_VOLUME=${MOUNT_HOST_AIRFLOW_VOLUME:="true"}
+    export MOUNT_HOST_AIRFLOW_VOLUME
+
+    # If this variable is set, we mount the whole sources directory to the host rather than
+    # selected volumes. This is needed to check ALL source files during licence check
+    # for example
+    MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS=${MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS="false"}
+    export MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS
+
+    # Set host user id to current user. This is used to set the ownership properly when exiting
+    # The container on Linux - all files created inside docker are created with root user
+    # but they should be restored back to the host user
+    HOST_USER_ID="$(id -ur)"
+    export HOST_USER_ID
+
+    # Set host group id to current group This is used to set the ownership properly when exiting
+    # The container on Linux - all files created inside docker are created with root user
+    # but they should be restored back to the host user
+    HOST_GROUP_ID="$(id -gr)"
+    export HOST_GROUP_ID
+
+    # Add the right volume mount for sources, depending which mount strategy is used
+    if [[ ${MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS} == "true" ]]; then
+        print_info
+        print_info "Mount whole airflow source directory for static checks"
+        print_info
+        EXTRA_DOCKER_FLAGS=( \
+          "-v" "${AIRFLOW_SOURCES}:/opt/airflow" \
+          "--env" "PYTHONDONTWRITEBYTECODE" \
+        )
+    elif [[ ${MOUNT_HOST_AIRFLOW_VOLUME} == "true" ]]; then
+        print_info
+        print_info "Mounting necessary host volumes to Docker"
+        print_info
+
+        read -r -a EXTRA_DOCKER_FLAGS <<< "$(convert_local_mounts_to_docker_params)"
+    else
+        print_info
+        print_info "Skip mounting host volumes to Docker"
+        print_info
+        EXTRA_DOCKER_FLAGS=( \
+            "--env" "PYTHONDONTWRITEBYTECODE" \
+        )
+    fi
+
+    # In case of the CI build get environment variables from codecov.io and
+    # set it as the extra docker flags. As described in https://docs.codecov.io/docs/testing-with-docker
+    if [[ ${CI:=} == "true" ]]; then
+        CI_CODECOV_ENV="$(bash <(curl -s https://codecov.io/env))"
+        for ENV_PARAM in ${CI_CODECOV_ENV}
+        do
+            EXTRA_DOCKER_FLAGS+=("${ENV_PARAM}")
+        done
+    fi
+
+    # We use pulled docker image cache by default to speed up the builds
+    # but we can also set different docker caching strategy (for example we can use local cache
+    # to build the images in case we iterate with the image
+    export DOCKER_CACHE=${DOCKER_CACHE:="pulled"}
+
+    # By default we are not upgrading to latest requirements when building Docker CI image
+    # This will only be done in cron jobs
+    export UPGRADE_TO_LATEST_REQUIREMENTS=${UPGRADE_TO_LATEST_REQUIREMENTS:="false"}
+
+    # In case of MacOS we need to use gstat - gnu version of the stats
+    export STAT_BIN=stat
+    if [[ "${OSTYPE}" == "darwin"* ]]; then
+        export STAT_BIN=gstat
+    fi
+
+    # Read airflow version from the version.py
+    AIRFLOW_VERSION=$(grep version "airflow/version.py" | awk '{print $3}' | sed "s/['+]//g")
+    export AIRFLOW_VERSION
+
+    # default version of python used to tag the "master" and "latest" images in DockerHub
+    export DEFAULT_PYTHON_MAJOR_MINOR_VERSION=3.6
+
+    # In case we are not in CI - we assume we run locally. There are subtle changes if you run
+    # CI scripts locally - for example requirements are eagerly updated if you do local run
+    # in generate requirements
+    if [[ ${CI:="false"} == "true" ]]; then
+        export LOCAL_RUN="false"
+    else
+        export LOCAL_RUN="true"
+    fi
+
+    # eager upgrade while generating requirements should only happen in locally run
+    # pre-commits or in cron job
+    if [[ ${LOCAL_RUN} == "true" ]]; then
+        export UPGRADE_WHILE_GENERATING_REQUIREMENTS="true"
+    else
+        export UPGRADE_WHILE_GENERATING_REQUIREMENTS=${UPGRADE_WHILE_GENERATING_REQUIREMENTS:="false"}
+    fi
+
+    # Default extras used for building CI image
+    export DEFAULT_CI_EXTRAS="devel_ci"
+
+    # Default extras used for building Production image. The master of this information is in the Dockerfile
+    DEFAULT_PROD_EXTRAS=$(grep "ARG AIRFLOW_EXTRAS=" "${AIRFLOW_SOURCES}/Dockerfile"|
+            awk 'BEGIN { FS="=" } { print $2 }' | tr -d '"')
+    export DEFAULT_PROD_EXTRAS
+
+    # By default we build CI images  but when we specify `--production-image` we switch to production image
+    export PRODUCTION_IMAGE="false"
+
+    # The SQLlite URL used for sqlite runs
+    export SQLITE_URL="sqlite:////root/airflow/airflow.db"
+
+    # Determines if airflow should be installed from a specified reference in GitHub
+    export INSTALL_AIRFLOW_REFERENCE=""
+
+    # Version of Kubernetes to run
+    export KUBERNETES_VERSION="${KUBERNETES_VERSION:="v1.15.3"}"
+
+    # Name of the KinD cluster to connect to
+    export KIND_CLUSTER_NAME=${KIND_CLUSTER_NAME:="airflow-python-${PYTHON_MAJOR_MINOR_VERSION}-${KUBERNETES_VERSION}"}
+
+    # Name of the KinD cluster to connect to when referred to via kubectl
+    export KUBECTL_CLUSTER_NAME=kind-${KIND_CLUSTER_NAME}
+
+}
+
+function set_mysql_encoding() {
+    export MYSQL_ENCODING="utf8mb4"
+
+    if [[ ${MYSQL_VERSION:="5.7"} == "5.6" ]]; then
+        # In MySQL 5.6 maximum size of the index is 1536 bytes and it is too small for
+        # Airflow for utf8mb4 encoding (it takes 4 bytes per character and some indexes
+        # Are bigger than 1536 in this case. So for MySQL 5.6 we need to change the encoding
+        # to utf8 (which is an alias to utf8mb3)
+        export MYSQL_ENCODING=utf8
+    fi
+}
diff --git a/scripts/ci/libraries/_kind.sh b/scripts/ci/libraries/_kind.sh
new file mode 100644
index 0000000..22cb652
--- /dev/null
+++ b/scripts/ci/libraries/_kind.sh
@@ -0,0 +1,450 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+function dump_kind_logs() {
+    echo "###########################################################################################"
+    echo "                   Dumping logs from KIND"
+    echo "###########################################################################################"
+
+    FILE_NAME="${1}"
+    kind --name "${KIND_CLUSTER_NAME}" export logs "${FILE_NAME}"
+}
+
+function send_kubernetes_logs_to_file_io() {
+    echo "##############################################################################"
+    echo
+    echo "   DUMPING LOG FILES FROM KIND AND SENDING THEM TO file.io"
+    echo
+    echo "##############################################################################"
+    DUMP_DIR_NAME=$(date "+%Y-%m-%d")_kind_${CI_BUILD_ID:="default"}_${CI_JOB_ID:="default"}
+    DUMP_DIR=/tmp/${DUMP_DIR_NAME}
+    dump_kind_logs "${DUMP_DIR}"
+    tar -cvzf "${DUMP_DIR}.tar.gz" -C /tmp "${DUMP_DIR_NAME}"
+    echo
+    echo "   Logs saved to ${DUMP_DIR}.tar.gz"
+    echo
+    echo "##############################################################################"
+    curl -F "file=@${DUMP_DIR}.tar.gz" https://file.io
+}
+
+function check_kind_and_kubectl_are_installed() {
+    SYSTEM=$(uname -s| tr '[:upper:]' '[:lower:]')
+    KIND_VERSION="v0.7.0"
+    KIND_URL="https://github.com/kubernetes-sigs/kind/releases/download/${KIND_VERSION}/kind-${SYSTEM}-amd64"
+    KIND_PATH="${BUILD_CACHE_DIR}/bin/kind"
+    KUBECTL_VERSION="v1.15.3"
+    KUBECTL_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/${SYSTEM}/amd64/kubectl"
+    KUBECTL_PATH="${BUILD_CACHE_DIR}/bin/kubectl"
+    mkdir -pv "${BUILD_CACHE_DIR}/bin"
+    if [[ ! -f "${KIND_PATH}" ]]; then
+        echo
+        echo "Downloading Kind version ${KIND_VERSION}"
+        echo
+        curl --fail --location "${KIND_URL}" --output "${KIND_PATH}"
+        chmod +x "${KIND_PATH}"
+    fi
+    if [[ ! -f "${KUBECTL_PATH}" ]]; then
+        echo
+        echo "Downloading Kubectl version ${KUBECTL_VERSION}"
+        echo
+        curl --fail --location "${KUBECTL_URL}" --output "${KUBECTL_PATH}"
+        chmod +x "${KUBECTL_PATH}"
+    fi
+    PATH=${PATH}:${BUILD_CACHE_DIR}/bin
+}
+
+function create_cluster() {
+    if [[ "${TRAVIS:="false"}" == "true" ]]; then
+        # Travis CI does not handle the nice output of Kind well, so we need to capture it
+        # And display only if kind fails to start
+        start_output_heartbeat "Creating kubernetes cluster" 10
+        set +e
+        if ! OUTPUT=$(kind create cluster \
+                        --name "${KIND_CLUSTER_NAME}" \
+                        --config "${AIRFLOW_SOURCES}/scripts/ci/kubernetes/kind-cluster-conf.yaml" \
+                        --image "kindest/node:${KUBERNETES_VERSION}" 2>&1); then
+            echo "${OUTPUT}"
+        fi
+        stop_output_heartbeat
+    else
+        kind create cluster \
+            --name "${KIND_CLUSTER_NAME}" \
+            --config "${AIRFLOW_SOURCES}/scripts/ci/kubernetes/kind-cluster-conf.yaml" \
+            --image "kindest/node:${KUBERNETES_VERSION}"
+    fi
+    echo
+    echo "Created cluster ${KIND_CLUSTER_NAME}"
+    echo
+
+}
+
+function delete_cluster() {
+    kind delete cluster --name "${KIND_CLUSTER_NAME}"
+    echo
+    echo "Deleted cluster ${KIND_CLUSTER_NAME}"
+    echo
+    rm -rf "${HOME}/.kube/*"
+}
+
+function perform_kind_cluster_operation() {
+    OPERATION="${1}"
+    ALL_CLUSTERS=$(kind get clusters || true)
+
+    echo
+    echo "Kubernetes mode: ${KUBERNETES_MODE}"
+    echo
+
+    if [[ ${OPERATION} == "status" ]]; then
+        if [[ ${ALL_CLUSTERS} == *"${KIND_CLUSTER_NAME}"* ]]; then
+            echo
+            echo "Cluster name: ${KIND_CLUSTER_NAME}"
+            echo
+            kind get nodes --name "${KIND_CLUSTER_NAME}"
+            echo
+            exit
+        else
+            echo
+            echo "Cluster ${KIND_CLUSTER_NAME} is not running"
+            echo
+            exit
+        fi
+    fi
+    if [[ ${ALL_CLUSTERS} == *"${KIND_CLUSTER_NAME}"* ]]; then
+        if [[ ${OPERATION} == "start" ]]; then
+            echo
+            echo "Cluster ${KIND_CLUSTER_NAME} is already created"
+            echo "Reusing previously created cluster"
+            echo
+        elif [[ ${OPERATION} == "restart" ]]; then
+            echo
+            echo "Recreating cluster"
+            echo
+            delete_cluster
+            create_cluster
+        elif [[ ${OPERATION} == "stop" ]]; then
+            echo
+            echo "Deleting cluster"
+            echo
+            delete_cluster
+            exit
+        elif [[ ${OPERATION} == "deploy" ]]; then
+            echo
+            echo "Deploying Airflow to KinD"
+            echo
+            get_ci_environment
+            check_kind_and_kubectl_are_installed
+            build_kubernetes_image
+            load_image_to_kind_cluster
+            prepare_kubernetes_app_variables
+            prepare_kubernetes_resources
+            apply_kubernetes_resources
+            wait_for_airflow_pods_up_and_running
+            wait_for_airflow_webserver_up_and_running
+        elif [[ ${OPERATION} == "test" ]]; then
+            echo
+            echo "Testing with kind to KinD"
+            echo
+            "${AIRFLOW_SOURCES}/scripts/ci/ci_run_kubernetes_tests.sh"
+        else
+            echo
+            echo "Wrong cluster operation: ${OPERATION}. Should be one of:"
+            echo "${FORMATTED_KIND_OPERATIONS}"
+            echo
+            exit 1
+        fi
+    else
+        if [[ ${OPERATION} == "start" ]]; then
+            echo
+            echo "Creating cluster"
+            echo
+            create_cluster
+        elif [[ ${OPERATION} == "recreate" ]]; then
+            echo
+            echo "Cluster ${KIND_CLUSTER_NAME} does not exist. Creating rather than recreating"
+            echo "Creating cluster"
+            echo
+            create_cluster
+        elif [[ ${OPERATION} == "stop" || ${OEPRATON} == "deploy" || ${OPERATION} == "test" ]]; then
+            echo
+            echo "Cluster ${KIND_CLUSTER_NAME} does not exist. It should exist for ${OPERATION} operation"
+            echo
+            exit 1
+        else
+            echo
+            echo "Wrong cluster operation: ${OPERATION}. Should be one of:"
+            echo "${FORMATTED_KIND_OPERATIONS}"
+            echo
+            exit 1
+        fi
+    fi
+}
+
+function check_cluster_ready_for_airflow() {
+    kubectl cluster-info --cluster "${KUBECTL_CLUSTER_NAME}"
+    kubectl get nodes --cluster "${KUBECTL_CLUSTER_NAME}"
+    echo
+    echo "Showing storageClass"
+    echo
+    kubectl get storageclass --cluster "${KUBECTL_CLUSTER_NAME}"
+    echo
+    echo "Showing kube-system pods"
+    echo
+    kubectl get -n kube-system pods --cluster "${KUBECTL_CLUSTER_NAME}"
+    echo
+    echo "Airflow environment on kubernetes is good to go!"
+    echo
+    kubectl create namespace test-namespace --cluster "${KUBECTL_CLUSTER_NAME}"
+}
+
+
+function build_kubernetes_image() {
+    cd "${AIRFLOW_SOURCES}" || exit 1
+    prepare_prod_build
+    if [[ $(docker images -q "${AIRFLOW_PROD_IMAGE}") == "" ||
+            ${FORCE_BUILD_IMAGES:="false"} == "true" ]]; then
+        build_prod_image
+    else
+        echo
+        echo "Skip rebuilding prod image. Use --force-build-images to rebuild prod image."
+        echo
+    fi
+    echo
+    echo "Adding kubernetes-specific scripts to prod image."
+    echo "Building ${AIRFLOW_KUBERNETES_IMAGE} from ${AIRFLOW_PROD_IMAGE} with latest sources."
+    echo
+    docker build \
+        --build-arg AIRFLOW_PROD_IMAGE="${AIRFLOW_PROD_IMAGE}" \
+        --cache-from "${AIRFLOW_PROD_IMAGE}" \
+        --tag="${AIRFLOW_KUBERNETES_IMAGE}" \
+        -f- . << 'EOF'
+    ARG AIRFLOW_PROD_IMAGE
+    FROM ${AIRFLOW_PROD_IMAGE}
+
+    ARG AIRFLOW_SOURCES=/home/airflow/airflow_sources/
+    ENV AIRFLOW_SOURCES=${AIRFLOW_SOURCES}
+
+    USER root
+
+    COPY --chown=airflow:airflow . ${AIRFLOW_SOURCES}
+
+    COPY scripts/ci/kubernetes/docker/airflow-test-env-init-db.sh /tmp/airflow-test-env-init-db.sh
+    COPY scripts/ci/kubernetes/docker/airflow-test-env-init-dags.sh /tmp/airflow-test-env-init-dags.sh
+    COPY scripts/ci/kubernetes/docker/bootstrap.sh /bootstrap.sh
+
+    RUN chmod +x /bootstrap.sh
+
+
+    USER airflow
+
+
+    ENTRYPOINT ["/bootstrap.sh"]
+EOF
+
+    echo "The ${AIRFLOW_KUBERNETES_IMAGE} is prepared for deployment."
+}
+
+function load_image_to_kind_cluster() {
+    echo
+    echo "Loading ${AIRFLOW_KUBERNETES_IMAGE} to ${KIND_CLUSTER_NAME}"
+    echo
+    kind load docker-image --name "${KIND_CLUSTER_NAME}" "${AIRFLOW_KUBERNETES_IMAGE}"
+}
+
+function prepare_kubernetes_app_variables() {
+    echo
+    echo "Preparing kubernetes variables"
+    echo
+    KUBERNETES_APP_DIR="${AIRFLOW_SOURCES}/scripts/ci/kubernetes/app"
+    TEMPLATE_DIRNAME="${KUBERNETES_APP_DIR}/templates"
+    BUILD_DIRNAME="${KUBERNETES_APP_DIR}/build"
+
+    # shellcheck source=common/_image_variables.sh
+    . "${AIRFLOW_SOURCES}/common/_image_variables.sh"
+
+    # Source branch will be set in DockerHub
+    SOURCE_BRANCH=${SOURCE_BRANCH:=${DEFAULT_BRANCH}}
+    BRANCH_NAME=${BRANCH_NAME:=${SOURCE_BRANCH}}
+
+    if [[ ! -d "${BUILD_DIRNAME}" ]]; then
+        mkdir -p "${BUILD_DIRNAME}"
+    fi
+
+    rm -f "${BUILD_DIRNAME}"/*
+    rm -f "${BUILD_DIRNAME}"/*
+
+    if [[ "${KUBERNETES_MODE}" == "image" ]]; then
+        INIT_DAGS_VOLUME_NAME=airflow-dags
+        POD_AIRFLOW_DAGS_VOLUME_NAME=airflow-dags
+        CONFIGMAP_DAGS_FOLDER=/opt/airflow/dags
+        CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT=
+        CONFIGMAP_DAGS_VOLUME_CLAIM=airflow-dags
+    else
+        INIT_DAGS_VOLUME_NAME=airflow-dags-fake
+        POD_AIRFLOW_DAGS_VOLUME_NAME=airflow-dags-git
+        CONFIGMAP_DAGS_FOLDER=/opt/airflow/dags/repo/airflow/example_dags
+        CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT=/opt/airflow/dags
+        CONFIGMAP_DAGS_VOLUME_CLAIM=
+    fi
+
+
+    CONFIGMAP_GIT_REPO=${CI_SOURCE_REPO}
+    CONFIGMAP_BRANCH=${CI_SOURCE_BRANCH}
+}
+
+function prepare_kubernetes_resources() {
+    echo
+    echo "Preparing kubernetes resources"
+    echo
+    if [[ "${KUBERNETES_MODE}" == "image" ]]; then
+        sed -e "s/{{INIT_GIT_SYNC}}//g" \
+            "${TEMPLATE_DIRNAME}/airflow.template.yaml" >"${BUILD_DIRNAME}/airflow.yaml"
+    else
+        sed -e "/{{INIT_GIT_SYNC}}/{r ${TEMPLATE_DIRNAME}/init_git_sync.template.yaml" -e 'd}' \
+            "${TEMPLATE_DIRNAME}/airflow.template.yaml" >"${BUILD_DIRNAME}/airflow.yaml"
+    fi
+    sed -i "s|{{AIRFLOW_KUBERNETES_IMAGE}}|${AIRFLOW_KUBERNETES_IMAGE}|g" "${BUILD_DIRNAME}/airflow.yaml"
+
+    sed -i "s|{{CONFIGMAP_GIT_REPO}}|${CONFIGMAP_GIT_REPO}|g" "${BUILD_DIRNAME}/airflow.yaml"
+    sed -i "s|{{CONFIGMAP_BRANCH}}|${CONFIGMAP_BRANCH}|g" "${BUILD_DIRNAME}/airflow.yaml"
+    sed -i "s|{{INIT_DAGS_VOLUME_NAME}}|${INIT_DAGS_VOLUME_NAME}|g" "${BUILD_DIRNAME}/airflow.yaml"
+    sed -i "s|{{POD_AIRFLOW_DAGS_VOLUME_NAME}}|${POD_AIRFLOW_DAGS_VOLUME_NAME}|g" \
+        "${BUILD_DIRNAME}/airflow.yaml"
+
+    sed "s|{{CONFIGMAP_DAGS_FOLDER}}|${CONFIGMAP_DAGS_FOLDER}|g" \
+        "${TEMPLATE_DIRNAME}/configmaps.template.yaml" >"${BUILD_DIRNAME}/configmaps.yaml"
+    sed -i "s|{{CONFIGMAP_GIT_REPO}}|${CONFIGMAP_GIT_REPO}|g" "${BUILD_DIRNAME}/configmaps.yaml"
+    sed -i "s|{{CONFIGMAP_BRANCH}}|${CONFIGMAP_BRANCH}|g" "${BUILD_DIRNAME}/configmaps.yaml"
+    sed -i "s|{{CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT}}|${CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT}|g" \
+        "${BUILD_DIRNAME}/configmaps.yaml"
+    sed -i "s|{{CONFIGMAP_DAGS_VOLUME_CLAIM}}|${CONFIGMAP_DAGS_VOLUME_CLAIM}|g" \
+        "${BUILD_DIRNAME}/configmaps.yaml"
+    sed -i "s|{{AIRFLOW_KUBERNETES_IMAGE_NAME}}|${AIRFLOW_KUBERNETES_IMAGE_NAME}|g" \
+        "${BUILD_DIRNAME}/configmaps.yaml"
+    sed -i "s|{{AIRFLOW_KUBERNETES_IMAGE_TAG}}|${AIRFLOW_KUBERNETES_IMAGE_TAG}|g" \
+        "${BUILD_DIRNAME}/configmaps.yaml"
+}
+
+function apply_kubernetes_resources() {
+    echo
+    echo "Apply kubernetes resources."
+    echo
+
+
+    kubectl delete -f "${KUBERNETES_APP_DIR}/postgres.yaml" --cluster "${KUBECTL_CLUSTER_NAME}" \
+        2>&1 | grep -v "NotFound" || true
+    kubectl delete -f "${BUILD_DIRNAME}/airflow.yaml" --cluster "${KUBECTL_CLUSTER_NAME}" \
+        2>&1 | grep -v "NotFound" || true
+    kubectl delete -f "${KUBERNETES_APP_DIR}/secrets.yaml" --cluster "${KUBECTL_CLUSTER_NAME}" \
+        2>&1 | grep -v "NotFound" || true
+
+    set -e
+
+    kubectl apply -f "${KUBERNETES_APP_DIR}/secrets.yaml" --cluster "${KUBECTL_CLUSTER_NAME}"
+    kubectl apply -f "${BUILD_DIRNAME}/configmaps.yaml" --cluster "${KUBECTL_CLUSTER_NAME}"
+    kubectl apply -f "${KUBERNETES_APP_DIR}/postgres.yaml" --cluster "${KUBECTL_CLUSTER_NAME}"
+    kubectl apply -f "${KUBERNETES_APP_DIR}/volumes.yaml" --cluster "${KUBECTL_CLUSTER_NAME}"
+    kubectl apply -f "${BUILD_DIRNAME}/airflow.yaml" --cluster "${KUBECTL_CLUSTER_NAME}"
+}
+
+
+function dump_kubernetes_logs() {
+    POD=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' \
+        --cluster "${KUBECTL_CLUSTER_NAME}" | grep airflow | head -1)
+    echo "------- pod description -------"
+    kubectl describe pod "${POD}" --cluster "${KUBECTL_CLUSTER_NAME}"
+    echo "------- webserver init container logs - init -------"
+    kubectl logs "${POD}" -c init --cluster "${KUBECTL_CLUSTER_NAME}" || true
+    if [[ "${KUBERNETES_MODE}" == "git" ]]; then
+        echo "------- webserver init container logs - git-sync-clone -------"
+        kubectl logs "${POD}" -c git-sync-clone --cluster "${KUBECTL_CLUSTER_NAME}" || true
+    fi
+    echo "------- webserver logs -------"
+    kubectl logs "${POD}" -c webserver --cluster "${KUBECTL_CLUSTER_NAME}" || true
+    echo "------- scheduler logs -------"
+    kubectl logs "${POD}" -c scheduler --cluster "${KUBECTL_CLUSTER_NAME}" || true
+    echo "--------------"
+}
+
+function wait_for_airflow_pods_up_and_running() {
+    set +o pipefail
+    # wait for up to 10 minutes for everything to be deployed
+    PODS_ARE_READY="0"
+    for i in {1..150}; do
+        echo "------- Running kubectl get pods: $i -------"
+        PODS=$(kubectl get pods --cluster "${KUBECTL_CLUSTER_NAME}" | awk 'NR>1 {print $0}')
+        echo "$PODS"
+        NUM_AIRFLOW_READY=$(echo "${PODS}" | grep airflow | awk '{print $2}' | grep -cE '([0-9])\/(\1)' \
+            | xargs)
+        NUM_POSTGRES_READY=$(echo "${PODS}" | grep postgres | awk '{print $2}' | grep -cE '([0-9])\/(\1)' \
+            | xargs)
+        if [[ "${NUM_AIRFLOW_READY}" == "1" && "${NUM_POSTGRES_READY}" == "1" ]]; then
+            PODS_ARE_READY="1"
+            break
+        fi
+        sleep 4
+    done
+    POD=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' \
+        --cluster "${KUBECTL_CLUSTER_NAME}" | grep airflow | head -1)
+
+    if [[ "${PODS_ARE_READY}" == "1" ]]; then
+        echo "PODS are ready."
+        set -o pipefail
+    else
+        echo >&2 "PODS are not ready after waiting for a long time. Exiting..."
+        dump_kubernetes_logs
+        exit 1
+    fi
+}
+
+
+function wait_for_airflow_webserver_up_and_running() {
+    set +o pipefail
+    # Wait until Airflow webserver is up
+    KUBERNETES_HOST=localhost
+    AIRFLOW_WEBSERVER_IS_READY="0"
+    CONSECUTIVE_SUCCESS_CALLS=0
+    for i in {1..30}; do
+        echo "------- Wait until webserver is up: $i -------"
+        PODS=$(kubectl get pods --cluster "${KUBECTL_CLUSTER_NAME}" | awk 'NR>1 {print $0}')
+        echo "$PODS"
+        HTTP_CODE=$(curl -LI "http://${KUBERNETES_HOST}:30809/health" -o /dev/null -w '%{http_code}\n' -sS) \
+            || true
+        if [[ "${HTTP_CODE}" == 200 ]]; then
+            ((CONSECUTIVE_SUCCESS_CALLS += 1))
+        else
+            CONSECUTIVE_SUCCESS_CALLS="0"
+        fi
+        if [[ "${CONSECUTIVE_SUCCESS_CALLS}" == 3 ]]; then
+            AIRFLOW_WEBSERVER_IS_READY="1"
+            break
+        fi
+        sleep 10
+    done
+    set -o pipefail
+    if [[ "${AIRFLOW_WEBSERVER_IS_READY}" == "1" ]]; then
+        echo
+        echo "Airflow webserver is ready."
+        echo
+    else
+        echo >&2
+        echo >&2 "Airflow webserver is not ready after waiting for a long time. Exiting..."
+        echo >&2
+        dump_kubernetes_logs
+        exit 1
+    fi
+}
diff --git a/scripts/ci/libraries/_local_mounts.sh b/scripts/ci/libraries/_local_mounts.sh
new file mode 100644
index 0000000..5750600
--- /dev/null
+++ b/scripts/ci/libraries/_local_mounts.sh
@@ -0,0 +1,66 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Those are files that are mounted locally when mounting local sources is requested
+# By default not the whole airflow sources directory is mounted because there are often
+# artifacts created there (for example .egg-info files) that are breaking the capability
+# of running different python versions in Breeze. So we only mount what is needed by default.
+function generate_local_mounts_list {
+    local prefix="$1"
+    LOCAL_MOUNTS=(
+        "$prefix".bash_aliases:/root/.bash_aliases:cached
+        "$prefix".bash_history:/root/.bash_history:cached
+        "$prefix".coveragerc:/opt/airflow/.coveragerc:cached
+        "$prefix".dockerignore:/opt/airflow/.dockerignore:cached
+        "$prefix".flake8:/opt/airflow/.flake8:cached
+        "$prefix".github:/opt/airflow/.github:cached
+        "$prefix".inputrc:/root/.inputrc:cached
+        "$prefix".rat-excludes:/opt/airflow/.rat-excludes:cached
+        "$prefix"CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached
+        "$prefix"LICENSE:/opt/airflow/LICENSE:cached
+        "$prefix"MANIFEST.in:/opt/airflow/MANIFEST.in:cached
+        "$prefix"NOTICE:/opt/airflow/NOTICE:cached
+        "$prefix"airflow:/opt/airflow/airflow:cached
+        "$prefix"common:/opt/airflow/common:cached
+        "$prefix"dags:/opt/airflow/dags:cached
+        "$prefix"dev:/opt/airflow/dev:cached
+        "$prefix"docs:/opt/airflow/docs:cached
+        "$prefix"files:/files:cached
+        "$prefix"dist:/dist:cached
+        "$prefix"hooks:/opt/airflow/hooks:cached
+        "$prefix"logs:/root/airflow/logs:cached
+        "$prefix"pytest.ini:/opt/airflow/pytest.ini:cached
+        "$prefix"requirements:/opt/airflow/requirements:cached
+        "$prefix"scripts:/opt/airflow/scripts:cached
+        "$prefix"scripts/ci/in_container/entrypoint_ci.sh:/entrypoint_ci.sh:cached
+        "$prefix"setup.cfg:/opt/airflow/setup.cfg:cached
+        "$prefix"setup.py:/opt/airflow/setup.py:cached
+        "$prefix"tests:/opt/airflow/tests:cached
+        "$prefix"kubernetes_tests:/opt/airflow/kubernetes_tests:cached
+        "$prefix"tmp:/opt/airflow/tmp:cached
+    )
+}
+
+# Converts the local mounts that we defined above to the right set of -v
+# volume mappings in docker-compose file. This is needed so that we only
+# maintain the volumes in one place (above)
+function convert_local_mounts_to_docker_params() {
+    generate_local_mounts_list "${AIRFLOW_SOURCES}/"
+    # Bash can't "return" arrays, so we need to quote any special characters
+    printf -- '-v %q ' "${LOCAL_MOUNTS[@]}"
+}
diff --git a/scripts/ci/libraries/_md5sum.sh b/scripts/ci/libraries/_md5sum.sh
new file mode 100644
index 0000000..c9a254a
--- /dev/null
+++ b/scripts/ci/libraries/_md5sum.sh
@@ -0,0 +1,137 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#
+# Verifies if stored md5sum of the file changed since the last tme ot was checked
+# The md5sum files are stored in .build directory - you can delete this directory
+# If you want to rebuild everything from the scratch
+#
+function calculate_file_md5sum {
+    local FILE="${1}"
+    local MD5SUM
+    local MD5SUM_CACHE_DIR="${BUILD_CACHE_DIR}/${DEFAULT_BRANCH}/${PYTHON_MAJOR_MINOR_VERSION}/${THE_IMAGE_TYPE}"
+    mkdir -pv "${MD5SUM_CACHE_DIR}"
+    MD5SUM=$(md5sum "${FILE}")
+    local MD5SUM_FILE
+    MD5SUM_FILE="${MD5SUM_CACHE_DIR}"/$(basename "${FILE}").md5sum
+    local MD5SUM_FILE_NEW
+    MD5SUM_FILE_NEW=${CACHE_TMP_FILE_DIR}/$(basename "${FILE}").md5sum.new
+    echo "${MD5SUM}" > "${MD5SUM_FILE_NEW}"
+    local RET_CODE=0
+    if [[ ! -f "${MD5SUM_FILE}" ]]; then
+        print_info "Missing md5sum for ${FILE#${AIRFLOW_SOURCES}} (${MD5SUM_FILE#${AIRFLOW_SOURCES}})"
+        RET_CODE=1
+    else
+        diff "${MD5SUM_FILE_NEW}" "${MD5SUM_FILE}" >/dev/null
+        RES=$?
+        if [[ "${RES}" != "0" ]]; then
+            print_info "The md5sum changed for ${FILE}"
+            RET_CODE=1
+        fi
+    fi
+    return ${RET_CODE}
+}
+
+#
+# Moves md5sum file from it's temporary location in CACHE_TMP_FILE_DIR to
+# BUILD_CACHE_DIR - thus updating stored MD5 sum fo the file
+#
+function move_file_md5sum {
+    local FILE="${1}"
+    local MD5SUM_FILE
+    local MD5SUM_CACHE_DIR="${BUILD_CACHE_DIR}/${DEFAULT_BRANCH}/${PYTHON_MAJOR_MINOR_VERSION}/${THE_IMAGE_TYPE}"
+    mkdir -pv "${MD5SUM_CACHE_DIR}"
+    MD5SUM_FILE="${MD5SUM_CACHE_DIR}"/$(basename "${FILE}").md5sum
+    local MD5SUM_FILE_NEW
+    MD5SUM_FILE_NEW=${CACHE_TMP_FILE_DIR}/$(basename "${FILE}").md5sum.new
+    if [[ -f "${MD5SUM_FILE_NEW}" ]]; then
+        mv "${MD5SUM_FILE_NEW}" "${MD5SUM_FILE}"
+        print_info "Updated md5sum file ${MD5SUM_FILE} for ${FILE}."
+    fi
+}
+
+#
+# Stores md5sum files for all important files and
+# records that we built the images locally so that next time we use
+# it from the local docker cache rather than pull (unless forced)
+#
+function update_all_md5_files() {
+    print_info
+    print_info "Updating md5sum files"
+    print_info
+    for FILE in "${FILES_FOR_REBUILD_CHECK[@]}"
+    do
+        move_file_md5sum "${AIRFLOW_SOURCES}/${FILE}"
+    done
+    mkdir -pv "${BUILD_CACHE_DIR}/${BRANCH_NAME}"
+    touch "${BUILT_IMAGE_FLAG_FILE}"
+}
+
+
+function calculate_md5sum_for_all_files() {
+    FILES_MODIFIED="false"
+    set +e
+    for FILE in "${FILES_FOR_REBUILD_CHECK[@]}"
+    do
+        if ! calculate_file_md5sum "${AIRFLOW_SOURCES}/${FILE}"; then
+            FILES_MODIFIED="true"
+        fi
+    done
+    set +e
+}
+
+#
+# Checks md5sum of all important files in order to optimise speed of running various operations
+# That mount sources of Airflow to container and require docker image built with latest dependencies.
+# the Docker image will only be marked for rebuilding only in case any of the important files change:
+# * setup.py
+# * setup.cfg
+# * Dockerfile.ci
+# * airflow/version.py
+#
+# This is needed because we want to skip rebuilding of the image when only airflow sources change but
+# Trigger rebuild in case we need to change dependencies (setup.py, setup.cfg, change version of Airflow
+# or the Dockerfile.ci itself changes.
+#
+# Another reason to skip rebuilding Docker is thar currently it takes a bit longer time than simple Docker
+# We need to fix group permissions of files in Docker because different linux build services have
+# different default umask and Docker uses group permissions in checking for cache invalidation.
+#
+# As result of this check - most of the static checks will start pretty much immediately.
+#
+function check_if_docker_build_is_needed() {
+    print_info
+    print_info "Checking if image build is needed for ${THE_IMAGE_TYPE} image."
+    print_info
+    if [[ ${FORCE_BUILD_IMAGES:=""} == "true" ]]; then
+        echo "Docker image build is forced for ${THE_IMAGE_TYPE} image"
+        calculate_md5sum_for_all_files
+        export NEEDS_DOCKER_BUILD="true"
+    else
+        calculate_md5sum_for_all_files
+        if [[ ${FILES_MODIFIED} == "true" ]]; then
+            export NEEDS_DOCKER_BUILD="true"
+        fi
+        if [[ ${NEEDS_DOCKER_BUILD} == "true" ]]; then
+            echo "Docker image build is needed for ${THE_IMAGE_TYPE} image!"
+        else
+            print_info "Docker image build is not needed for ${THE_IMAGE_TYPE} image!"
+        fi
+    fi
+    print_info
+}
diff --git a/scripts/ci/libraries/_parameters.sh b/scripts/ci/libraries/_parameters.sh
new file mode 100644
index 0000000..7828bb5
--- /dev/null
+++ b/scripts/ci/libraries/_parameters.sh
@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Reads environment variable passed as first parameter from the .build cache file
+function read_from_file {
+    cat "${BUILD_CACHE_DIR}/.$1" 2>/dev/null || true
+}
+
+# Saves environment variable passed as first parameter to the .build cache file
+function save_to_file {
+    # shellcheck disable=SC2005
+    echo "$(eval echo "\$$1")" > "${BUILD_CACHE_DIR}/.$1"
+}
+
+# check if parameter set for the variable is allowed (should be on the _BREEZE_ALLOWED list)
+# and if it is, it saves it to .build cache file. In case the parameter is wrong, the
+# saved variable is removed (so that bad value is not used again in case it comes from there)
+# and exits with an error
+function check_and_save_allowed_param {
+    _VARIABLE_NAME="${1}"
+    _VARIABLE_DESCRIPTIVE_NAME="${2}"
+    _FLAG="${3}"
+    _ALLOWED_VALUES_ENV_NAME="_BREEZE_ALLOWED_${_VARIABLE_NAME}S"
+    _ALLOWED_VALUES=" ${!_ALLOWED_VALUES_ENV_NAME//$'\n'/ } "
+    _VALUE=${!_VARIABLE_NAME}
+    if [[ ${_ALLOWED_VALUES:=} != *" ${_VALUE} "* ]]; then
+        echo >&2
+        echo >&2 "ERROR:  Allowed ${_VARIABLE_DESCRIPTIVE_NAME}: [${_ALLOWED_VALUES}]. Is: '${!_VARIABLE_NAME}'."
+        echo >&2
+        echo >&2 "Switch to supported value with ${_FLAG} flag."
+
+        if [[ -n ${!_VARIABLE_NAME} && \
+            -f "${BUILD_CACHE_DIR}/.${_VARIABLE_NAME}" && \
+            ${!_VARIABLE_NAME} == $(cat "${BUILD_CACHE_DIR}/.${_VARIABLE_NAME}" ) ]]; then
+            echo >&2
+            echo >&2 "Removing ${BUILD_CACHE_DIR}/.${_VARIABLE_NAME}. Next time you run it, it should be OK."
+            echo >&2
+            rm -f "${BUILD_CACHE_DIR}/.${_VARIABLE_NAME}"
+        fi
+        exit 1
+    fi
+    save_to_file "${_VARIABLE_NAME}"
+}
diff --git a/scripts/ci/libraries/_permissions.sh b/scripts/ci/libraries/_permissions.sh
new file mode 100644
index 0000000..1b936fe
--- /dev/null
+++ b/scripts/ci/libraries/_permissions.sh
@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+# Fixing permissions for all important files that are going to be added to Docker context
+# This is necessary, because there are different default umask settings on different *NIX
+# In case of some systems (especially in the CI environments) there is default +w group permission
+# set automatically via UMASK when git checkout is performed.
+#    https://unix.stackexchange.com/questions/315121/why-is-the-default-umask-002-or-022-in-many-unix-systems-seems-insecure-by-defa
+# Unfortunately default setting in git is to use UMASK by default:
+#    https://git-scm.com/docs/git-config/1.6.3.1#git-config-coresharedRepository
+# This messes around with Docker context invalidation because the same files have different permissions
+# and effectively different hash used for context validation calculation.
+#
+# We fix it by removing write permissions for other/group for all files that are in the Docker context.
+#
+# Since we can't (easily) tell what dockerignore would restrict, we'll just to
+# it to "all" files in the git repo, making sure to exclude the www/static/docs
+# symlink which is broken until the docs are built.
+function filterout_deleted_files {
+  # Take NUL-separated stdin, return only files that exist on stdout NUL-separated
+  # This is to cope with users deleting files or folders locally but not doing `git rm`
+  xargs -0 "$STAT_BIN" --printf '%n\0' 2>/dev/null || true;
+}
+
+# Fixes permissions for groups for all the files that are quickly filtered using the filterout_deleted_files
+function fix_group_permissions() {
+    if [[ ${PERMISSIONS_FIXED:=} == "true" ]]; then
+        echo
+        echo "Permissions already fixed"
+        echo
+        return
+    fi
+    echo
+    pushd "${AIRFLOW_SOURCES}" >/dev/null  || exit 1
+    # This deals with files
+    git ls-files -z -- ./ | filterout_deleted_files | xargs -0 chmod og-w
+    # and this deals with directories
+    git ls-tree -z -r -d --name-only HEAD | filterout_deleted_files | xargs -0 chmod og-w,og+x
+    popd >/dev/null || exit 1
+    echo "Fixed group permissions"
+    echo
+    export PERMISSIONS_FIXED="true"
+}
diff --git a/scripts/ci/libraries/_push_pull_remove_images.sh b/scripts/ci/libraries/_push_pull_remove_images.sh
new file mode 100644
index 0000000..c09ff57
--- /dev/null
+++ b/scripts/ci/libraries/_push_pull_remove_images.sh
@@ -0,0 +1,183 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Pulls image in case it is needed (either has never been pulled or pulling was forced
+# Should be run with set +e
+# Parameters:
+#   $1 -> image to pull
+function pull_image_if_needed() {
+    local IMAGE_TO_PULL="${1}"
+    local IMAGE_HASH
+    IMAGE_HASH=$(docker images -q "${IMAGE_TO_PULL}" 2> /dev/null || true)
+    local PULL_IMAGE=${FORCE_PULL_IMAGES}
+
+    if [[ "${IMAGE_HASH}" == "" ]]; then
+        PULL_IMAGE="true"
+    fi
+    if [[ "${PULL_IMAGE}" == "true" ]]; then
+        echo
+        echo "Pulling the image ${IMAGE_TO_PULL}"
+        echo
+        verbose_docker pull "${IMAGE_TO_PULL}" | tee -a "${OUTPUT_LOG}"
+        EXIT_VALUE="$?"
+        echo
+        return ${EXIT_VALUE}
+    fi
+}
+
+# Pulls image if needed but tries to pull it from cache (for example GitHub registry) before
+# It attempts to pull it from the main repository. This is used to speed up the builds
+# In GitHub Actions.
+# Parameters:
+#   $1 -> image to pull
+#   $2 -> cache image to pull first
+function pull_image_possibly_from_cache() {
+    local IMAGE="${1}"
+    local CACHED_IMAGE="${2}"
+    local IMAGE_PULL_RETURN_VALUE=-1
+
+    set +e
+    if [[ ${CACHED_IMAGE:=} != "" ]]; then
+        pull_image_if_needed "${CACHED_IMAGE}"
+        IMAGE_PULL_RETURN_VALUE="$?"
+        if [[ ${IMAGE_PULL_RETURN_VALUE} == "0" ]]; then
+            # Tag the image to be the target one
+            verbose_docker tag "${CACHED_IMAGE}" "${IMAGE}"
+        fi
+    fi
+    if [[ ${IMAGE_PULL_RETURN_VALUE} != "0" ]]; then
+        pull_image_if_needed "${IMAGE}"
+    fi
+    set -e
+}
+
+# Pulls CI image in case caching strategy is "pulled" and the image needs to be pulled
+function pull_ci_image_if_needed() {
+    # Whether to force pull images to populate cache
+    export FORCE_PULL_IMAGES=${FORCE_PULL_IMAGES:="false"}
+
+    if [[ "${DOCKER_CACHE}" == "pulled" ]]; then
+        if [[ "${FORCE_PULL_IMAGES}" == "true" ]]; then
+            echo
+            echo "Force pull base image ${PYTHON_BASE_IMAGE}"
+            echo
+            if [[ -n ${DETECTED_TERMINAL:=""} ]]; then
+                echo -n "
+Docker pulling ${PYTHON_BASE_IMAGE}.
+                    " > "${DETECTED_TERMINAL}"
+            fi
+            if [[ ${PULL_PYTHON_BASE_IMAGES_FROM_CACHE:="true"} == "true" ]]; then
+                pull_image_possibly_from_cache "${PYTHON_BASE_IMAGE}" "${CACHED_PYTHON_BASE_IMAGE}"
+            else
+                verbose_docker pull "${PYTHON_BASE_IMAGE}" | tee -a "${OUTPUT_LOG}"
+            fi
+            echo
+        fi
+        pull_image_possibly_from_cache "${AIRFLOW_CI_IMAGE}" "${CACHED_AIRFLOW_CI_IMAGE}"
+    fi
+}
+
+
+# Pulls PROD image in case caching strategy is "pulled" and the image needs to be pulled
+function pull_prod_images_if_needed() {
+    # Whether to force pull images to populate cache
+    export FORCE_PULL_IMAGES=${FORCE_PULL_IMAGES:="false"}
+
+    if [[ "${DOCKER_CACHE}" == "pulled" ]]; then
+        if [[ "${FORCE_PULL_IMAGES}" == "true" ]]; then
+            echo
+            echo "Force pull base image ${PYTHON_BASE_IMAGE}"
+            echo
+            if [[ ${PULL_PYTHON_BASE_IMAGES_FROM_CACHE:="true"} == "true" ]]; then
+                pull_image_possibly_from_cache "${PYTHON_BASE_IMAGE}" "${CACHED_PYTHON_BASE_IMAGE}"
+            else
+                verbose_docker pull "${PYTHON_BASE_IMAGE}" | tee -a "${OUTPUT_LOG}"
+            fi
+            echo
+        fi
+        # "Build" segment of production image
+        pull_image_possibly_from_cache "${AIRFLOW_PROD_BUILD_IMAGE}" "${CACHED_AIRFLOW_PROD_BUILD_IMAGE}"
+        # Main segment of production image
+        pull_image_possibly_from_cache "${AIRFLOW_PROD_IMAGE}" "${CACHED_AIRFLOW_PROD_IMAGE}"
+    fi
+}
+
+# Pushes Ci image and it's manifest to the registry. In case the image was taken from cache registry
+# it is pushed to the cache, not to the main registry. Manifest is only pushed to the main registry
+function push_ci_image() {
+    if [[ ${CACHED_AIRFLOW_CI_IMAGE:=} != "" ]]; then
+        verbose_docker tag "${AIRFLOW_CI_IMAGE}" "${CACHED_AIRFLOW_CI_IMAGE}"
+        IMAGE_TO_PUSH="${CACHED_AIRFLOW_CI_IMAGE}"
+    else
+        IMAGE_TO_PUSH="${AIRFLOW_CI_IMAGE}"
+    fi
+    verbose_docker push "${IMAGE_TO_PUSH}"
+    if [[ ${CACHED_AIRFLOW_CI_IMAGE} == "" ]]; then
+        # Only push manifest image for builds that are not using CI cache
+        verbose_docker tag "${AIRFLOW_CI_LOCAL_MANIFEST_IMAGE}" "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}"
+        verbose_docker push "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}"
+        if [[ -n ${DEFAULT_IMAGE:=""} ]]; then
+            verbose_docker push "${DEFAULT_IMAGE}"
+        fi
+    fi
+    if [[ ${CACHED_PYTHON_BASE_IMAGE} != "" ]]; then
+        verbose_docker tag "${PYTHON_BASE_IMAGE}" "${CACHED_PYTHON_BASE_IMAGE}"
+        verbose_docker push "${CACHED_PYTHON_BASE_IMAGE}"
+    fi
+
+}
+
+# Pushes PROD image to the registry. In case the image was taken from cache registry
+# it is also pushed to the cache, not to the main registry
+function push_prod_images() {
+    if [[ ${CACHED_AIRFLOW_PROD_IMAGE:=} != "" ]]; then
+        verbose_docker tag "${AIRFLOW_PROD_IMAGE}" "${CACHED_AIRFLOW_PROD_IMAGE}"
+        IMAGE_TO_PUSH="${CACHED_AIRFLOW_PROD_IMAGE}"
+    else
+        IMAGE_TO_PUSH="${AIRFLOW_PROD_IMAGE}"
+    fi
+    if [[ ${CACHED_AIRFLOW_PROD_BUILD_IMAGE:=} != "" ]]; then
+        verbose_docker tag "${AIRFLOW_PROD_BUILD_IMAGE}" "${CACHED_AIRFLOW_PROD_BUILD_IMAGE}"
+        IMAGE_TO_PUSH_BUILD="${CACHED_AIRFLOW_PROD_BUILD_IMAGE}"
+    else
+        IMAGE_TO_PUSH_BUILD="${AIRFLOW_PROD_BUILD_IMAGE}"
+    fi
+    verbose_docker push "${IMAGE_TO_PUSH}"
+    verbose_docker push "${IMAGE_TO_PUSH_BUILD}"
+    if [[ -n ${DEFAULT_IMAGE:=""} && ${CACHED_AIRFLOW_PROD_IMAGE} == "" ]]; then
+        verbose_docker push "${DEFAULT_IMAGE}"
+    fi
+
+    # we do not need to push PYTHON base image here - they are already pushed in the CI push
+}
+
+# Removes airflow CI and base images
+function remove_all_images() {
+    echo
+    "${AIRFLOW_SOURCES}/confirm" "Removing all local images ."
+    echo
+    verbose_docker rmi "${PYTHON_BASE_IMAGE}" || true
+    verbose_docker rmi "${AIRFLOW_CI_IMAGE}" || true
+    echo
+    echo "###################################################################"
+    echo "NOTE!! Removed Airflow images for Python version ${PYTHON_MAJOR_MINOR_VERSION}."
+    echo "       But the disk space in docker will be reclaimed only after"
+    echo "       running 'docker system prune' command."
+    echo "###################################################################"
+    echo
+}
diff --git a/scripts/ci/libraries/_runs.sh b/scripts/ci/libraries/_runs.sh
new file mode 100644
index 0000000..cd5a44a
--- /dev/null
+++ b/scripts/ci/libraries/_runs.sh
@@ -0,0 +1,50 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Docker command to build documentation
+function run_docs() {
+    verbose_docker run "${EXTRA_DOCKER_FLAGS[@]}" -t \
+            --entrypoint "/usr/local/bin/dumb-init"  \
+            --env PYTHONDONTWRITEBYTECODE \
+            --env VERBOSE \
+            --env VERBOSE_COMMANDS \
+            --env HOST_USER_ID="$(id -ur)" \
+            --env HOST_GROUP_ID="$(id -gr)" \
+            --rm \
+            "${AIRFLOW_CI_IMAGE}" \
+            "--" "/opt/airflow/docs/build" \
+            | tee -a "${OUTPUT_LOG}"
+}
+
+# Docker command to generate constraint requirement files.
+function run_generate_requirements() {
+    docker run "${EXTRA_DOCKER_FLAGS[@]}" \
+        --entrypoint "/usr/local/bin/dumb-init"  \
+        --env PYTHONDONTWRITEBYTECODE \
+        --env VERBOSE \
+        --env VERBOSE_COMMANDS \
+        --env HOST_USER_ID="$(id -ur)" \
+        --env HOST_GROUP_ID="$(id -gr)" \
+        --env UPGRADE_WHILE_GENERATING_REQUIREMENTS \
+        --env PYTHON_MAJOR_MINOR_VERSION \
+        --env CHECK_REQUIREMENTS_ONLY \
+        --rm \
+        "${AIRFLOW_CI_IMAGE}" \
+        "--" "/opt/airflow/scripts/ci/in_container/run_generate_requirements.sh" \
+        | tee -a "${OUTPUT_LOG}"
+}
diff --git a/scripts/ci/libraries/_sanity_checks.sh b/scripts/ci/libraries/_sanity_checks.sh
new file mode 100644
index 0000000..d2bc01e
--- /dev/null
+++ b/scripts/ci/libraries/_sanity_checks.sh
@@ -0,0 +1,166 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Fixes a file that is expected to be a file. If - for whatever reason - the local file is not created
+# When mounting it to container, docker assumes it is a missing directory and creates it. Such mistakenly
+# Created directories should be removed and replaced with files
+function sanitize_file() {
+    if [[ -d "${1}" ]]; then
+        rm -rf "${1}"
+    fi
+    touch "${1}"
+
+}
+
+# Those files are mounted into container when run locally
+# .bash_history is preserved and you can modify .bash_aliases and .inputrc
+# according to your liking
+function sanitize_mounted_files() {
+    sanitize_file "${AIRFLOW_SOURCES}/.bash_history"
+    sanitize_file "${AIRFLOW_SOURCES}/.bash_aliases"
+    sanitize_file "${AIRFLOW_SOURCES}/.inputrc"
+
+    # When KinD cluster is created, the folder keeps authentication information
+    # across sessions
+    mkdir -p "${AIRFLOW_SOURCES}/.kube" >/dev/null 2>&1
+}
+
+#
+# Creates cache directory where we will keep temporary files needed for the docker build
+#
+# This directory will be automatically deleted when the script is killed or exists (via trap)
+# Unless SKIP_CACHE_DELETION variable is set. You can set this variable and then see
+# the output/files generated by the scripts in this directory.
+#
+# Most useful is out.log file in this directory storing verbose output of the scripts.
+#
+function create_cache_directory() {
+    CACHE_TMP_FILE_DIR=$(mktemp -d)
+    export CACHE_TMP_FILE_DIR
+
+    if [[ ${SKIP_CACHE_DELETION:=} != "true" ]]; then
+        trap 'rm -rf -- "${CACHE_TMP_FILE_DIR}"' INT TERM HUP
+    fi
+
+    OUTPUT_LOG="${CACHE_TMP_FILE_DIR}/out.log"
+    export OUTPUT_LOG
+}
+
+# Removes the cache temporary directory
+function remove_cache_directory() {
+    if [[ -d ${CACHE_TMP_FILE_DIR} ]]; then
+        rm -rf "${CACHE_TMP_FILE_DIR}"
+    fi
+}
+
+#
+# Checks if core utils required in the host system are installed and explain what needs to be done if not
+#
+function check_if_coreutils_installed() {
+    set +e
+    getopt -T >/dev/null
+    GETOPT_RETVAL=$?
+
+    if [[ $(uname -s) == 'Darwin' ]] ; then
+        command -v gstat >/dev/null
+        STAT_PRESENT=$?
+    else
+        command -v stat >/dev/null
+        STAT_PRESENT=$?
+    fi
+
+    command -v md5sum >/dev/null
+    MD5SUM_PRESENT=$?
+
+    set -e
+
+    CMDNAME="$(basename -- "$0")"
+
+    ####################  Parsing options/arguments
+    if [[ ${GETOPT_RETVAL} != 4 || "${STAT_PRESENT}" != "0" || "${MD5SUM_PRESENT}" != "0" ]]; then
+        print_info
+        if [[ $(uname -s) == 'Darwin' ]] ; then
+            echo >&2 "You are running ${CMDNAME} in OSX environment"
+            echo >&2 "And you need to install gnu commands"
+            echo >&2
+            echo >&2 "Run 'brew install gnu-getopt coreutils'"
+            echo >&2
+            echo >&2 "Then link the gnu-getopt to become default as suggested by brew."
+            echo >&2
+            echo >&2 "If you use bash, you should run this command:"
+            echo >&2
+            echo >&2 "echo 'export PATH=\"/usr/local/opt/gnu-getopt/bin:\$PATH\"' >> ~/.bash_profile"
+            echo >&2 ". ~/.bash_profile"
+            echo >&2
+            echo >&2 "If you use zsh, you should run this command:"
+            echo >&2
+            echo >&2 "echo 'export PATH=\"/usr/local/opt/gnu-getopt/bin:\$PATH\"' >> ~/.zprofile"
+            echo >&2 ". ~/.zprofile"
+            echo >&2
+            echo >&2 "Login and logout afterwards !!"
+            echo >&2
+            echo >&2 "After re-login, your PATH variable should start with \"/usr/local/opt/gnu-getopt/bin\""
+            echo >&2 "Your current path is ${PATH}"
+            echo >&2
+        else
+            echo >&2 "You do not have necessary tools in your path (getopt, stat, md5sum)."
+            echo >&2 "Please install latest/GNU version of getopt and coreutils."
+            echo >&2 "This can usually be done with 'apt install util-linux coreutils'"
+        fi
+        print_info
+        exit 1
+    fi
+}
+
+#
+# Asserts that we are not inside of the container
+#
+function assert_not_in_container() {
+    if [[ ${SKIP_IN_CONTAINER_CHECK:=} == "true" ]]; then
+        return
+    fi
+    if [[ -f /.dockerenv ]]; then
+        echo >&2
+        echo >&2 "You are inside the Airflow docker container!"
+        echo >&2 "You should only run this script from the host."
+        echo >&2 "Learn more about how we develop and test airflow in:"
+        echo >&2 "https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst"
+        echo >&2
+        exit 1
+    fi
+}
+
+# Changes directory to local sources
+function go_to_airflow_sources {
+    print_info
+    pushd "${AIRFLOW_SOURCES}" &>/dev/null || exit 1
+    print_info
+    print_info "Running in host in $(pwd)"
+    print_info
+}
+
+#
+# Performs basic sanity checks common for most of the scripts in this directory
+#
+function basic_sanity_checks() {
+    assert_not_in_container
+    go_to_airflow_sources
+    check_if_coreutils_installed
+    create_cache_directory
+    sanitize_mounted_files
+}
diff --git a/scripts/ci/libraries/_spinner.sh b/scripts/ci/libraries/_spinner.sh
new file mode 100644
index 0000000..c4ce390
--- /dev/null
+++ b/scripts/ci/libraries/_spinner.sh
@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Function to spin ASCII spinner during pull and build in pre-commits to give the user indication that
+# Pull/Build is happening. It only spins if the output log changes, so if pull/build is stalled
+# The spinner will not move.
+function spin() {
+    local FILE_TO_MONITOR=${1}
+    local SPIN=("-" "\\" "|" "/")
+    echo -n "
+Build log: ${FILE_TO_MONITOR}
+" > "${DETECTED_TERMINAL}"
+
+    LAST_STEP=""
+    while "true"
+    do
+      for i in "${SPIN[@]}"
+      do
+            echo -ne "\r${LAST_STEP}$i" > "${DETECTED_TERMINAL}"
+            local LAST_FILE_SIZE
+            local FILE_SIZE
+            LAST_FILE_SIZE=$(set +e; wc -c "${FILE_TO_MONITOR}" 2>/dev/null | awk '{print $1}' || true)
+            FILE_SIZE=${LAST_FILE_SIZE}
+            while [[ "${LAST_FILE_SIZE}" == "${FILE_SIZE}" ]];
+            do
+                FILE_SIZE=$(set +e; wc -c "${FILE_TO_MONITOR}" 2>/dev/null | awk '{print $1}' || true)
+                sleep 0.2
+            done
+            LAST_FILE_SIZE=FILE_SIZE
+            sleep 0.2
+            if [[ ! -f "${FILE_TO_MONITOR}" ]]; then
+                exit
+            fi
+            LAST_LINE=$(set +e; grep "Step" <"${FILE_TO_MONITOR}" | tail -1 || true)
+            [[ ${LAST_LINE} =~ ^(Step [0-9/]*)\ : ]] && LAST_STEP="${BASH_REMATCH[1]} :"
+      done
+    done
+}
diff --git a/scripts/ci/libraries/_start_end.sh b/scripts/ci/libraries/_start_end.sh
new file mode 100644
index 0000000..b50a405
--- /dev/null
+++ b/scripts/ci/libraries/_start_end.sh
@@ -0,0 +1,79 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#
+# Starts the script.
+# If VERBOSE_COMMANDS variable is set to true, it enables verbose output of commands executed
+# Also prints some useful diagnostics information at start of the script if VERBOSE is set to true
+#
+function script_start {
+    print_info
+    print_info "Running $(basename "$0")"
+    print_info
+    print_info "Log is redirected to ${OUTPUT_LOG}"
+    print_info
+    if [[ ${VERBOSE_COMMANDS:="false"} == "true" ]]; then
+        print_info
+        print_info "Variable VERBOSE_COMMANDS Set to \"true\""
+        print_info "You will see a lot of output"
+        print_info
+        set -x
+    else
+        print_info "You can increase verbosity by running 'export VERBOSE_COMMANDS=\"true\""
+        if [[ ${SKIP_CACHE_DELETION:=} != "true" ]]; then
+            print_info "And skip deleting the output file with 'export SKIP_CACHE_DELETION=\"true\""
+        fi
+        print_info
+        set +x
+    fi
+    START_SCRIPT_TIME=$(date +%s)
+}
+
+#
+# Trap function executed always at the end of the script. In case of verbose output it also
+# Prints the exit code that the script exits with. Removes verbosity of commands in case it was run with
+# command verbosity and in case the script was not run from Breeze (so via ci scripts) it displays
+# total time spent in the script so that we can easily see it.
+#
+function script_end {
+    #shellcheck disable=2181
+    EXIT_CODE=$?
+    if [[ ${EXIT_CODE} != 0 ]]; then
+        print_info "###########################################################################################"
+        print_info "                   EXITING WITH STATUS CODE ${EXIT_CODE}"
+        print_info "###########################################################################################"
+    fi
+    if [[ ${VERBOSE_COMMANDS:="false"} == "true" ]]; then
+        set +x
+    fi
+
+    if [[ ${#FILES_TO_CLEANUP_ON_EXIT[@]} -gt 0 ]]; then
+      rm -rf -- "${FILES_TO_CLEANUP_ON_EXIT[@]}"
+    fi
+
+    END_SCRIPT_TIME=$(date +%s)
+    RUN_SCRIPT_TIME=$((END_SCRIPT_TIME-START_SCRIPT_TIME))
+    if [[ ${BREEZE:=} != "true" ]]; then
+        print_info
+        print_info "Finished the script $(basename "$0")"
+        print_info "Elapsed time spent in the script: ${RUN_SCRIPT_TIME} seconds"
+        print_info "Exit code ${EXIT_CODE}"
+        print_info
+    fi
+    remove_cache_directory
+}
diff --git a/scripts/ci/_script_init.sh b/scripts/ci/libraries/_verbosity.sh
old mode 100755
new mode 100644
similarity index 51%
copy from scripts/ci/_script_init.sh
copy to scripts/ci/libraries/_verbosity.sh
index 514be24..9958635
--- a/scripts/ci/_script_init.sh
+++ b/scripts/ci/libraries/_verbosity.sh
@@ -16,19 +16,28 @@
 # specific language governing permissions and limitations
 # under the License.
 
-set -euo pipefail
+# In case "VERBOSE_COMMANDS" is set to "true" set -x is used to enable debugging
+function check_verbose_setup {
+    if [[ ${VERBOSE_COMMANDS:="false"} == "true" ]]; then
+        set -x
+    else
+        set +x
+    fi
+}
 
-# This should only be sourced from CI directory!
-# shellcheck source=scripts/ci/_utils.sh
-. "$( dirname "${BASH_SOURCE[0]}" )/_utils.sh"
+# In case "VERBOSE" is set to "true" (--verbose flag in Breeze) all docker commands run will be
+# printed before execution
+function verbose_docker {
+    if [[ ${VERBOSE:="false"} == "true" && ${VERBOSE_COMMANDS:=} != "true" ]]; then
+       # do not print echo if VERBOSE_COMMAND is set (set -x does it already)
+        echo "docker" "${@}"
+    fi
+    docker "${@}"
+}
 
-MY_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
-export MY_DIR
-
-initialize_common_environment
-
-basic_sanity_checks
-
-script_start
-
-trap script_end EXIT
+# Prints verbose information in case VERBOSE variable is set
+function print_info() {
+    if [[ ${VERBOSE:="false"} == "true" ]]; then
+        echo "$@"
+    fi
+}
diff --git a/scripts/ci/pre_commit_local_yml_mounts.sh b/scripts/ci/pre_commit_local_yml_mounts.sh
index d77a528..b638a08 100755
--- a/scripts/ci/pre_commit_local_yml_mounts.sh
+++ b/scripts/ci/pre_commit_local_yml_mounts.sh
@@ -30,8 +30,8 @@ trap 'rm -rf -- "${TMP_OUTPUT}" 2>/dev/null' EXIT
 
 LOCAL_YML_FILE="${MY_DIR}/docker-compose/local.yml"
 
-LEAD='      # START automatically generated volumes from LOCAL_MOUNTS in _utils.sh'
-TAIL='      # END automatically generated volumes from LOCAL_MOUNTS in _utils.sh'
+LEAD='      # START automatically generated volumes from LOCAL_MOUNTS in _local_mounts.sh'
+TAIL='      # END automatically generated volumes from LOCAL_MOUNTS in _local_mounts.sh'
 
 generate_local_mounts_list "      - ../../../"
 
diff --git a/tests/bats/bats_utils.bash b/tests/bats/bats_utils.bash
index bf89f64..7a29da9 100644
--- a/tests/bats/bats_utils.bash
+++ b/tests/bats/bats_utils.bash
@@ -17,4 +17,6 @@
 
 AIRFLOW_SOURCES=$(pwd)
 export AIRFLOW_SOURCES
-source scripts/ci/_utils.sh
+export SCRIPTS_CI_DIR=${AIRFLOW_SOURCES}/scripts/ci
+# shellcheck source=scripts/ci/_all_libs.sh
+source "${SCRIPTS_CI_DIR}/_all_libs.sh"


[airflow] 20/36: You can push with Breeze as separate command and to cache (#8976)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit eed7d335982404d3cda12f587799f2da4b176bc2
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Tue Jun 2 10:24:47 2020 +0200

    You can push with Breeze as separate command and to cache (#8976)
    
    Breeze had --push-images switch to also push images to repo
    but it was often needed to build and push images separately.
    
    We have now a possibility to push an already built image with
    separate push-image command instead and also you can choose
    to push to cache registry in GitHub rather than to DockerHub
    with --registry-cache switch.
    
    (cherry picked from commit a6216a760ca7449cfbf5aeee0f74caeda2f3f999)
---
 .github/workflows/ci.yml                       |   2 +-
 BREEZE.rst                                     | 148 ++++++++++++++++++-------
 breeze                                         | 106 +++++++++++++-----
 breeze-complete                                |  13 ++-
 scripts/ci/_utils.sh                           |  35 +++---
 scripts/ci/ci_fix_ownership.sh                 |   9 ++
 scripts/ci/in_container/_in_container_utils.sh |   2 +-
 7 files changed, 234 insertions(+), 81 deletions(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index e4f16e7..afa911c 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -32,7 +32,7 @@ env:
   VERBOSE: "true"
   UPGRADE_TO_LATEST_REQUIREMENTS: "false"
   PYTHON_MAJOR_MINOR_VERSION: 3.5
-  CACHE_REGISTRY: "docker.pkg.github.com"
+  ENABLE_REGISTRY_CACHE: "true"
   CACHE_IMAGE_PREFIX: ${{ github.repository }}
   CACHE_REGISTRY_USERNAME: ${{ github.actor }}
   CACHE_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
diff --git a/BREEZE.rst b/BREEZE.rst
index 262ddcd..0d90763 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -78,36 +78,46 @@ Docker Compose
 
 - **Permissions**: Configure to run the ``docker-compose`` command.
 
-Docker in WSL
--------------
+Docker in WSL 2
+---------------
+
+- **WSL 2 installation** :
+    Install WSL 2 and a Linux Distro (e.g. Ubuntu) see
+    `WSL 2 Installation Guide <https://docs.microsoft.com/en-us/windows/wsl/install-win10>`_ for details.
 
-- **WSL installation** :
-    `WSL Installation Guide <https://docs.microsoft.com/en-us/windows/wsl/install-win10>`_ for details.
+- **Docker Desktop installation** :
+    Install Docker Desktop for Windows. For Windows Home follow the
+    `Docker Windows Home Installation Guide <https://docs.docker.com/docker-for-windows/install-windows-home>`_.
+    For Windows Pro, Enterprise, or Education follow the
+    `Docker Windows Installation Guide <https://docs.docker.com/docker-for-windows/install/>`_.
 
-- **Docker installation** :
-    You should install docker in WSL.
-    follow `Docker Installtion Guide <https://docs.docker.com/install/linux/docker-ce/ubuntu/>`_
-    only docker-ce without docker-ce-cli containerd.io.
 - **Docker setting** :
-    You should expose Docker daemon,
+    WSL integration needs to be enabled
 
-.. image:: images/docker_expose_daemon.png
+.. image:: images/docker_wsl_integration.png
     :align: left
-    :alt: Docker expose daemon
-
-and set env variable DOCKER_HOST.
-
-.. code-block:: bash
-
-    echo "export DOCKER_HOST=tcp://localhost:2375" >> ~/.bashrc && source ~/.bashrc
-
-- **WSL problems** :
-  There is a mounting problem in docker because docker could not recognize ``/mnt/c``, ``/mnt/d`` driver path.
-  run this command in Windows Version 18.03+ and reboot Windows
-
-.. code-block:: bash
-
-    printf '[automount]\nroot = /\n options = "metadata"\n' >> /etc/wsl.conf
+    :alt: Docker WSL2 integration
+
+- **WSL 2 Filesystem Performance** :
+    Accessing the host Windows filesystem incurs a performance penalty,
+    it is therefore recommended to do development on the Linux filesystem.
+    E.g. Run ``cd ~`` and create a development folder in your Linux distro home
+    and git pull the Airflow repo there.
+
+- **WSL 2 Memory Usage** :
+    WSL 2 can consume a lot of memory under the process name "Vmmem". To reclaim
+    the memory after development you can:
+      * On the Linux distro clear cached memory: ``sudo sysctl -w vm.drop_caches=3``
+      * If no longer using Docker you can quit Docker Desktop
+        (right click system try icon and select "Quit Docker Desktop")
+      * If no longer using WSL you can shut it down on the Windows Host
+        with the following command: ``wsl --shutdown``
+
+- **Developing in WSL 2** :
+    You can use all the standard Linux command line utilities to develop on WSL 2.
+    Further VS Code supports developing in Windows but remotely executing in WSL.
+    If VS Code is installed on the Windows host system then in the WSL Linux Distro
+    you can run ``code .`` in the root directory of you Airflow repo to launch VS Code.
 
 Docker Images Used by Breeze
 ----------------------------
@@ -182,6 +192,8 @@ On macOS, 2GB of RAM are available for your Docker containers by default, but mo
 (4GB should be comfortable). For details see
 `Docker for Mac - Advanced tab <https://docs.docker.com/v17.12/docker-for-mac/#advanced-tab>`_.
 
+On Windows WSL 2 expect the Linux Disto and Docker containers to use 7 - 8 GB of RAM.
+
 Airflow Directory Structure inside Docker
 -----------------------------------------
 
@@ -655,6 +667,7 @@ This is the current syntax for  `./breeze <./breeze>`_:
     cleanup-image                            Cleans up the container image created
     exec                                     Execs into running breeze container in new terminal
     generate-requirements                    Generates pinned requirements for pip dependencies
+    push-image                               Pushes images to registry
     initialize-local-virtualenv              Initializes local virtualenv
     setup-autocomplete                       Sets up autocomplete for breeze
     stop                                     Stops the docker-compose environment
@@ -729,7 +742,7 @@ This is the current syntax for  `./breeze <./breeze>`_:
   breeze [FLAGS] build-image -- <EXTRA_ARGS>
 
         Builds docker image (CI or production) without entering the container. You can pass
-        aditional options to this command, such as '--force-build-image',
+        additional options to this command, such as '--force-build-image',
         '--force-pull-image' '--python' '--use-local-cache'' in order to modify build behaviour.
         You can also pass '--production-image' flag to build production image rather than CI image.
 
@@ -788,17 +801,23 @@ This is the current syntax for  `./breeze <./breeze>`_:
           Uses local cache to build images. No pulled images will be used, but results of local
           builds in the Docker cache are used instead.
 
-  -u, --push-images
-          After building - uploads the images to DockerHub
-          It is useful in case you use your own DockerHub user to store images and you want
-          to build them locally. Note that you need to use 'docker login' before you upload images.
-
   -D, --dockerhub-user
           DockerHub user used to pull, push and build images. Default: apache.
 
   -H, --dockerhub-repo
           DockerHub repository used to pull, push, build images. Default: airflow.
 
+  -c, --registry-cache
+          If registry cache is enabled, pulls and pushes are done from the registry cache in github.
+          You need to be logged in to the registry in order to be able to pull/push from it and you
+          need to be committer to push to airflow registry.
+
+  -G, --github-organisation
+          GitHub organisation used to pull, push images when cache is used. Default: apache.
+
+  -g, --github-repo
+          GitHub repository used to pull, push images when cache is used. Default: airflow.
+
   -v, --verbose
           Show verbose information about executed commands (enabled by default for running test).
           Note that you can further increase verbosity and see all the commands executed by breeze
@@ -876,6 +895,55 @@ This is the current syntax for  `./breeze <./breeze>`_:
   ####################################################################################################
 
 
+  Detailed usage for command: push-image
+
+  breeze [FLAGS] push-image -- <EXTRA_ARGS>
+
+        Pushes images to docker registry. You can push the images to DockerHub registry (default)
+        or to the GitHub cache registry (if --registry-cache flag is used).
+
+        For DockerHub pushes --dockerhub-user and --dockerhub-repo flags can be used to specify
+        the repository to push to. For GitHub repository --github-organisation and --github-repo
+        flags can be used for the same purpose.
+
+        You can also add --production-image flag to switch to production image (default is CI one)
+
+        Examples:
+
+        'breeze push-image' or
+        'breeze push-image --dockerhub-user user' to push to your private registry or
+        'breeze push-image --production-image' - to push production image or
+        'breeze push-image --registry-cache' - to push to GitHub cache or
+        'breeze push-image --registry-cache --github-organisation org' - for other organisation
+
+  Flags:
+
+  -D, --dockerhub-user
+          DockerHub user used to pull, push and build images. Default: apache.
+
+  -H, --dockerhub-repo
+          DockerHub repository used to pull, push, build images. Default: airflow.
+
+  -c, --registry-cache
+          If registry cache is enabled, pulls and pushes are done from the registry cache in github.
+          You need to be logged in to the registry in order to be able to pull/push from it and you
+          need to be committer to push to airflow registry.
+
+  -G, --github-organisation
+          GitHub organisation used to pull, push images when cache is used. Default: apache.
+
+  -g, --github-repo
+          GitHub repository used to pull, push images when cache is used. Default: airflow.
+
+  -v, --verbose
+          Show verbose information about executed commands (enabled by default for running test).
+          Note that you can further increase verbosity and see all the commands executed by breeze
+          by running 'export VERBOSE_COMMANDS="true"' before running breeze.
+
+
+  ####################################################################################################
+
+
   Detailed usage for command: initialize-local-virtualenv
 
   breeze [FLAGS] initialize-local-virtualenv -- <EXTRA_ARGS>
@@ -1333,12 +1401,7 @@ This is the current syntax for  `./breeze <./breeze>`_:
           builds in the Docker cache are used instead.
 
   ****************************************************************************************************
-   Flags for pushing Docker images (both CI and production)
-
-  -u, --push-images
-          After building - uploads the images to DockerHub
-          It is useful in case you use your own DockerHub user to store images and you want
-          to build them locally. Note that you need to use 'docker login' before you upload images.
+   Flags for pulling/pushing Docker images (both CI and production)
 
   -D, --dockerhub-user
           DockerHub user used to pull, push and build images. Default: apache.
@@ -1346,6 +1409,17 @@ This is the current syntax for  `./breeze <./breeze>`_:
   -H, --dockerhub-repo
           DockerHub repository used to pull, push, build images. Default: airflow.
 
+  -c, --registry-cache
+          If registry cache is enabled, pulls and pushes are done from the registry cache in github.
+          You need to be logged in to the registry in order to be able to pull/push from it and you
+          need to be committer to push to airflow registry.
+
+  -G, --github-organisation
+          GitHub organisation used to pull, push images when cache is used. Default: apache.
+
+  -g, --github-repo
+          GitHub repository used to pull, push images when cache is used. Default: airflow.
+
   ****************************************************************************************************
    Increase verbosity of the scripts
 
diff --git a/breeze b/breeze
index 75c00eb..fcea655 100755
--- a/breeze
+++ b/breeze
@@ -67,7 +67,7 @@ function setup_default_breeze_variables() {
 
     # We have different versions of images depending on the python version used. We keep up with the
     # Latest patch-level changes in Python (this is done automatically during CI builds) so we have
-    # To only take into account MAJOR and MINOR version of python. This variable keeps the major/mninor
+    # To only take into account MAJOR and MINOR version of python. This variable keeps the major/minor
     # version of python in X.Y format (3.6, 3.7 etc).
     export PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION:=$(read_from_file PYTHON_MAJOR_MINOR_VERSION)}"
 
@@ -158,7 +158,7 @@ function setup_default_breeze_variables() {
 
 # Initializes development-friendly virtualenv if you are already in such env. It installs all the necessary
 # packages from PyPI and it case of problems it provides useful hints on what prerequisites should be
-# installed. It also removes and resets the existing AIRFOW_HOME installation to make sure that you
+# installed. It also removes and resets the existing AIRFLOW_HOME installation to make sure that you
 # have it synchronized with the version of airflow installed. It resets the airflow's sqlite database to
 # a clean state. You can use this function if your virtualenv is broken, to clean it up
 function initialize_virtualenv() {
@@ -792,13 +792,24 @@ function parse_arguments() {
           echo
           export FORWARD_CREDENTIALS="true"
           shift 1 ;;
-        -u|--push-images)
+        -c|--registry-cache)
           echo
-          echo "Pushing images to DockerHub"
+          echo "Use cache for the container registry"
           echo
-          export PUSH_IMAGES="true"
-          export FORCE_BUILD_IMAGES="true"
+          export ENABLE_REGISTRY_CACHE="true"
           shift ;;
+        -G|--github-organisation)
+          echo
+          echo "GitHub organisation"
+          echo
+          export GITHUB_ORGANISATION="${2}"
+          shift 2;;
+        -g|--github-repo)
+          echo
+          echo "GitHub repository"
+          echo
+          export GITHUB_REPOSITORY="${2}"
+          shift 2;;
         --)
           shift ;
           break ;;
@@ -860,6 +871,15 @@ function parse_arguments() {
         generate-requirements)
           LAST_SUBCOMMAND="${1}"
           COMMAND_TO_RUN="perform_generate_requirements"
+          # if you want to  generate requirement - you want to build the image too :)
+          export FORCE_ANSWER_TO_QUESTIONS="yes"
+          # and assume you want to build it no matter if it is needed
+          export FORCE_BUILD_IMAGES="true"
+          shift ;;
+        push-image)
+          LAST_SUBCOMMAND="${1}"
+          COMMAND_TO_RUN="perform_push_image"
+          SKIP_CHECK_REMOTE_IMAGE="true"
           shift ;;
         initialize-local-virtualenv)
           LAST_SUBCOMMAND="${1}"
@@ -1050,6 +1070,7 @@ function prepare_usage() {
     export USAGE_FLAGS="Shows all breeze's flags"
     export USAGE_GENERATE_REQUIREMENTS="Generates pinned requirements for pip dependencies"
     export USAGE_INITIALIZE_LOCAL_VIRTUALENV="Initializes local virtualenv"
+    export USAGE_PUSH_IMAGE="Pushes images to registry"
     export USAGE_SETUP_AUTOCOMPLETE="Sets up autocomplete for breeze"
     export USAGE_STOP="Stops the docker-compose environment"
     export USAGE_RESTART="Stops the docker-compose environment including DB cleanup"
@@ -1100,7 +1121,7 @@ $(flag_footer)
     # shellcheck disable=SC2089
     DETAILED_USAGE_BUILD_IMAGE="
       Builds docker image (CI or production) without entering the container. You can pass
-      aditional options to this command, such as '--force-build-image',
+      additional options to this command, such as '--force-build-image',
       '--force-pull-image' '--python' '--use-local-cache'' in order to modify build behaviour.
       You can also pass '--production-image' flag to build production image rather than CI image.
 
@@ -1109,7 +1130,7 @@ $(flag_airflow_variants)
 $(flag_choose_different_airflow_version)
 $(flag_production_image)
 $(flag_build_docker_images)
-$(flag_push_docker_images)
+$(flag_pull_push_docker_images)
 $(flag_verbosity)
 "
     # shellcheck disable=SC2090
@@ -1187,6 +1208,31 @@ $(flag_airflow_variants)
 "
     # shellcheck disable=SC2090
     export DETAILED_USAGE_INITIALIZE_LOCAL_VIRTUALENV
+    # shellcheck disable=SC2089
+    DETAILED_USAGE_PUSH_IMAGE="
+      Pushes images to docker registry. You can push the images to DockerHub registry (default)
+      or to the GitHub cache registry (if --registry-cache flag is used).
+
+      For DockerHub pushes --dockerhub-user and --dockerhub-repo flags can be used to specify
+      the repository to push to. For GitHub repository --github-organisation and --github-repo
+      flags can be used for the same purpose.
+
+      You can also add --production-image flag to switch to production image (default is CI one)
+
+      Examples:
+
+      '${CMDNAME} push-image' or
+      '${CMDNAME} push-image --dockerhub-user user' to push to your private registry or
+      '${CMDNAME} push-image --production-image' - to push production image or
+      '${CMDNAME} push-image --registry-cache' - to push to GitHub cache or
+      '${CMDNAME} push-image --registry-cache --github-organisation org' - for other organisation
+
+Flags:
+$(flag_pull_push_docker_images)
+$(flag_verbosity)
+"
+    # shellcheck disable=SC2090
+    export DETAILED_USAGE_PUSH_IMAGE
     export DETAILED_USAGE_SETUP_AUTOCOMPLETE="
       Sets up autocomplete for breeze commands. Once you do it you need to re-enter the bash
       shell and when typing breeze command <TAB> will provide autocomplete for
@@ -1548,19 +1594,26 @@ ${FORMATTED_DEFAULT_PROD_EXTRAS}
 "
 }
 
-# Prints flags controlling docker push process
-function flag_push_docker_images() {
+# Prints flags controlling docker pull and push process
+function flag_pull_push_docker_images() {
        echo "
--u, --push-images
-        After building - uploads the images to DockerHub
-        It is useful in case you use your own DockerHub user to store images and you want
-        to build them locally. Note that you need to use 'docker login' before you upload images.
-
 -D, --dockerhub-user
         DockerHub user used to pull, push and build images. Default: ${_BREEZE_DEFAULT_DOCKERHUB_USER:=}.
 
 -H, --dockerhub-repo
         DockerHub repository used to pull, push, build images. Default: ${_BREEZE_DEFAULT_DOCKERHUB_REPO:=}.
+
+-c, --registry-cache
+        If registry cache is enabled, pulls and pushes are done from the registry cache in github.
+        You need to be logged in to the registry in order to be able to pull/push from it and you
+        need to be committer to push to airflow registry.
+
+-G, --github-organisation
+        GitHub organisation used to pull, push images when cache is used. Default: ${_BREEZE_DEFAULT_GITHUB_ORGANISATION:=}.
+
+-g, --github-repo
+        GitHub repository used to pull, push images when cache is used. Default: ${_BREEZE_DEFAULT_GITHUB_REPO:=}.
+
 "
 }
 
@@ -1613,8 +1666,8 @@ $(print_star_line)
 $(flag_build_docker_images)
 
 $(print_star_line)
- Flags for pushing Docker images (both CI and production)
-$(flag_push_docker_images)
+ Flags for pulling/pushing Docker images (both CI and production)
+$(flag_pull_push_docker_images)
 
 $(print_star_line)
  Increase verbosity of the scripts
@@ -1822,7 +1875,7 @@ function run_static_checks {
     fi
 }
 
-# Runs Build before a comman - it will check and if needed rebuild necessary image, depending on the
+# Runs Build before a command - it will check and if needed rebuild necessary image, depending on the
 # command chosen
 function run_build_command {
     case "${COMMAND_TO_RUN}" in
@@ -1838,7 +1891,7 @@ function run_build_command {
                 rebuild_ci_image_if_needed
             fi
             ;;
-        build_docs|perform_static_checks|perform_generate_requirements)
+        build_docs|perform_static_checks|perform_generate_requirements|perform_push_image)
             prepare_ci_build
             rebuild_ci_image_if_needed
             ;;
@@ -1866,6 +1919,7 @@ function run_build_command {
           echo >&2
           echo >&2 "ERROR: Unknown command to run ${COMMAND_TO_RUN}"
           echo >&2
+          exit 1
           ;;
     esac
 }
@@ -1923,13 +1977,6 @@ function run_breeze_command {
             run_static_checks "$@"
             ;;
         build_image)
-            if [[ ${PUSH_IMAGES} == "true" ]]; then
-                if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
-                    push_prod_images
-                else
-                    push_ci_image
-                fi
-            fi
             ;;
         cleanup_image)
             remove_images
@@ -1937,6 +1984,13 @@ function run_breeze_command {
         perform_generate_requirements)
             run_generate_requirements
             ;;
+        perform_push_image)
+            if [[ ${PRODUCTION_IMAGE} == "true" ]]; then
+                push_prod_images
+            else
+                push_ci_image
+            fi
+            ;;
         perform_initialize_local_virtualenv)
             initialize_virtualenv
             ;;
diff --git a/breeze-complete b/breeze-complete
index 8afe550..de39a83 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -79,6 +79,8 @@ EOF
 
 _BREEZE_DEFAULT_DOCKERHUB_USER="apache"
 _BREEZE_DEFAULT_DOCKERHUB_REPO="airflow"
+_BREEZE_DEFAULT_GITHUB_ORGANISATION="apache"
+_BREEZE_DEFAULT_GITHUB_REPO="airflow"
 
 _BREEZE_SHORT_OPTIONS="
 h p: b: i:
@@ -87,7 +89,7 @@ s x r
 l a: t: d:
 v y n q f
 F P I E: C L
-D: R: u
+D: R: c g: G:
 "
 
 _BREEZE_LONG_OPTIONS="
@@ -97,7 +99,7 @@ kind-cluster-start kind-cluster-stop kind-cluster-recreate
 skip-mounting-local-sources install-airflow-version: install-airflow-reference: db-reset
 verbose assume-yes assume-no assume-quit forward-credentials
 force-build-images force-pull-images production-image extras: force-clean-images use-local-cache
-dockerhub-user: dockerhub-repo: push-images
+dockerhub-user: dockerhub-repo: registry-cache github-organisation: github-repo:
 postgres-version: mysql-version:
 additional-extras: additional-python-deps:
 "
@@ -109,6 +111,7 @@ build-image
 cleanup-image
 exec
 generate-requirements
+push-image
 initialize-local-virtualenv
 setup-autocomplete
 stop
@@ -179,6 +182,12 @@ function _get_known_values_breeze() {
     -R | --dockerhub-repo)
         _BREEZE_KNOWN_VALUES="${_BREEZE_DEFAULT_DOCKERHUB_REPO}"
         ;;
+    -G | --github-organisation)
+        _BREEZE_KNOWN_VALUES="${_BREEZE_DEFAULT_GITHUB_ORGANISATION}"
+        ;;
+    -g | --github-repo)
+        _BREEZE_KNOWN_VALUES="${_BREEZE_DEFAULT_GITHUB_REPO}"
+        ;;
     *)
         _BREEZE_KNOWN_VALUES=""
         ;;
diff --git a/scripts/ci/_utils.sh b/scripts/ci/_utils.sh
index 6c0204c..4ba7d83 100644
--- a/scripts/ci/_utils.sh
+++ b/scripts/ci/_utils.sh
@@ -83,6 +83,11 @@ function initialize_common_environment {
     # Default branch name for triggered builds is the one configured in default branch
     export BRANCH_NAME=${BRANCH_NAME:=${DEFAULT_BRANCH}}
 
+    export GITHUB_ORGANISATION=${GITHUB_ORGANISATION:="apache"}
+    export GITHUB_REPO=${GITHUB_REPO:="airflow"}
+    export CACHE_REGISTRY=${CACHE_REGISTRY:="docker.pkg.github.com"}
+    export ENABLE_REGISTRY_CACHE=${ENABLE_REGISTRY_CACHE:="false"}
+
     # Default port numbers for forwarded ports
     export WEBSERVER_HOST_PORT=${WEBSERVER_HOST_PORT:="28080"}
     export POSTGRES_HOST_PORT=${POSTGRES_HOST_PORT:="25433"}
@@ -92,10 +97,6 @@ function initialize_common_environment {
     export POSTGRES_VERSION=${POSTGRES_VERSION:="9.6"}
     export MYSQL_VERSION=${MYSQL_VERSION:="5.6"}
 
-    # Do not push images by default (push them directly from the build script on Dockerhub or when
-    # --push-images flag is specified
-    export PUSH_IMAGES=${PUSH_IMAGES:="false"}
-
     # Whether base python images should be pulled from cache
     export PULL_PYTHON_BASE_IMAGES_FROM_CACHE=${PULL_PYTHON_BASE_IMAGES_FROM_CACHE:="true"}
 
@@ -1455,11 +1456,14 @@ function prepare_ci_build() {
     export AIRFLOW_CI_LOCAL_MANIFEST_IMAGE="local/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}-manifest"
     export AIRFLOW_CI_REMOTE_MANIFEST_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}-manifest"
     export AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}"
-    if [[ ${CACHE_REGISTRY=""} != "" ]]; then
-        echo "${CACHE_REGISTRY_PASSWORD}" | docker login \
-            --username "${CACHE_REGISTRY_USERNAME}" \
-            --password-stdin \
-            "${CACHE_REGISTRY}"
+    if [[ ${ENABLE_REGISTRY_CACHE="false"} == "true" ]]; then
+        if [[ ${CACHE_REGISTRY_PASSWORD:=} != "" ]]; then
+            echo "${CACHE_REGISTRY_PASSWORD}" | docker login \
+                --username "${CACHE_REGISTRY_USERNAME}" \
+                --password-stdin \
+                "${CACHE_REGISTRY}"
+        fi
+        export CACHE_IMAGE_PREFIX=${CACHE_IMAGE_PREFX:=${GITHUB_ORGANISATION}/${GITHUB_REPO}}
         export CACHED_AIRFLOW_CI_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_CI_BASE_TAG}"
         export CACHED_PYTHON_BASE_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
     else
@@ -1543,11 +1547,14 @@ function prepare_prod_build() {
     export ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS:=""}"
     export AIRFLOW_IMAGE="${AIRFLOW_PROD_IMAGE}"
 
-    if [[ ${CACHE_REGISTRY=""} != "" ]]; then
-        echo "${CACHE_REGISTRY_PASSWORD}" | docker login \
-            --username "${CACHE_REGISTRY_USERNAME}" \
-            --password-stdin \
-            "${CACHE_REGISTRY}"
+    if [[ ${ENABLE_REGISTRY_CACHE="false"} == "true" ]]; then
+        if [[ ${CACHE_REGISTRY_PASSWORD:=} != "" ]]; then
+            echo "${CACHE_REGISTRY_PASSWORD}" | docker login \
+                --username "${CACHE_REGISTRY_USERNAME}" \
+                --password-stdin \
+                "${CACHE_REGISTRY}"
+        fi
+        export CACHE_IMAGE_PREFIX=${CACHE_IMAGE_PREFX:=${GITHUB_ORGANISATION}/${GITHUB_REPO}}
         export CACHED_AIRFLOW_PROD_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_PROD_BASE_TAG}"
         export CACHED_AIRFLOW_PROD_BUILD_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_PROD_BASE_TAG}-build"
         export CACHED_PYTHON_BASE_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
diff --git a/scripts/ci/ci_fix_ownership.sh b/scripts/ci/ci_fix_ownership.sh
index b47d83a..775cc58 100755
--- a/scripts/ci/ci_fix_ownership.sh
+++ b/scripts/ci/ci_fix_ownership.sh
@@ -19,6 +19,7 @@
 #
 # Fixes ownership for files created inside container (files owned by root will be owned by host user)
 #
+set -x
 
 export PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:-3.5}
 
@@ -28,6 +29,14 @@ export PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:-3.5}
 export AIRFLOW_CI_IMAGE=\
 ${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci
 
+export AIRFLOW_IMAGE=${AIRFLOW_CI_IMAGE}
+export WEBSERVER_HOST_PORT=28080
+HOST_USER_ID="$(id -ur)"
+HOST_GROUP_ID="$(id -gr)"
+
+export HOST_USER_ID
+export HOST_GROUP_ID
+
 docker-compose \
     -f "${MY_DIR}/docker-compose/base.yml" \
     -f "${MY_DIR}/docker-compose/local.yml" \
diff --git a/scripts/ci/in_container/_in_container_utils.sh b/scripts/ci/in_container/_in_container_utils.sh
index be386e0..9b9814a 100644
--- a/scripts/ci/in_container/_in_container_utils.sh
+++ b/scripts/ci/in_container/_in_container_utils.sh
@@ -93,7 +93,7 @@ function in_container_cleanup_pycache() {
 #
 function in_container_fix_ownership() {
     set +o pipefail
-    sudo find . -user root -print0 \
+    sudo find "${AIRFLOW_SOURCES}" -print0 -user root \
     | sudo xargs --null chown -v "${HOST_USER_ID}.${HOST_GROUP_ID}" --no-dereference >/dev/null 2>&1
     set -o pipefail
 }


[airflow] 24/36: Kubernetes Cluster is started on host not in the container (#8265)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 8dacc27fb2d75493063e0a0f1e3ee4951c366d8d
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Wed Jun 3 20:58:38 2020 +0200

    Kubernetes Cluster is started on host not in the container (#8265)
    
    Tests requiring Kubernetes Cluster are now moved out of
    the regular CI tests and moved to "kubernetes_tests" folder
    so that they can be run entirely on host without having
    the CI image built at all. They use production image
    to run the tests on KinD cluster and we add tooling
    to start/stop/deploy the application to the KinD cluster
    automatically - for both CI testing and local development.
    
    This is a pre-requisite to convert the tests to convert the
    tests to use the official Helm Chart and Docker images or
    Apache Airflow.
    
    It closes #8782
    
    (cherry picked from commit ff5dcccbbd49e7a4632f93fa915565ac31730110)
---
 .github/workflows/ci.yml                           |  37 +-
 .gitignore                                         |   4 +-
 BREEZE.rst                                         | 223 +++++++--
 Dockerfile.ci                                      |  17 +-
 IMAGES.rst                                         |   4 -
 README.md                                          |   2 +
 TESTING.rst                                        | 230 +++++++---
 airflow/kubernetes/refresh_config.py               |   1 -
 airflow/kubernetes/volume_mount.py                 |  36 +-
 breeze                                             | 211 +++++----
 breeze-complete                                    |  12 +-
 .../kubernetes => kubernetes_tests}/__init__.py    |   0
 .../test_kubernetes_executor.py                    | 101 +++--
 .../test_kubernetes_pod_operator.py                | 432 +++++++++++++-----
 scripts/ci/_utils.sh                               | 503 ++++++++++++++++++++-
 ...bootstrap.sh => ci_deploy_app_to_kubernetes.sh} |  25 +-
 .../run_ci_tests.sh => ci_load_image_to_kind.sh}   |  40 +-
 ...rap.sh => ci_perform_kind_cluster_operation.sh} |  15 +-
 scripts/ci/ci_run_airflow_testing.sh               |  33 +-
 scripts/ci/ci_run_kubernetes_tests.sh              | 105 +++++
 scripts/ci/docker-compose/base.yml                 |   1 -
 scripts/ci/docker-compose/local.yml                |   1 +
 scripts/ci/in_container/_in_container_utils.sh     |  28 --
 scripts/ci/in_container/entrypoint_ci.sh           |  73 +--
 .../ci/in_container/kubernetes/app/deploy_app.sh   | 210 ---------
 .../kubernetes/docker/rebuild_airflow_image.sh     |  74 ---
 .../in_container/kubernetes/setup_kind_cluster.sh  | 187 --------
 scripts/ci/in_container/run_ci_tests.sh            |   4 -
 .../kubernetes/app/postgres.yaml                   |   0
 .../{in_container => }/kubernetes/app/secrets.yaml |   0
 .../kubernetes/app/templates/airflow.template.yaml |  66 ++-
 .../app/templates/configmaps.template.yaml         |   0
 .../app/templates/init_git_sync.template.yaml      |  38 +-
 .../{in_container => }/kubernetes/app/volumes.yaml |   0
 .../docker/airflow-test-env-init-dags.sh}          |  21 +-
 .../docker/airflow-test-env-init-db.sh}            |  39 +-
 scripts/ci/kubernetes/docker/bootstrap.sh          |  74 +++
 .../kubernetes/kind-cluster-conf.yaml              |   0
 tests/conftest.py                                  |  37 --
 .../kubernetes/operators/test_kubernetes_pod.py    | 134 ++++++
 tests/runtime/__init__.py                          |  18 -
 41 files changed, 1909 insertions(+), 1127 deletions(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index afa911c..c48a7ac 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -15,7 +15,6 @@
 # specific language governing permissions and limitations
 # under the License.
 #
-
 ---
 name: CI Build
 on:
@@ -23,7 +22,9 @@ on:
     branches: ['master', 'v1-10-test', 'v1-10-stable']
   pull_request:
     branches: ['master', 'v1-10-test', 'v1-10-stable']
+
 env:
+
   MOUNT_LOCAL_SOURCES: "false"
   FORCE_ANSWER_TO_QUESTIONS: "yes"
   SKIP_CHECK_REMOTE_IMAGE: "true"
@@ -36,6 +37,7 @@ env:
   CACHE_IMAGE_PREFIX: ${{ github.repository }}
   CACHE_REGISTRY_USERNAME: ${{ github.actor }}
   CACHE_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
+
 jobs:
 
   static-checks:
@@ -51,9 +53,9 @@ jobs:
         with:
           python-version: '3.x'
       - name: Cache pre-commit env
-        uses: actions/cache@v1
+        uses: actions/cache@v2
         env:
-          cache-name: cache-pre-commit-epoch2
+          cache-name: cache-pre-commit-v2
         with:
           path: ~/.cache/pre-commit
           key: ${{ env.cache-name }}-${{ github.job }}-${{ hashFiles('.pre-commit-config.yaml') }}
@@ -98,15 +100,15 @@ jobs:
 
   tests-kubernetes:
     timeout-minutes: 80
-    name: "kubernetes-test-${{matrix.kube-mode}}-kube"
+    name: "K8s: ${{matrix.kube-mode}} ${{matrix.python-version}} ${{matrix.kubernetes-version}}"
     runs-on: ubuntu-latest
     needs: [static-checks]
     strategy:
       matrix:
+        python-version: [3.6, 3.7]
         kube-mode:
-          - persistent_mode
-#          #TODO These tests will be unblocked when k8s tests run on host
-#          - git_mode
+          - image
+          - git
         kubernetes-version:
           - "v1.15.3"
       fail-fast: false
@@ -115,12 +117,14 @@ jobs:
       TEST_TYPE: ${{ matrix.test-type }}
       RUN_TESTS: "true"
       CI_JOB_TYPE: "Tests"
-      PYTHON_MAJOR_MINOR_VERSION: "3.6"
       SKIP_CI_IMAGE_CHECK: "true"
       RUNTIME: "kubernetes"
       ENABLE_KIND_CLUSTER: "true"
+      PYTHON_MAJOR_MINOR_VERSION: "${{ matrix.python-version }}"
       KUBERNETES_MODE: "${{ matrix.kube-mode }}"
       KUBERNETES_VERSION: "${{ matrix.kubernetes-version }}"
+    # For pull requests only run tests when python files changed
+    if: needs.pyfiles.outputs.count != '0' || github.event_name != 'pull_request'
     steps:
       - uses: actions/checkout@master
       - uses: actions/setup-python@v1
@@ -128,12 +132,22 @@ jobs:
           python-version: '3.x'
       - name: "Free space"
         run: ./scripts/ci/ci_free_space_on_ci.sh
-      - name: "Build CI image ${{ matrix.python-version }}"
-        run: ./scripts/ci/ci_prepare_image_on_ci.sh
       - name: "Build PROD image ${{ matrix.python-version }}"
         run: ./scripts/ci/ci_build_production_images.sh
+      - name: "Setup KinD cluster"
+        run: ./scripts/ci/ci_perform_kind_cluster_operation.sh start
+      - name: "Deploy app to cluster"
+        run: ./scripts/ci/ci_deploy_app_to_kubernetes.sh
+      - name: Cache virtualenv for kubernetes testing
+        uses: actions/cache@v2
+        env:
+          cache-name: cache-kubernetes-tests-virtualenv-v2
+        with:
+          path: .build/.kubernetes_venv
+          key: "${{ env.cache-name }}-${{ github.job }}-\
+${{ hashFiles('requirements/requirements-python${{matrix.python-version}}.txt') }}"
       - name: "Tests"
-        run: ./scripts/ci/ci_run_airflow_testing.sh
+        run: ./scripts/ci/ci_run_kubernetes_tests.sh
 
   pyfiles:
     timeout-minutes: 10
@@ -149,7 +163,6 @@ jobs:
           ./scripts/ci/ci_count_changed_files.sh ${GITHUB_SHA} '\.py$|.github/workflows/|^Dockerfile'
           echo "::set-output name=count::$?"
         id: pyfiles
-
   tests-postgres:
     timeout-minutes: 80
     name: "${{matrix.test-type}}:Pg${{matrix.postgres-version}},Py${{matrix.python-version}}"
diff --git a/.gitignore b/.gitignore
index 42c4dbe..4eca815 100644
--- a/.gitignore
+++ b/.gitignore
@@ -154,8 +154,8 @@ rat-results.txt
 # Kubernetes generated templated files
 *.generated
 *.tar.gz
-scripts/ci/in_container/kubernetes/kube/.generated/airflow.yaml
-scripts/ci/in_container/kubernetes/docker/requirements.txt
+scripts/ci/kubernetes/kube/.generated/airflow.yaml
+scripts/ci/kubernetes/docker/requirements.txt
 
 # Node & Webpack Stuff
 *.entry.js
diff --git a/BREEZE.rst b/BREEZE.rst
index 0d90763..7648b9d 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -245,7 +245,7 @@ Manage environments - CI (default) or Production - if ``--production-image`` fla
     * Build docker images with ``breeze build-image`` command
     * Enter interactive shell when no command are specified (default behaviour)
     * Join running interactive shell with ``breeze exec`` command
-    * Start Kind Kubernetes cluster for Kubernetes tests if ``--start-kind-cluster`` flag is specified
+    * Start/stops/restarts Kind Kubernetes cluster with ``kind-cluster`` command
     * Stop running interactive environment with ``breeze stop`` command
     * Restart running interactive environment with ``breeze restart`` command
     * Optionally reset database if specified as extra ``--db-reset`` flag
@@ -669,6 +669,7 @@ This is the current syntax for  `./breeze <./breeze>`_:
     generate-requirements                    Generates pinned requirements for pip dependencies
     push-image                               Pushes images to registry
     initialize-local-virtualenv              Initializes local virtualenv
+    kind-cluster                             Manages KinD cluster on the host
     setup-autocomplete                       Sets up autocomplete for breeze
     stop                                     Stops the docker-compose environment
     restart                                  Stops the docker-compose environment including DB cleanup
@@ -679,6 +680,7 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
     docker-compose                <ARG>      Executes specified docker-compose command
     execute-command               <ARG>      Executes specified command in the container
+    kind-cluster                  <ARG>      Manages KinD cluster on the host
     static-check                  <ARG>      Performs selected static check for changed files
     static-check-all-files        <ARG>      Performs selected static check for all files
     test-target                   <ARG>      Runs selected test target in the container
@@ -698,7 +700,8 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: shell
 
-  breeze [FLAGS] shell -- <EXTRA_ARGS>
+
+  breeze shell [FLAGS] -- <EXTRA_ARGS>
 
         This is default subcommand if no subcommand is used.
 
@@ -725,7 +728,8 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: build-docs
 
-  breeze [FLAGS] build-docs -- <EXTRA_ARGS>
+
+  breeze build-docs
 
         Builds Airflow documentation. The documentation is build inside docker container - to
         maintain the same build environment for everyone. Appropriate sources are mapped from
@@ -739,7 +743,8 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: build-image
 
-  breeze [FLAGS] build-image -- <EXTRA_ARGS>
+
+  breeze build-image [FLAGS]
 
         Builds docker image (CI or production) without entering the container. You can pass
         additional options to this command, such as '--force-build-image',
@@ -829,7 +834,8 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: cleanup-image
 
-  breeze [FLAGS] cleanup-image -- <EXTRA_ARGS>
+
+  breeze cleanup-image [FLAGS]
 
         Removes the breeze-related images created in your local docker image cache. This will
         not reclaim space in docker cache. You need to 'docker system prune' (optionally
@@ -857,7 +863,8 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: exec
 
-  breeze [FLAGS] exec -- <EXTRA_ARGS>
+
+  breeze exec
 
         Execs into interactive shell to an already running container. The container mus be started
         already by breeze shell command. If you are not familiar with tmux, this is the best
@@ -870,7 +877,8 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: generate-requirements
 
-  breeze [FLAGS] generate-requirements -- <EXTRA_ARGS>
+
+  breeze generate-requirements [FLAGS]
 
         Generates pinned requirements from setup.py. Those requirements are generated in requirements
         directory - separately for different python version. Those requirements are used to run
@@ -897,7 +905,8 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: push-image
 
-  breeze [FLAGS] push-image -- <EXTRA_ARGS>
+
+  breeze push_image [FLAGS]
 
         Pushes images to docker registry. You can push the images to DockerHub registry (default)
         or to the GitHub cache registry (if --registry-cache flag is used).
@@ -946,7 +955,8 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: initialize-local-virtualenv
 
-  breeze [FLAGS] initialize-local-virtualenv -- <EXTRA_ARGS>
+
+  breeze initialize-local-virtualenv [FLAGS]
 
         Initializes locally created virtualenv installing all dependencies of Airflow
         taking into account the frozen requirements from requirements folder.
@@ -966,9 +976,70 @@ This is the current syntax for  `./breeze <./breeze>`_:
   ####################################################################################################
 
 
+  Detailed usage for command: kind-cluster
+
+
+  breeze kind-cluster [FLAGS] OPERATION
+
+        Manages host-side Kind Kubernetes cluster that is used to run Kubernetes integration tests.
+        It allows to start/stop/restart/status the Kind Kubernetes cluster and deploy Airflow to it.
+        This enables you to run tests inside the breeze environment with latest airflow images loaded.
+        Note that in case of deploying airflow, the first step is to rebuild the image and loading it
+        to the cluster so you can also pass appropriate build image flags that will influence
+        rebuilding the production image. Operation is one of:
+
+                 start stop restart status deploy test
+
+  Flags:
+
+  -p, --python <PYTHON_MAJOR_MINOR_VERSION>
+          Python version used for the image. This is always major/minor version.
+          One of:
+
+                 2.7 3.5 3.6 3.7
+
+  -F, --force-build-images
+          Forces building of the local docker images. The images are rebuilt
+          automatically for the first time or when changes are detected in
+          package-related files, but you can force it using this flag.
+
+  -P, --force-pull-images
+          Forces pulling of images from DockerHub before building to populate cache. The
+          images are pulled by default only for the first time you run the
+          environment, later the locally build images are used as cache.
+
+  -E, --extras
+          Extras to pass to build images The default are different for CI and production images:
+
+          CI image:
+                 devel_ci
+
+          Production image:
+                 async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,
+                 ssh,statsd,virtualenv
+
+  --additional-extras
+          Additional extras to pass to build images The default is no additional extras.
+
+  --additional-python-deps
+          Additional python dependencies to use when building the images.
+
+  -C, --force-clean-images
+          Force build images with cache disabled. This will remove the pulled or build images
+          and start building images from scratch. This might take a long time.
+
+  -L, --use-local-cache
+          Uses local cache to build images. No pulled images will be used, but results of local
+          builds in the Docker cache are used instead.
+
+
+  ####################################################################################################
+
+
   Detailed usage for command: setup-autocomplete
 
-  breeze [FLAGS] setup-autocomplete -- <EXTRA_ARGS>
+
+  breeze setup-autocomplete
 
         Sets up autocomplete for breeze commands. Once you do it you need to re-enter the bash
         shell and when typing breeze command <TAB> will provide autocomplete for
@@ -980,7 +1051,8 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: stop
 
-  breeze [FLAGS] stop -- <EXTRA_ARGS>
+
+  breeze stop
 
         Brings down running docker compose environment. When you start the environment, the docker
         containers will continue running so that startup time is shorter. But they take quite a lot of
@@ -992,7 +1064,8 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: restart
 
-  breeze [FLAGS] restart -- <EXTRA_ARGS>
+
+  breeze restart [FLAGS]
 
         Restarts running docker compose environment. When you restart the environment, the docker
         containers will be restarted. That includes cleaning up the databases. This is
@@ -1008,7 +1081,8 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: toggle-suppress-cheatsheet
 
-  breeze [FLAGS] toggle-suppress-cheatsheet -- <EXTRA_ARGS>
+
+  breeze toggle-suppress-cheatsheet
 
         Toggles on/off cheatsheet displayed before starting bash shell.
 
@@ -1018,7 +1092,8 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: toggle-suppress-asciiart
 
-  breeze [FLAGS] toggle-suppress-asciiart -- <EXTRA_ARGS>
+
+  breeze toggle-suppress-asciiart
 
         Toggles on/off asciiart displayed before starting bash shell.
 
@@ -1028,7 +1103,8 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: docker-compose
 
-  breeze [FLAGS] docker-compose <DOCKER_COMPOSE_COMMAND> -- <EXTRA_ARGS>
+
+  breeze docker-compose [FLAGS] COMMAND -- <EXTRA_ARGS>
 
         Run docker-compose command instead of entering the environment. Use 'help' as command
         to see available commands. The <EXTRA_ARGS> passed after -- are treated
@@ -1073,7 +1149,8 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: execute-command
 
-  breeze [FLAGS] execute-command -- <EXTRA_ARGS>
+
+  breeze execute-command [FLAGS] COMMAND -- <EXTRA_ARGS>
 
         Run chosen command instead of entering the environment. The command is run using
         'bash -c "<command with args>" if you need to pass arguments to your command, you need
@@ -1118,9 +1195,70 @@ This is the current syntax for  `./breeze <./breeze>`_:
   ####################################################################################################
 
 
+  Detailed usage for command: kind-cluster
+
+
+  breeze kind-cluster [FLAGS] OPERATION
+
+        Manages host-side Kind Kubernetes cluster that is used to run Kubernetes integration tests.
+        It allows to start/stop/restart/status the Kind Kubernetes cluster and deploy Airflow to it.
+        This enables you to run tests inside the breeze environment with latest airflow images loaded.
+        Note that in case of deploying airflow, the first step is to rebuild the image and loading it
+        to the cluster so you can also pass appropriate build image flags that will influence
+        rebuilding the production image. Operation is one of:
+
+                 start stop restart status deploy test
+
+  Flags:
+
+  -p, --python <PYTHON_MAJOR_MINOR_VERSION>
+          Python version used for the image. This is always major/minor version.
+          One of:
+
+                 2.7 3.5 3.6 3.7
+
+  -F, --force-build-images
+          Forces building of the local docker images. The images are rebuilt
+          automatically for the first time or when changes are detected in
+          package-related files, but you can force it using this flag.
+
+  -P, --force-pull-images
+          Forces pulling of images from DockerHub before building to populate cache. The
+          images are pulled by default only for the first time you run the
+          environment, later the locally build images are used as cache.
+
+  -E, --extras
+          Extras to pass to build images The default are different for CI and production images:
+
+          CI image:
+                 devel_ci
+
+          Production image:
+                 async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,
+                 ssh,statsd,virtualenv
+
+  --additional-extras
+          Additional extras to pass to build images The default is no additional extras.
+
+  --additional-python-deps
+          Additional python dependencies to use when building the images.
+
+  -C, --force-clean-images
+          Force build images with cache disabled. This will remove the pulled or build images
+          and start building images from scratch. This might take a long time.
+
+  -L, --use-local-cache
+          Uses local cache to build images. No pulled images will be used, but results of local
+          builds in the Docker cache are used instead.
+
+
+  ####################################################################################################
+
+
   Detailed usage for command: static-check
 
-  breeze [FLAGS] static-check <STATIC_CHECK> -- <EXTRA_ARGS>
+
+  breeze static-check [FLAGS] STATIC_CHECK
 
         Run selected static checks for currently changed files. You should specify static check that
         you would like to run or 'all' to run all checks. One of:
@@ -1149,7 +1287,8 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: static-check-all-files
 
-  breeze [FLAGS] static-check-all-files <STATIC_CHECK> -- <EXTRA_ARGS>
+
+  breeze static-check-all [FLAGS] STATIC_CHECK
 
         Run selected static checks for all applicable files. You should specify static check that
         you would like to run or 'all' to run all checks. One of:
@@ -1178,7 +1317,8 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: test-target
 
-  breeze [FLAGS] test-target <TEST_TARGET> -- <EXTRA_ARGS>
+
+  breeze test-target [FLAGS] TEST_TARGET -- <EXTRA_ARGS>
 
         Run the specified unit test target. There might be multiple
         targets specified separated with comas. The <EXTRA_ARGS> passed after -- are treated
@@ -1196,7 +1336,6 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: flags
 
-  breeze [FLAGS] flags -- <EXTRA_ARGS>
 
         Explains in detail all the flags that can be used with breeze.
 
@@ -1206,9 +1345,10 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: help
 
-  breeze [FLAGS] help -- <EXTRA_ARGS>
 
-        Shows this help message.
+  breeze help
+
+        Shows general help message for all commands.
 
 
   ####################################################################################################
@@ -1216,7 +1356,8 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: help-all
 
-  breeze [FLAGS] help-all -- <EXTRA_ARGS>
+
+  breeze help-all
 
         Shows detailed help for all commands and flags.
 
@@ -1284,43 +1425,23 @@ This is the current syntax for  `./breeze <./breeze>`_:
                  cassandra kerberos mongo openldap rabbitmq redis
 
   ****************************************************************************************************
-   Manage Kind kubernetes cluster (optional)
+   Kind kubernetes and Kubernetes tests configuration(optional)
 
-  Action for the cluster : only one of the --kind-cluster-* flags can be used at a time:
-
-  -s, --kind-cluster-start
-          Starts KinD Kubernetes cluster after entering the environment. The cluster is started using
-          Kubernetes Mode selected and Kubernetes version specified via --kubernetes-mode and
-          --kubernetes-version flags.
-
-  -x, --kind-cluster-stop
-          Stops KinD Kubernetes cluster if one has already been created. By default, if you do not
-          stop environment, the Kubernetes cluster created for testing is continuously running and
-          when you start Kubernetes testing again it will be reused. You can force deletion and
-          recreation of such cluster with this flag.
-
-  -r, --kind-cluster-recreate
-
-          Recreates KinD Kubernetes cluster if one has already been created. By default, if you do
-          not stop environment, the Kubernetes cluster created for testing is continuously running
-          and when you start Kubernetes testing again it will be reused. You can force deletion and
-          recreation of such cluster with this flag.
-
-  Kubernetes mode/version flags:
+  Configuration for the KinD Kubernetes cluster and tests:
 
   -K, --kubernetes-mode <KUBERNETES_MODE>
           Kubernetes mode - only used in case one of --kind-cluster-* commands is used.
           One of:
 
-                 persistent_mode git_mode
+                 image git
 
-          Default: git_mode
+          Default: image
 
   -V, --kubernetes-version <KUBERNETES_VERSION>
           Kubernetes version - only used in case one of --kind-cluster-* commands is used.
           One of:
 
-                 v1.15.3 v1.16.2
+                 v1.15.3
 
           Default: v1.15.3
 
@@ -1428,6 +1549,12 @@ This is the current syntax for  `./breeze <./breeze>`_:
           Note that you can further increase verbosity and see all the commands executed by breeze
           by running 'export VERBOSE_COMMANDS="true"' before running breeze.
 
+  ****************************************************************************************************
+   Print detailed help message
+
+  -h, --help
+          Shows detailed help message for the command specified.
+
  .. END BREEZE HELP MARKER
 
 Convenience Scripts
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 15038a5..1c93f27 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -154,20 +154,6 @@ RUN curl --fail --location https://download.docker.com/linux/debian/gpg | apt-ke
     && apt-get autoremove -yqq --purge \
     && apt-get clean && rm -rf /var/lib/apt/lists/*
 
-# Install kubectl
-ARG KUBECTL_VERSION="v1.15.3"
-
-RUN KUBECTL_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
-  && curl --fail --location  "${KUBECTL_URL}" --output "/usr/local/bin/kubectl" \
-  && chmod +x /usr/local/bin/kubectl
-
-# Install Kind
-ARG KIND_VERSION="v0.6.1"
-
-RUN KIND_URL="https://github.com/kubernetes-sigs/kind/releases/download/${KIND_VERSION}/kind-linux-amd64" \
-   && curl --fail --location "${KIND_URL}" --output "/usr/local/bin/kind" \
-   && chmod +x /usr/local/bin/kind
-
 # Setup PIP
 # By default PIP install run without cache to make image smaller
 ARG PIP_NO_CACHE_DIR="true"
@@ -342,8 +328,7 @@ COPY Dockerfile.ci ${AIRFLOW_SOURCES}/Dockerfile.ci
 RUN register-python-argcomplete airflow >> ~/.bashrc
 
 # Install autocomplete for Kubeclt
-RUN echo "source /etc/bash_completion" >> ~/.bashrc \
-    && kubectl completion bash >> ~/.bashrc
+RUN echo "source /etc/bash_completion" >> ~/.bashrc
 
 WORKDIR ${AIRFLOW_SOURCES}
 
diff --git a/IMAGES.rst b/IMAGES.rst
index 3038bfe..37b8d03 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -160,10 +160,6 @@ The following build arguments (``--build-arg`` in docker build command) can be u
 | ``DEPENDENCIES_EPOCH_NUMBER``            | ``2``                                    | increasing this number will reinstall    |
 |                                          |                                          | all apt dependencies                     |
 +------------------------------------------+------------------------------------------+------------------------------------------+
-| ``KUBECTL_VERSION``                      | ``v1.15.3``                              | version of kubectl installed             |
-+------------------------------------------+------------------------------------------+------------------------------------------+
-| ``KIND_VERSION``                         | ``v0.6.1``                               | version of kind installed                |
-+------------------------------------------+------------------------------------------+------------------------------------------+
 | ``PIP_NO_CACHE_DIR``                     | ``true``                                 | if true, then no pip cache will be       |
 |                                          |                                          | stored                                   |
 +------------------------------------------+------------------------------------------+------------------------------------------+
diff --git a/README.md b/README.md
index a0dc473..ad8915b 100644
--- a/README.md
+++ b/README.md
@@ -65,6 +65,7 @@ Apache Airflow is tested with:
 * Postgres DB: 9.6, 10
 * MySQL DB: 5.7
 * Sqlite - latest stable (it is used mainly for development purpose)
+* Kubernetes - 1.16.2, 1.17.0
 
 ### Stable version (1.10.9)
 
@@ -72,6 +73,7 @@ Apache Airflow is tested with:
 * Postgres DB: 9.6, 10
 * MySQL DB: 5.6, 5.7
 * Sqlite - latest stable (it is used mainly for development purpose)
+* Kubernetes - 1.16.2, 1.17.0
 
 ## Getting started
 Please visit the Airflow Platform documentation (latest **stable** release) for help with [installing Airflow](https://airflow.apache.org/installation.html), getting a [quick start](https://airflow.apache.org/start.html), or a more complete [tutorial](https://airflow.apache.org/tutorial.html).
diff --git a/TESTING.rst b/TESTING.rst
index 3c53fdd..fc8a046 100644
--- a/TESTING.rst
+++ b/TESTING.rst
@@ -325,98 +325,224 @@ Those tests are skipped by default. You can enable them with ``--include-quarant
 can also decide to only run tests with ``-m quarantined`` flag to run only those tests.
 
 Running Tests with Kubernetes
------------------------------
+=============================
+
+Starting Kubernetes Cluster
+---------------------------
+
+For your testing you manage Kind cluster with ``kind-cluster`` breeze command:
+
+.. code-block:: bash
+
+    ./breeze kind-cluster [ start | stop | recreate | status ]
+
 
-Starting Kubernetes Cluster when Starting Breeze
-................................................
+The command allows you to start/stop/recreate/status Kind Kubernetes cluster
+in your docker engine as well as deploy airflow to use it for testing (with ``deploy`` command).
 
-To run Kubernetes in Breeze, you can start Breeze with the ``--kind-cluster-start`` switch. This
-automatically creates a Kind Kubernetes cluster in the same ``docker`` engine that is used to run Breeze.
-Setting up the Kubernetes cluster takes some time so the cluster continues running
-until it is stopped with the ``--kind-cluster-stop`` switch or until the ``--kind-cluster-recreate``
-switch is used rather than ``--kind-cluster-start``. Starting Breeze with the Kind Cluster automatically
-sets ``runtime`` to ``kubernetes`` (see below).
+Setting up the Kind Kubernetes cluster takes some time so once you started it, the cluster continues running
+until it is stopped with the ``kind-cluster stop`` command or until ``kind-cluster recreate``
+command is used (it will stop and recreate the cluster image).
 
-The cluster name follows the pattern ``airflow-python-X.Y.Z-vA.B.C`` where X.Y.Z is a Python version
+The cluster name follows the pattern ``airflow-python-X.Y-vA.B.C`` where X.Y is a Python version
 and A.B.C is a Kubernetes version. This way you can have multiple clusters set up and running at the same
 time for different Python versions and different Kubernetes versions.
 
-The Control Plane is available from inside the Docker image via ``<CLUSTER_NAME>-control-plane:6443``
-host:port, the worker of the Kind Cluster is available at  <CLUSTER_NAME>-worker
+The Control Plane is available from inside the Docker image via ``<KIND_CLUSTER_NAME>-control-plane:6443``
+host:port, the worker of the Kind Cluster is available at  <KIND_CLUSTER_NAME>-worker
 and the webserver port for the worker is 30809.
 
-After the Kubernetes Cluster is started, you need to deploy Airflow to the cluster:
+Deploying Airflow to Kubernetes Cluster
+---------------------------------------
 
-1. Build the image.
-2. Load it to the Kubernetes cluster.
-3. Deploy the Airflow application.
+Deploying Airflow to the Kubernetes cluster created is also done via ``kind-cluster`` breeze command:
 
-It can be done with a single script: ``./scripts/ci/in_container/kubernetes/deploy_airflow_to_kubernetes.sh``.
+.. code-block:: bash
 
-You can, however, work separately on the image in Kubernetes and deploying the Airflow app in the cluster.
+    ./breeze kind-cluster deploy
 
-Building and Loading Airflow Images to Kubernetes Cluster
-..............................................................
 
-Use the script ``./scripts/ci/in_container/kubernetes/docker/rebuild_airflow_image.sh`` that does the following:
+The deploy commands performs tthose steps:
 
-1. Rebuilds the latest ``apache/airflow:master-pythonX.Y-ci`` images using the latest sources.
-2. Builds a new Kubernetes image based on the  ``apache/airflow:master-pythonX.Y-ci`` using
+1. If needed, it rebuilds the latest ``apache/airflow:master-pythonX.Y`` production images using the
+   latest sources. You can also force the build with ``--force-build-image`` flag.
+2. Builds a new Kubernetes image based on the  ``apache/airflow:master-pythonX.Y`` using
    necessary scripts added to run in Kubernetes. The image is tagged as
-   ``apache/airflow:master-pythonX.Y-ci-kubernetes``.
+   ``apache/airflow:master-pythonX.Y-kubernetes``.
 3. Loads the image to the Kind Cluster using the ``kind load`` command.
-
-Deploying the Airflow Application in the Kubernetes Cluster
-...........................................................
-
-Use the script ``./scripts/ci/in_container/kubernetes/app/deploy_app.sh`` that does the following:
-
-1. Prepares Kubernetes resources by processing a template from the ``template`` directory and replacing
+4. Prepares Kubernetes resources by processing a template from the ``template`` directory and replacing
    variables with the right images and locations:
    - configmaps.yaml
    - airflow.yaml
-2. Uses the existing resources without replacing any variables inside:
+5. Uses the existing resources without replacing any variables inside:
    - secrets.yaml
    - postgres.yaml
    - volumes.yaml
-3. Applies all the resources to the Kind Cluster.
-4. Waits for all the applications to be ready and reachable.
+6. Applies all the resources to the Kind Cluster.
+7. Waits for all the applications to be ready and reachable.
 
-After the deployment is finished, you can run Kubernetes tests immediately in the same way as other tests.
-The Kubernetes tests are available in the ``tests/runtime/kubernetes`` folder.
 
-You can run all the integration tests for Kubernetes with ``pytest tests/runtime/kubernetes``.
+Running tests with Kubernetes Cluster
+-------------------------------------
 
+After the deployment is finished, you can run Kubernetes tests via ``scripts/ci/ci_run_kubernetes_tests.sh``.
 
-Running Runtime-Specific Tests
-------------------------------
+You can either run all tests or you can select which tests to run. You can also enter interactive virtualenv
+to run the tests manually one by one.
 
-Tests using a specific runtime are marked with a custom pytest marker ``pytest.mark.runtime``.
-The marker has a single parameter - the name of a runtime. At the moment the only supported runtime is
-``kubernetes``. This runtime is set when you run Breeze with one of the ``--kind-cluster-*`` flags.
-Runtime-specific tests run only when the selectd runtime is started.
 
+.. code-block:: bash
 
-.. code-block:: python
+    Running kubernetes tests
+
+      ./scripts/ci/ci_run_kubernetes_tests.sh                      - runs all kubernetes tests
+      ./scripts/ci/ci_run_kubernetes_tests.sh TEST [TEST ...]      - runs selected kubernetes tests (from kubernetes_tests folder)
+      ./scripts/ci/ci_run_kubernetes_tests.sh [-i|--interactive]   - Activates virtual environment ready to run tests and drops you in
+      ./scripts/ci/ci_run_kubernetes_tests.sh [--help]             - Prints this help message
+
+
+Typical testing pattern for Kubernetes tests
+--------------------------------------------
+
+The typical session for tests with Kubernetes looks like follows:
+
+1. Start the Kind cluster:
+
+.. code-block:: bash
+
+    ./breeze kind-cluster start
+
+    Starts Kind Kubernetes cluster
+
+       Use CI image.
+
+       Branch name:             master
+       Docker image:            apache/airflow:master-python3.7-ci
+
+       Airflow source version:  2.0.0.dev0
+       Python version:          3.7
+       DockerHub user:          apache
+       DockerHub repo:          airflow
+       Backend:                 postgres 9.6
+
+    No kind clusters found.
+
+    Creating cluster
+
+    Creating cluster "airflow-python-3.7-v1.17.0" ...
+     ✓ Ensuring node image (kindest/node:v1.17.0) 🖼
+     ✓ Preparing nodes 📦 📦
+     ✓ Writing configuration 📜
+     ✓ Starting control-plane 🕹️
+     ✓ Installing CNI 🔌
+    Could not read storage manifest, falling back on old k8s.io/host-path default ...
+     ✓ Installing StorageClass 💾
+     ✓ Joining worker nodes 🚜
+    Set kubectl context to "kind-airflow-python-3.7-v1.17.0"
+    You can now use your cluster with:
+
+    kubectl cluster-info --context kind-airflow-python-3.7-v1.17.0
+
+    Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂
+
+    Created cluster airflow-python-3.7-v1.17.0
+
+
+2. Check the status of the cluster
+
+.. code-block:: bash
+
+    ./breeze kind-cluster status
+
+    Checks status of Kind Kubernetes cluster
 
-    @pytest.mark.runtime("kubernetes")
-    class TestKubernetesExecutor(unittest.TestCase):
+       Use CI image.
 
+       Branch name:             master
+       Docker image:            apache/airflow:master-python3.7-ci
 
-You can use the custom ``--runtime`` switch in pytest to only run tests specific for that backend.
+       Airflow source version:  2.0.0.dev0
+       Python version:          3.7
+       DockerHub user:          apache
+       DockerHub repo:          airflow
+       Backend:                 postgres 9.6
+
+    airflow-python-3.7-v1.17.0-control-plane
+    airflow-python-3.7-v1.17.0-worker
+
+3. Deploy Airflow to the cluster
+
+.. code-block:: bash
+
+    ./breeze kind-cluster deploy
+
+4. Run Kubernetes tests
+
+Note that the tests are executed in production container not in the CI container.
+There is no need for the tests to run inside the Airflow CI container image as they only
+communicate with the Kubernetes-run Airflow deployed via the production image.
+Those Kubernetes tests require virtualenv to be created locally with airflow installed.
+The virtualenv required will be created automatically when the scripts are run.
+
+
+Either run all the tests:
+
+
+.. code-block:: bash
+
+    ./breeze kind-cluster test
+
+
+Or enter the interactive virtualenv (the environment is in ``.build/.kubernetes_venv`` folder:
 
-To run only kubernetes-runtime backend tests, enter:
 
 .. code-block:: bash
 
-    pytest --runtime kubernetes
+     ./scripts/ci/ci_run_kubernetes_tests.sh -i
+
+
+Once you enter the environment you get this information:
 
-**NOTE:** For convenience and faster search, all runtime tests are stored in the ``tests.runtime`` package. In this case, you
-can speed up the collection of tests by running:
 
 .. code-block:: bash
 
-    pytest --runtime kubernetes tests/runtime
+    Activating the virtual environment for kubernetes testing
+
+    You can run kubernetes testing via 'pytest kubernetes_tests/....'
+    You can add -s to see the output of your tests on screen
+
+    The webserver is available at http://localhost:30809/
+
+    User/password: airflow/airflow
+
+    You are entering the virtualenv now. Type exit to exit back to the original shell
+
+
+You can iterate with tests while you are in the virtualenv:
+
+
+.. code-block:: bash
+
+    pytest kubernetes_tests/test_kubernetes_executor.py::TestKubernetesExecutor::test_integration_run_dag_with_scheduler_failure -s
+
+
+You can modify the tests or KubernetesPodOperator and re-run them without re-deploying
+airflow to KinD cluster.
+
+However when you change the Airflow Kubernetes executor implementation you need to redeploy
+Airflow to the cluster.
+
+.. code-block:: bash
+
+    ./breeze kind-cluster deploy
+
+
+5. Stop KinD cluster when you are done
+
+.. code-block:: bash
+
+    ./breeze kind-cluster stop
+
 
 Airflow System Tests
 ====================
diff --git a/airflow/kubernetes/refresh_config.py b/airflow/kubernetes/refresh_config.py
index b060d25..e7c5c13 100644
--- a/airflow/kubernetes/refresh_config.py
+++ b/airflow/kubernetes/refresh_config.py
@@ -115,5 +115,4 @@ def load_kube_config(client_configuration, config_file=None, context=None):
 
     loader = _get_kube_config_loader_for_yaml_file(
         config_file, active_context=context, config_persister=None)
-
     loader.load_and_set(client_configuration)
diff --git a/airflow/kubernetes/volume_mount.py b/airflow/kubernetes/volume_mount.py
index 74563b4..0dbca5f 100644
--- a/airflow/kubernetes/volume_mount.py
+++ b/airflow/kubernetes/volume_mount.py
@@ -24,26 +24,32 @@ from airflow.kubernetes.k8s_model import K8SModel
 
 
 class VolumeMount(K8SModel):
-    """Defines Kubernetes Volume Mount"""
+    """
+    Initialize a Kubernetes Volume Mount. Used to mount pod level volumes to
+    running container.
 
+    :param name: the name of the volume mount
+    :type name: str
+    :param mount_path:
+    :type mount_path: str
+    :param sub_path: subpath within the volume mount
+    :type sub_path: Optional[str]
+    :param read_only: whether to access pod with read-only mode
+    :type read_only: bool
+    """
     def __init__(self, name, mount_path, sub_path, read_only):
-        """Initialize a Kubernetes Volume Mount. Used to mount pod level volumes to
-        running container.
-        :param name: the name of the volume mount
-        :type name: str
-        :param mount_path:
-        :type mount_path: str
-        :param sub_path: subpath within the volume mount
-        :type sub_path: str
-        :param read_only: whether to access pod with read-only mode
-        :type read_only: bool
-        """
         self.name = name
         self.mount_path = mount_path
         self.sub_path = sub_path
         self.read_only = read_only
 
     def to_k8s_client_obj(self):
+        """
+        Converts to k8s object.
+
+        :return Volume Mount k8s object
+
+        """
         return k8s.V1VolumeMount(
             name=self.name,
             mount_path=self.mount_path,
@@ -52,6 +58,12 @@ class VolumeMount(K8SModel):
         )
 
     def attach_to_pod(self, pod):
+        """
+        Attaches to pod
+
+        :return Copy of the Pod object
+
+        """
         cp_pod = copy.deepcopy(pod)
         volume_mount = self.to_k8s_client_obj()
         cp_pod.spec.containers[0].volume_mounts = pod.spec.containers[0].volume_mounts or []
diff --git a/breeze b/breeze
index fcea655..7fa6e28 100755
--- a/breeze
+++ b/breeze
@@ -120,8 +120,11 @@ function setup_default_breeze_variables() {
     # This can be overridden by '--force-pull-images' flag
     export FORCE_PULL_IMAGES="false"
 
-    # Runtime is empty by default (might be set to kubernetes in case kubernetes is chosen)
-    RUNTIME=""
+    # Do not enable Kind Kubernetes cluster by default
+    export ENABLE_KIND_CLUSTER="false"
+
+    # By default we do not push images. This can be overridden by -u flag.
+    export PUSH_IMAGES=${PUSH_IMAGES:="false"}
 
     # Forward common host credentials to docker (gcloud, aws etc.).
     export FORWARD_CREDENTIALS="false"
@@ -148,11 +151,11 @@ function setup_default_breeze_variables() {
     # Default values for the flags used
 
     _BREEZE_DEFAULT_BACKEND="sqlite"
-    _BREEZE_DEFAULT_KUBERNETES_MODE="git_mode"
+    _BREEZE_DEFAULT_KUBERNETES_MODE="image"
     _BREEZE_DEFAULT_KUBERNETES_VERSION="v1.15.3"
     _BREEZE_DEFAULT_POSTGRES_VERSION="9.6"
+    _BREEZE_DEFAULT_POSTGRES_VERSION="9.6"
     _BREEZE_DEFAULT_MYSQL_VERSION="5.7"
-
     STATIC_CHECK_PYTHON_MAJOR_MINOR_VERSION=3.6
 }
 
@@ -376,16 +379,14 @@ EOF
 EOF
         fi
 
-        if [[ ${RUNTIME} == "kubernetes" ]]; then
+        if [[ ${INTEGRATIONS[*]} == *"kubernetes"* ]]; then
             cat <<EOF
 
-                               Kubernetes RUNTIME
-
-                               Kubernetes mode:    ${KUBERNETES_MODE}
-                               Kubernetes version: ${KUBERNETES_VERSION}
+                               Kubernetes mode:     ${KUBERNETES_MODE}
+                               Kubernetes version:  ${KUBERNETES_VERSION}
 
-                               Enable KinD:        ${ENABLE_KIND_CLUSTER}
-                               Cluster operation:  ${KIND_CLUSTER_OPERATION}
+                               Enable KinD cluster: ${ENABLE_KIND_CLUSTER}
+                               Cluster operation:   ${KIND_CLUSTER_OPERATION}
 EOF
         fi
     else
@@ -422,16 +423,14 @@ EOF
 EOF
         fi
 
-        if [[ ${RUNTIME} == "kubernetes" ]]; then
+        if [[ ${INTEGRATIONS[*]} == *"kubernetes"* ]]; then
             cat <<EOF
 
-   Kubernetes RUNTIME
+   Kubernetes mode:     ${KUBERNETES_MODE}
+   Kubernetes version:  ${KUBERNETES_VERSION}
 
-   Kubernetes mode:    ${KUBERNETES_MODE}
-   Kubernetes version: ${KUBERNETES_VERSION}
-
-   Enable KinD:        ${ENABLE_KIND_CLUSTER}
-   Cluster operation:  ${KIND_CLUSTER_OPERATION}
+   Enable KinD cluster: ${ENABLE_KIND_CLUSTER}
+   Cluster operation:   ${KIND_CLUSTER_OPERATION}
 EOF
         fi
     fi
@@ -458,7 +457,6 @@ export DOCKERHUB_REPO=${DOCKERHUB_REPO}
 export COMPOSE_FILE="${COMPOSE_FILE}"
 export PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION}"
 export BACKEND="${BACKEND}"
-export RUNTIME="${RUNTIME}"
 export ENABLE_KIND_CLUSTER="${ENABLE_KIND_CLUSTER}"
 export KUBERNETES_MODE="${KUBERNETES_MODE}"
 export KUBERNETES_VERSION="${KUBERNETES_VERSION}"
@@ -495,7 +493,6 @@ function prepare_command_files() {
     BACKEND_DOCKER_COMPOSE_FILE=${SCRIPTS_CI_DIR}/docker-compose/backend-${BACKEND}.yml
     LOCAL_DOCKER_COMPOSE_FILE=${SCRIPTS_CI_DIR}/docker-compose/local.yml
     LOCAL_PROD_DOCKER_COMPOSE_FILE=${SCRIPTS_CI_DIR}/docker-compose/local-prod.yml
-    KUBERNETES_DOCKER_COMPOSE_FILE=${SCRIPTS_CI_DIR}/docker-compose/runtime-kubernetes.yml
     REMOVE_SOURCES_DOCKER_COMPOSE_FILE=${SCRIPTS_CI_DIR}/docker-compose/remove-sources.yml
     FORWARD_CREDENTIALS_DOCKER_COMPOSE_FILE=${SCRIPTS_CI_DIR}/docker-compose/forward-credentials.yml
 
@@ -516,11 +513,6 @@ function prepare_command_files() {
         COMPOSE_CI_FILE=${COMPOSE_CI_FILE}:${REMOVE_SOURCES_DOCKER_COMPOSE_FILE}
     fi
 
-    if [[ ${RUNTIME} == "kubernetes" ]]; then
-        COMPOSE_CI_FILE=${COMPOSE_CI_FILE}:${KUBERNETES_DOCKER_COMPOSE_FILE}
-        COMPOSE_PROD_FILE=${COMPOSE_PROD_FILE}:${KUBERNETES_DOCKER_COMPOSE_FILE}
-    fi
-
     set +u
     # shellcheck disable=SC2207
     UNIQUE_INTEGRATIONS=($(echo "${INTEGRATIONS[@]}" | tr ' ' '\n' | sort -u | tr '\n' ' '))
@@ -724,27 +716,6 @@ function parse_arguments() {
           export DOCKER_CACHE="no-cache"
           export FORCE_BUILD_IMAGES="true"
           shift ;;
-        -s|--kind-cluster-start)
-          export RUNTIME=kubernetes
-          export ENABLE_KIND_CLUSTER="true"
-          export KIND_CLUSTER_OPERATION="start"
-          echo "Starting kubernetes KinD cluster"
-          echo
-          shift ;;
-        -r|--kind-cluster-recreate)
-          export RUNTIME=kubernetes
-          export ENABLE_KIND_CLUSTER="true"
-          export KIND_CLUSTER_OPERATION="recreate"
-          echo "Recreating KinD cluster"
-          echo
-          shift ;;
-        -x|--kind-cluster-stop)
-          export RUNTIME=kubernetes
-          export ENABLE_KIND_CLUSTER="true"
-          export KIND_CLUSTER_OPERATION="stop"
-          echo "Stop KinD cluster"
-          echo
-          shift ;;
         -L|--use-local-cache)
           echo "Use local cache to build images"
           echo
@@ -887,6 +858,16 @@ function parse_arguments() {
           echo
           COMMAND_TO_RUN="perform_initialize_local_virtualenv"
           shift ;;
+        kind-cluster)
+          LAST_SUBCOMMAND="${1}"
+          COMMAND_TO_RUN="manage_kind_cluster"
+          export KIND_CLUSTER_OPERATION="${2:-}"
+          if [[ ${KIND_CLUSTER_OPERATION} != "" ]]; then
+              shift 2
+          else
+              shift
+          fi
+          ;;
         setup-autocomplete)
           LAST_SUBCOMMAND="${1}"
           echo "Setting up autocomplete"
@@ -1044,6 +1025,8 @@ function prepare_formatted_versions() {
       fold -w "${WIDTH}" -s | sed "s/^/${LIST_PREFIX}/")
     FORMATTED_KUBERNETES_VERSIONS=$(echo "${_BREEZE_ALLOWED_KUBERNETES_VERSIONS=""}" | tr '\n' ' ' | \
       fold -w "${WIDTH}" -s | sed "s/^/${LIST_PREFIX}/")
+    FORMATTED_KIND_OPERATIONS=$(echo "${_BREEZE_ALLOWED_KIND_OPERATIONS=""}" | tr '\n' ' ' | \
+      fold -w "${WIDTH}" -s | sed "s/^/${LIST_PREFIX}/")
     FORMATTED_INSTALL_AIRFLOW_VERSIONS=$(echo "${_BREEZE_ALLOWED_INSTALL_AIRFLOW_VERSIONS=""}" | \
       tr '\n' ' ' | fold -w "${WIDTH}" -s | sed "s/^/${LIST_PREFIX}/")
     FORMATTED_POSTGRES_VERSIONS=$(echo "${_BREEZE_ALLOWED_POSTGRES_VERSIONS=""}" | \
@@ -1071,6 +1054,7 @@ function prepare_usage() {
     export USAGE_GENERATE_REQUIREMENTS="Generates pinned requirements for pip dependencies"
     export USAGE_INITIALIZE_LOCAL_VIRTUALENV="Initializes local virtualenv"
     export USAGE_PUSH_IMAGE="Pushes images to registry"
+    export USAGE_KIND_CLUSTER="Manages KinD cluster on the host"
     export USAGE_SETUP_AUTOCOMPLETE="Sets up autocomplete for breeze"
     export USAGE_STOP="Stops the docker-compose environment"
     export USAGE_RESTART="Stops the docker-compose environment including DB cleanup"
@@ -1085,6 +1069,8 @@ function prepare_usage() {
 
     # shellcheck disable=SC2089
     DETAILED_USAGE_SHELL="
+${CMDNAME} shell [FLAGS] -- <EXTRA_ARGS>
+
       This is default subcommand if no subcommand is used.
 
       Enters interactive shell where you can run all tests, start Airflow webserver, scheduler,
@@ -1106,12 +1092,16 @@ $(flag_footer)
     # shellcheck disable=SC2090
     export DETAILED_USAGE_SHELL
     export DETAILED_USAGE_EXEC="
+${CMDNAME} exec
+
       Execs into interactive shell to an already running container. The container mus be started
       already by breeze shell command. If you are not familiar with tmux, this is the best
       way to run multiple processes in the same container at the same time for example scheduler,
       webserver, workers, database console and interactive terminal.
 "
     export DETAILED_USAGE_BUILD_DOCS="
+${CMDNAME} build-docs
+
       Builds Airflow documentation. The documentation is build inside docker container - to
       maintain the same build environment for everyone. Appropriate sources are mapped from
       the host to the container so that latest sources are used. The folders where documentation
@@ -1120,6 +1110,8 @@ $(flag_footer)
 "
     # shellcheck disable=SC2089
     DETAILED_USAGE_BUILD_IMAGE="
+${CMDNAME} build-image [FLAGS]
+
       Builds docker image (CI or production) without entering the container. You can pass
       additional options to this command, such as '--force-build-image',
       '--force-pull-image' '--python' '--use-local-cache'' in order to modify build behaviour.
@@ -1137,6 +1129,8 @@ $(flag_verbosity)
     export DETAILED_USAGE_BUILD_IMAGE
     # shellcheck disable=SC2089
     DETAILED_USAGE_CLEANUP_IMAGE="
+${CMDNAME} cleanup-image [FLAGS]
+
       Removes the breeze-related images created in your local docker image cache. This will
       not reclaim space in docker cache. You need to 'docker system prune' (optionally
       with --all) to reclaim that space.
@@ -1150,6 +1144,8 @@ $(flag_verbosity)
     export DETAILED_USAGE_CLEANUP_IMAGE
     # shellcheck disable=SC2089
     DETAILED_USAGE_DOCKER_COMPOSE="
+${CMDNAME} docker-compose [FLAGS] COMMAND -- <EXTRA_ARGS>
+
       Run docker-compose command instead of entering the environment. Use 'help' as command
       to see available commands. The <EXTRA_ARGS> passed after -- are treated
       as additional options passed to docker-compose. For example
@@ -1165,6 +1161,8 @@ $(flag_verbosity)
     export DETAILED_USAGE_DOCKER_COMPOSE
     # shellcheck disable=SC2089
     DETAILED_USAGE_EXECUTE_COMMAND="
+${CMDNAME} execute-command [FLAGS] COMMAND -- <EXTRA_ARGS>
+
       Run chosen command instead of entering the environment. The command is run using
       'bash -c \"<command with args>\" if you need to pass arguments to your command, you need
       to pass them together with command surrounded with \" or '. Alternatively you can
@@ -1184,6 +1182,8 @@ $(flag_verbosity)
       Explains in detail all the flags that can be used with breeze.
 "
     DETAILED_USAGE_GENERATE_REQUIREMENTS="
+${CMDNAME} generate-requirements [FLAGS]
+
       Generates pinned requirements from setup.py. Those requirements are generated in requirements
       directory - separately for different python version. Those requirements are used to run
       CI builds as well as run repeatable production image builds. You can use those requirements
@@ -1197,6 +1197,8 @@ $(flag_verbosity)
     # shellcheck disable=SC2090
     export DETAILED_USAGE_GENERATE_REQUIREMENTS
     DETAILED_USAGE_INITIALIZE_LOCAL_VIRTUALENV="
+${CMDNAME} initialize-local-virtualenv [FLAGS]
+
       Initializes locally created virtualenv installing all dependencies of Airflow
       taking into account the frozen requirements from requirements folder.
       This local virtualenv can be used to aid autocompletion and IDE support as
@@ -1210,6 +1212,8 @@ $(flag_airflow_variants)
     export DETAILED_USAGE_INITIALIZE_LOCAL_VIRTUALENV
     # shellcheck disable=SC2089
     DETAILED_USAGE_PUSH_IMAGE="
+${CMDNAME} push_image [FLAGS]
+
       Pushes images to docker registry. You can push the images to DockerHub registry (default)
       or to the GitHub cache registry (if --registry-cache flag is used).
 
@@ -1233,17 +1237,40 @@ $(flag_verbosity)
 "
     # shellcheck disable=SC2090
     export DETAILED_USAGE_PUSH_IMAGE
+    DETAILED_USAGE_KIND_CLUSTER="
+${CMDNAME} kind-cluster [FLAGS] OPERATION
+
+      Manages host-side Kind Kubernetes cluster that is used to run Kubernetes integration tests.
+      It allows to start/stop/restart/status the Kind Kubernetes cluster and deploy Airflow to it.
+      This enables you to run tests inside the breeze environment with latest airflow images loaded.
+      Note that in case of deploying airflow, the first step is to rebuild the image and loading it
+      to the cluster so you can also pass appropriate build image flags that will influence
+      rebuilding the production image. Operation is one of:
+
+${FORMATTED_KIND_OPERATIONS}
+
+Flags:
+$(flag_airflow_variants)
+$(flag_build_docker_images)
+"
+    export DETAILED_USAGE_KIND_CLUSTER
     export DETAILED_USAGE_SETUP_AUTOCOMPLETE="
+${CMDNAME} setup-autocomplete
+
       Sets up autocomplete for breeze commands. Once you do it you need to re-enter the bash
       shell and when typing breeze command <TAB> will provide autocomplete for
       parameters and values.
 "
     export DETAILED_USAGE_STOP="
+${CMDNAME} stop
+
       Brings down running docker compose environment. When you start the environment, the docker
       containers will continue running so that startup time is shorter. But they take quite a lot of
       memory and CPU. This command stops all running containers from the environment.
 "
     DETAILED_USAGE_RESTART="
+${CMDNAME} restart [FLAGS]
+
       Restarts running docker compose environment. When you restart the environment, the docker
       containers will be restarted. That includes cleaning up the databases. This is
       especially useful if you switch between different versions of Airflow.
@@ -1253,6 +1280,8 @@ $(flag_footer)
 "
     export DETAILED_USAGE_RESTART
     export DETAILED_USAGE_STATIC_CHECK="
+${CMDNAME} static-check [FLAGS] STATIC_CHECK
+
       Run selected static checks for currently changed files. You should specify static check that
       you would like to run or 'all' to run all checks. One of:
 
@@ -1269,6 +1298,8 @@ ${FORMATTED_STATIC_CHECKS}
       '${CMDNAME} static-check mypy -- --help'
 "
     export DETAILED_USAGE_STATIC_CHECK_ALL_FILES="
+${CMDNAME} static-check-all [FLAGS] STATIC_CHECK
+
       Run selected static checks for all applicable files. You should specify static check that
       you would like to run or 'all' to run all checks. One of:
 
@@ -1286,6 +1317,8 @@ ${FORMATTED_STATIC_CHECKS}
 "
     # shellcheck disable=SC2089
     DETAILED_USAGE_TEST_TARGET="
+${CMDNAME} test-target [FLAGS] TEST_TARGET -- <EXTRA_ARGS>
+
       Run the specified unit test target. There might be multiple
       targets specified separated with comas. The <EXTRA_ARGS> passed after -- are treated
       as additional options passed to pytest. For example:
@@ -1298,15 +1331,23 @@ $(flag_footer)
     # shellcheck disable=SC2090
     export DETAILED_USAGE_TEST_TARGET
     export DETAILED_USAGE_TOGGLE_SUPPRESS_CHEATSHEET="
+${CMDNAME} toggle-suppress-cheatsheet
+
       Toggles on/off cheatsheet displayed before starting bash shell.
 "
     export DETAILED_USAGE_TOGGLE_SUPPRESS_ASCIIART="
+${CMDNAME} toggle-suppress-asciiart
+
       Toggles on/off asciiart displayed before starting bash shell.
 "
     export DETAILED_USAGE_HELP="
-      Shows this help message.
+${CMDNAME} help
+
+      Shows general help message for all commands.
 "
     export DETAILED_USAGE_HELP_ALL="
+${CMDNAME} help-all
+
       Shows detailed help for all commands and flags.
 "
 }
@@ -1366,21 +1407,10 @@ Help commands:
 # Prints detailed usage for command specified
 function detailed_usage() {
       SUBCOMMAND=${1}
-      COMMAND_PARAMETER=""
-      if [[ ${SUBCOMMAND} == "static-check" ]]; then
-        COMMAND_PARAMETER="<STATIC_CHECK> "
-      elif [[ ${SUBCOMMAND} == "static-check-all-files" ]]; then
-        COMMAND_PARAMETER="<STATIC_CHECK> "
-      elif [[ ${SUBCOMMAND} == "test-target" ]]; then
-        COMMAND_PARAMETER="<TEST_TARGET> "
-      elif [[ ${SUBCOMMAND} == "docker-compose" ]]; then
-        COMMAND_PARAMETER="<DOCKER_COMPOSE_COMMAND> "
-      fi
       echo "
 
 Detailed usage for command: ${SUBCOMMAND}
 
-${CMDNAME} [FLAGS] ${SUBCOMMAND} ${COMMAND_PARAMETER}-- <EXTRA_ARGS>
 $(get_detailed_usage "${SUBCOMMAND}")
 
 "
@@ -1457,29 +1487,9 @@ ${FORMATTED_INTEGRATIONS}
 }
 
 # Prints Kubernetes action flags
-function flag_kubernetes_actions() {
+function flag_kubernetes_configuration() {
       echo "
-Action for the cluster : only one of the --kind-cluster-* flags can be used at a time:
-
--s, --kind-cluster-start
-        Starts KinD Kubernetes cluster after entering the environment. The cluster is started using
-        Kubernetes Mode selected and Kubernetes version specified via --kubernetes-mode and
-        --kubernetes-version flags.
-
--x, --kind-cluster-stop
-        Stops KinD Kubernetes cluster if one has already been created. By default, if you do not
-        stop environment, the Kubernetes cluster created for testing is continuously running and
-        when you start Kubernetes testing again it will be reused. You can force deletion and
-        recreation of such cluster with this flag.
-
--r, --kind-cluster-recreate
-
-        Recreates KinD Kubernetes cluster if one has already been created. By default, if you do
-        not stop environment, the Kubernetes cluster created for testing is continuously running
-        and when you start Kubernetes testing again it will be reused. You can force deletion and
-        recreation of such cluster with this flag.
-
-Kubernetes mode/version flags:
+Configuration for the KinD Kubernetes cluster and tests:
 
 -K, --kubernetes-mode <KUBERNETES_MODE>
         Kubernetes mode - only used in case one of --kind-cluster-* commands is used.
@@ -1556,6 +1566,13 @@ function flag_verbosity() {
 "
 }
 
+function flag_help() {
+      echo "
+-h, --help
+        Shows detailed help message for the command specified.
+"
+}
+
 # Prints flags controlling docker build process
 function flag_build_docker_images() {
       echo "
@@ -1642,8 +1659,8 @@ $(print_star_line)
 $(flag_breeze_actions)
 
 $(print_star_line)
- Manage Kind kubernetes cluster (optional)
-$(flag_kubernetes_actions)
+ Kind kubernetes and Kubernetes tests configuration(optional)
+$(flag_kubernetes_configuration)
 
 $(print_star_line)
  Manage mounting local files
@@ -1672,6 +1689,10 @@ $(flag_pull_push_docker_images)
 $(print_star_line)
  Increase verbosity of the scripts
 $(flag_verbosity)
+
+$(print_star_line)
+ Print detailed help message
+$(flag_help)
 "
 }
 
@@ -1915,6 +1936,28 @@ function run_build_command {
             ;;
         perform_initialize_local_virtualenv|perform_setup_autocomplete)
             ;;
+        manage_kind_cluster)
+            if [[ ${KIND_CLUSTER_OPERATION} == "start" ]] ; then
+              echo "Starts KinD cluster"
+            elif [[ ${KIND_CLUSTER_OPERATION} == "stop" ]] ; then
+              echo "Stops KinD cluster"
+            elif [[ ${KIND_CLUSTER_OPERATION} == "restart" ]] ; then
+              echo "Restarts KinD cluster"
+            elif [[ ${KIND_CLUSTER_OPERATION} == "status" ]] ; then
+              echo "Checks status of KinD cluster"
+            elif [[ ${KIND_CLUSTER_OPERATION} == "deploy" ]] ; then
+              echo "Deploys Airflow to KinD cluster"
+            elif [[ ${KIND_CLUSTER_OPERATION} == "test" ]] ; then
+              echo "Run Kubernetes tests with the KinD cluster "
+            else
+              echo "ERROR: Unknown Kind Kubernetes cluster operation: '${KIND_CLUSTER_OPERATION}'"
+              echo
+              echo "Should be one of:"
+              echo "${FORMATTED_KIND_OPERATIONS}"
+              echo
+              exit 1
+            fi
+            ;;
         *)
           echo >&2
           echo >&2 "ERROR: Unknown command to run ${COMMAND_TO_RUN}"
@@ -1997,6 +2040,10 @@ function run_breeze_command {
         perform_setup_autocomplete)
             setup_autocomplete
             ;;
+        manage_kind_cluster)
+            check_kind_and_kubectl_are_installed
+            perform_kind_cluster_operation "${KIND_CLUSTER_OPERATION}"
+            ;;
         build_docs)
             run_docs
             ;;
diff --git a/breeze-complete b/breeze-complete
index de39a83..d1df5b9 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -20,10 +20,11 @@
 _BREEZE_ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS="2.7 3.5 3.6 3.7"
 _BREEZE_ALLOWED_BACKENDS="sqlite mysql postgres"
 _BREEZE_ALLOWED_INTEGRATIONS="cassandra kerberos mongo openldap rabbitmq redis"
-_BREEZE_ALLOWED_KUBERNETES_MODES="persistent_mode git_mode"
-_BREEZE_ALLOWED_KUBERNETES_VERSIONS="v1.15.3 v1.16.2"
+_BREEZE_ALLOWED_KUBERNETES_MODES="image git"
+_BREEZE_ALLOWED_KUBERNETES_VERSIONS="v1.15.3"
 _BREEZE_ALLOWED_MYSQL_VERSIONS="5.6 5.7"
 _BREEZE_ALLOWED_POSTGRES_VERSIONS="9.6 10"
+_BREEZE_ALLOWED_KIND_OPERATIONS="start stop restart status deploy test"
 
 _BREEZE_ALLOWED_INSTALL_AIRFLOW_VERSIONS=$(cat <<-EOF
 1.10.10
@@ -85,7 +86,6 @@ _BREEZE_DEFAULT_GITHUB_REPO="airflow"
 _BREEZE_SHORT_OPTIONS="
 h p: b: i:
 K: V:
-s x r
 l a: t: d:
 v y n q f
 F P I E: C L
@@ -95,7 +95,6 @@ D: R: c g: G:
 _BREEZE_LONG_OPTIONS="
 help python: backend: integration:
 kubernetes-mode: kubernetes-version:
-kind-cluster-start kind-cluster-stop kind-cluster-recreate
 skip-mounting-local-sources install-airflow-version: install-airflow-reference: db-reset
 verbose assume-yes assume-no assume-quit forward-credentials
 force-build-images force-pull-images production-image extras: force-clean-images use-local-cache
@@ -113,6 +112,7 @@ exec
 generate-requirements
 push-image
 initialize-local-virtualenv
+kind-cluster
 setup-autocomplete
 stop
 restart
@@ -122,6 +122,7 @@ toggle-suppress-asciiart"
 export BREEZE_EXTRA_ARG_COMMANDS="
 docker-compose
 execute-command
+kind-cluster
 static-check
 static-check-all-files
 test-target"
@@ -188,6 +189,9 @@ function _get_known_values_breeze() {
     -g | --github-repo)
         _BREEZE_KNOWN_VALUES="${_BREEZE_DEFAULT_GITHUB_REPO}"
         ;;
+    kind-cluster)
+        _BREEZE_KNOWN_VALUES="${_BREEZE_ALLOWED_KIND_OPERATIONS}"
+        ;;
     *)
         _BREEZE_KNOWN_VALUES=""
         ;;
diff --git a/tests/runtime/kubernetes/__init__.py b/kubernetes_tests/__init__.py
similarity index 100%
rename from tests/runtime/kubernetes/__init__.py
rename to kubernetes_tests/__init__.py
diff --git a/tests/runtime/kubernetes/test_kubernetes_executor.py b/kubernetes_tests/test_kubernetes_executor.py
similarity index 71%
rename from tests/runtime/kubernetes/test_kubernetes_executor.py
rename to kubernetes_tests/test_kubernetes_executor.py
index ac44f81..3b86c7e 100644
--- a/tests/runtime/kubernetes/test_kubernetes_executor.py
+++ b/kubernetes_tests/test_kubernetes_executor.py
@@ -20,16 +20,18 @@ import time
 import unittest
 from subprocess import check_call, check_output
 
-import pytest
 import requests
 import requests.exceptions
 from requests.adapters import HTTPAdapter
 from urllib3.util.retry import Retry
 
-KUBERNETES_HOST = (os.environ.get('CLUSTER_NAME') or "docker") + "-worker:30809"
+KUBERNETES_HOST_PORT = (os.environ.get('CLUSTER_HOST') or "localhost") + ":30809"
+
+print()
+print("Cluster host/port used: ${KUBERNETES_HOST_PORT}".format(KUBERNETES_HOST_PORT=KUBERNETES_HOST_PORT))
+print()
 
 
-@pytest.mark.runtime("kubernetes")
 class TestKubernetesExecutor(unittest.TestCase):
 
     @staticmethod
@@ -56,7 +58,7 @@ class TestKubernetesExecutor(unittest.TestCase):
 
     def _ensure_airflow_webserver_is_healthy(self):
         response = self.session.get(
-            "http://{host}/health".format(host=KUBERNETES_HOST),
+            "http://{host}/health".format(host=KUBERNETES_HOST_PORT),
             timeout=1,
         )
 
@@ -80,16 +82,15 @@ class TestKubernetesExecutor(unittest.TestCase):
 
             # Trigger a new dagrun
             try:
-                result = self.session.get(
-                    'http://{host}/api/experimental/dags/{dag_id}/'
-                    'dag_runs/{execution_date}/tasks/{task_id}'
-                    .format(host=host,
-                            dag_id=dag_id,
-                            execution_date=execution_date,
-                            task_id=task_id)
-                )
+                get_string = \
+                    'http://{host}/api/experimental/dags/{dag_id}/' \
+                    'dag_runs/{execution_date}/tasks/{task_id}'.format(
+                        host=host, dag_id=dag_id, execution_date=execution_date, task_id=task_id)
+                print("Calling [monitor_task]#1 {get_string}".format(get_string=get_string))
+                result = self.session.get(get_string)
                 self.assertEqual(result.status_code, 200, "Could not get the status")
                 result_json = result.json()
+                print("Received [monitor_task]#2: {result_json}".format(result_json=result_json))
                 state = result_json['state']
                 print("Attempt {}: Current state of operator is {}".format(tries, state))
 
@@ -112,18 +113,16 @@ class TestKubernetesExecutor(unittest.TestCase):
         while tries < max_tries:
             time.sleep(5)
 
+            get_string = \
+                'http://{host}/api/experimental/dags/{dag_id}/' \
+                'dag_runs/{execution_date}'.format(host=host,
+                                                   dag_id=dag_id, execution_date=execution_date)
+            print("Calling {get_string}".format(get_string=get_string))
             # Trigger a new dagrun
-            result = self.session.get(
-                'http://{host}/api/experimental/dags/{dag_id}/'
-                'dag_runs/{execution_date}'
-                .format(host=host,
-                        dag_id=dag_id,
-                        execution_date=execution_date)
-            )
-            print(result)
+            result = self.session.get(get_string)
             self.assertEqual(result.status_code, 200, "Could not get the status")
             result_json = result.json()
-            print(result_json)
+            print("Received: {result}".format(result=result))
             state = result_json['state']
             check_call(
                 ["echo", "Attempt {}: Current state of dag is {}".format(tries, state)])
@@ -138,53 +137,60 @@ class TestKubernetesExecutor(unittest.TestCase):
         # Maybe check if we can retrieve the logs, but then we need to extend the API
 
     def start_dag(self, dag_id, host):
-        result = self.session.get(
-            'http://{host}/api/experimental/'
-            'dags/{dag_id}/paused/false'.format(host=host, dag_id=dag_id)
-        )
+        get_string = 'http://{host}/api/experimental/' \
+                     'dags/{dag_id}/paused/false'.format(host=host, dag_id=dag_id)
+        print("Calling [start_dag]#1 {get_string}".format(get_string=get_string))
+        result = self.session.get(get_string)
         try:
             result_json = result.json()
         except ValueError:
             result_json = str(result)
-
+        print("Received [start_dag]#1 {result_json}".format(result_json=result_json))
         self.assertEqual(result.status_code, 200, "Could not enable DAG: {result}"
                          .format(result=result_json))
-
+        post_string = 'http://{host}/api/experimental/' \
+                      'dags/{dag_id}/dag_runs'.format(host=host, dag_id=dag_id)
+        print("Calling [start_dag]#2 {post_string}".format(post_string=post_string))
         # Trigger a new dagrun
-        result = self.session.post(
-            'http://{host}/api/experimental/'
-            'dags/{dag_id}/dag_runs'.format(host=host, dag_id=dag_id),
-            json={}
-        )
+        result = self.session.post(post_string, json={})
         try:
             result_json = result.json()
         except ValueError:
             result_json = str(result)
-
+        print("Received [start_dag]#2 {result_json}".format(result_json=result_json))
         self.assertEqual(result.status_code, 200, "Could not trigger a DAG-run: {result}"
                          .format(result=result_json))
 
         time.sleep(1)
 
-        result = self.session.get(
-            'http://{}/api/experimental/latest_runs'.format(host)
-        )
+        get_string = 'http://{host}/api/experimental/latest_runs'.format(host=host)
+        print("Calling [start_dag]#3 {get_string}".format(get_string=get_string))
+        result = self.session.get(get_string)
         self.assertEqual(result.status_code, 200, "Could not get the latest DAG-run:"
                                                   " {result}"
                          .format(result=result.json()))
         result_json = result.json()
+        print("Received: [start_dag]#3 {result_json}".format(result_json=result_json))
         return result_json
 
-    def test_integration_run_dag(self):
-        host = KUBERNETES_HOST
-        dag_id = 'example_kubernetes_executor_config'
-
+    def start_job_in_kubernetes(self, dag_id, host):
         result_json = self.start_dag(dag_id=dag_id, host=host)
-
         self.assertGreater(len(result_json['items']), 0)
+        execution_date = None
+        for dag_run in result_json['items']:
+            if dag_run['dag_id'] == dag_id:
+                execution_date = dag_run['execution_date']
+                break
+        self.assertIsNotNone(execution_date,
+                             "No execution_date can be found for the dag with {dag_id}".format(dag_id=dag_id))
+        return execution_date
+
+    def test_integration_run_dag(self):
+        host = KUBERNETES_HOST_PORT
+        dag_id = 'example_kubernetes_executor_config'
 
-        execution_date = result_json['items'][0]['execution_date']
-        print("Found the job with execution date {}".format(execution_date))
+        execution_date = self.start_job_in_kubernetes(dag_id, host)
+        print("Found the job with execution date {execution_date}".format(execution_date=execution_date))
 
         # Wait some time for the operator to complete
         self.monitor_task(host=host,
@@ -199,15 +205,10 @@ class TestKubernetesExecutor(unittest.TestCase):
                                        expected_final_state='success', timeout=200)
 
     def test_integration_run_dag_with_scheduler_failure(self):
-        host = KUBERNETES_HOST
+        host = KUBERNETES_HOST_PORT
         dag_id = 'example_kubernetes_executor_config'
 
-        result_json = self.start_dag(dag_id=dag_id, host=host)
-
-        self.assertGreater(len(result_json['items']), 0)
-
-        execution_date = result_json['items'][0]['execution_date']
-        print("Found the job with execution date {}".format(execution_date))
+        execution_date = self.start_job_in_kubernetes(dag_id, host)
 
         self._delete_airflow_pod()
 
diff --git a/tests/runtime/kubernetes/test_kubernetes_pod_operator.py b/kubernetes_tests/test_kubernetes_pod_operator.py
similarity index 64%
rename from tests/runtime/kubernetes/test_kubernetes_pod_operator.py
rename to kubernetes_tests/test_kubernetes_pod_operator.py
index f533659..7eb4beb 100644
--- a/tests/runtime/kubernetes/test_kubernetes_pod_operator.py
+++ b/kubernetes_tests/test_kubernetes_pod_operator.py
@@ -1,3 +1,4 @@
+# pylint: disable=unused-argument
 # Licensed to the Apache Software Foundation (ASF) under one
 # or more contributor license agreements.  See the NOTICE file
 # distributed with this work for additional information
@@ -20,25 +21,45 @@ import os
 import shutil
 import unittest
 
+from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
+from tests.compat import mock, patch
+
 import kubernetes.client.models as k8s
-import pytest
+import pendulum
 from kubernetes.client.api_client import ApiClient
 from kubernetes.client.rest import ApiException
 
-from airflow import AirflowException
-from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
+from airflow.exceptions import AirflowException
+from airflow.kubernetes import kube_client
 from airflow.kubernetes.pod import Port
 from airflow.kubernetes.pod_generator import PodDefaults
 from airflow.kubernetes.pod_launcher import PodLauncher
 from airflow.kubernetes.secret import Secret
 from airflow.kubernetes.volume import Volume
 from airflow.kubernetes.volume_mount import VolumeMount
+from airflow.models import DAG, TaskInstance
+
+from airflow.utils import timezone
 from airflow.version import version as airflow_version
-from tests.compat import mock
 
 
-@pytest.mark.runtime("kubernetes")
-class TestKubernetesPodOperator(unittest.TestCase):
+# noinspection DuplicatedCode
+def create_context(task):
+    dag = DAG(dag_id="dag")
+    tzinfo = pendulum.timezone("Europe/Amsterdam")
+    execution_date = timezone.datetime(2016, 1, 1, 1, 0, 0, tzinfo=tzinfo)
+    task_instance = TaskInstance(task=task,
+                                 execution_date=execution_date)
+    return {
+        "dag": dag,
+        "ts": execution_date.isoformat(),
+        "task": task,
+        "ti": task_instance,
+    }
+
+
+# noinspection DuplicatedCode,PyUnusedLocal
+class TestKubernetesPodOperatorSystem(unittest.TestCase):
 
     def setUp(self):
         self.maxDiff = None  # pylint: disable=invalid-name
@@ -51,10 +72,12 @@ class TestKubernetesPodOperator(unittest.TestCase):
                 'name': mock.ANY,
                 'annotations': {},
                 'labels': {
-                    'foo': 'bar',
-                    'kubernetes_pod_operator': 'True',
-                    'airflow_version': airflow_version.replace('+', '-')
-                }
+                    'foo': 'bar', 'kubernetes_pod_operator': 'True',
+                    'airflow_version': airflow_version.replace('+', '-'),
+                    'execution_date': '2016-01-01T0100000100-a2f50a31f',
+                    'dag_id': 'dag',
+                    'task_id': 'task',
+                    'try_number': '1'},
             },
             'spec': {
                 'affinity': {},
@@ -67,15 +90,11 @@ class TestKubernetesPodOperator(unittest.TestCase):
                     'envFrom': [],
                     'name': 'base',
                     'ports': [],
-                    'resources': {'limits': {'cpu': None,
-                                             'memory': None,
-                                             'nvidia.com/gpu': None},
-                                  'requests': {'cpu': None,
-                                               'memory': None}},
                     'volumeMounts': [],
                 }],
                 'hostNetwork': False,
                 'imagePullSecrets': [],
+                'initContainers': [],
                 'nodeSelector': {},
                 'restartPolicy': 'Never',
                 'securityContext': {},
@@ -85,10 +104,15 @@ class TestKubernetesPodOperator(unittest.TestCase):
             }
         }
 
-    def test_config_path_move(self):
+    def tearDown(self):
+        client = kube_client.get_kube_client(in_cluster=False)
+        client.delete_collection_namespaced_pod(namespace="default")
+
+    def test_do_xcom_push_defaults_false(self):
         new_config_path = '/tmp/kube_config'
         old_config_path = os.path.expanduser('~/.kube/config')
         shutil.copy(old_config_path, new_config_path)
+
         k = KubernetesPodOperator(
             namespace='default',
             image="ubuntu:16.04",
@@ -101,43 +125,31 @@ class TestKubernetesPodOperator(unittest.TestCase):
             do_xcom_push=False,
             config_file=new_config_path,
         )
-        k.execute(None)
-        actual_pod = self.api_client.sanitize_for_serialization(k.pod)
-        self.assertEqual(self.expected_pod, actual_pod)
+        self.assertFalse(k.do_xcom_push)
 
-    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.run_pod")
-    @mock.patch("airflow.kubernetes.kube_client.get_kube_client")
-    def test_config_path(self, client_mock, launcher_mock):
-        from airflow.utils.state import State
+    def test_config_path_move(self):
+        new_config_path = '/tmp/kube_config'
+        old_config_path = os.path.expanduser('~/.kube/config')
+        shutil.copy(old_config_path, new_config_path)
 
-        file_path = "/tmp/fake_file"
         k = KubernetesPodOperator(
             namespace='default',
             image="ubuntu:16.04",
             cmds=["bash", "-cx"],
             arguments=["echo 10"],
             labels={"foo": "bar"},
-            name="test",
+            name="test1",
             task_id="task",
             in_cluster=False,
             do_xcom_push=False,
-            config_file=file_path,
-            cluster_context='default',
-        )
-        launcher_mock.return_value = (State.SUCCESS, None)
-        k.execute(None)
-        client_mock.assert_called_once_with(
-            in_cluster=False,
-            cluster_context='default',
-            config_file=file_path,
+            config_file=new_config_path,
         )
+        context = create_context(k)
+        k.execute(context)
+        actual_pod = self.api_client.sanitize_for_serialization(k.pod)
+        self.assertEqual(self.expected_pod, actual_pod)
 
-    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.run_pod")
-    @mock.patch("airflow.kubernetes.kube_client.get_kube_client")
-    def test_image_pull_secrets_correctly_set(self, mock_client, mock_launcher):
-        from airflow.utils.state import State
-
-        fake_pull_secrets = "fakeSecret"
+    def test_working_pod(self):
         k = KubernetesPodOperator(
             namespace='default',
             image="ubuntu:16.04",
@@ -146,22 +158,16 @@ class TestKubernetesPodOperator(unittest.TestCase):
             labels={"foo": "bar"},
             name="test",
             task_id="task",
-            image_pull_secrets=fake_pull_secrets,
             in_cluster=False,
             do_xcom_push=False,
-            cluster_context='default',
-        )
-        mock_launcher.return_value = (State.SUCCESS, None)
-        k.execute(None)
-        self.assertEqual(
-            mock_launcher.call_args[0][0].spec.image_pull_secrets,
-            [k8s.V1LocalObjectReference(name=fake_pull_secrets)]
         )
+        context = create_context(k)
+        k.execute(context)
+        actual_pod = self.api_client.sanitize_for_serialization(k.pod)
+        self.assertEqual(self.expected_pod['spec'], actual_pod['spec'])
+        self.assertEqual(self.expected_pod['metadata']['labels'], actual_pod['metadata']['labels'])
 
-    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.run_pod")
-    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.delete_pod")
-    @mock.patch("airflow.kubernetes.kube_client.get_kube_client")
-    def test_pod_delete_even_on_launcher_error(self, mock_client, delete_pod_mock, run_pod_mock):
+    def test_delete_operator_pod(self):
         k = KubernetesPodOperator(
             namespace='default',
             image="ubuntu:16.04",
@@ -172,31 +178,15 @@ class TestKubernetesPodOperator(unittest.TestCase):
             task_id="task",
             in_cluster=False,
             do_xcom_push=False,
-            cluster_context='default',
             is_delete_operator_pod=True,
         )
-        run_pod_mock.side_effect = AirflowException('fake failure')
-        with self.assertRaises(AirflowException):
-            k.execute(None)
-        assert delete_pod_mock.called
-
-    def test_working_pod(self):
-        k = KubernetesPodOperator(
-            namespace='default',
-            image="ubuntu:16.04",
-            cmds=["bash", "-cx"],
-            arguments=["echo 10"],
-            labels={"foo": "bar"},
-            name="test",
-            task_id="task",
-            in_cluster=False,
-            do_xcom_push=False,
-        )
-        k.execute(None)
+        context = create_context(k)
+        k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
-        self.assertEqual(self.expected_pod, actual_pod)
+        self.assertEqual(self.expected_pod['spec'], actual_pod['spec'])
+        self.assertEqual(self.expected_pod['metadata']['labels'], actual_pod['metadata']['labels'])
 
-    def test_delete_operator_pod(self):
+    def test_pod_hostnetwork(self):
         k = KubernetesPodOperator(
             namespace='default',
             image="ubuntu:16.04",
@@ -205,15 +195,19 @@ class TestKubernetesPodOperator(unittest.TestCase):
             labels={"foo": "bar"},
             name="test",
             task_id="task",
-            is_delete_operator_pod=True,
             in_cluster=False,
             do_xcom_push=False,
+            hostnetwork=True,
         )
-        k.execute(None)
+        context = create_context(k)
+        k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
-        self.assertEqual(self.expected_pod, actual_pod)
+        self.expected_pod['spec']['hostNetwork'] = True
+        self.assertEqual(self.expected_pod['spec'], actual_pod['spec'])
+        self.assertEqual(self.expected_pod['metadata']['labels'], actual_pod['metadata']['labels'])
 
-    def test_pod_hostnetwork(self):
+    def test_pod_dnspolicy(self):
+        dns_policy = "ClusterFirstWithHostNet"
         k = KubernetesPodOperator(
             namespace='default',
             image="ubuntu:16.04",
@@ -225,16 +219,20 @@ class TestKubernetesPodOperator(unittest.TestCase):
             in_cluster=False,
             do_xcom_push=False,
             hostnetwork=True,
+            dnspolicy=dns_policy
         )
-        k.execute(None)
+        context = create_context(k)
+        k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.expected_pod['spec']['hostNetwork'] = True
-        self.assertEqual(self.expected_pod, actual_pod)
+        self.expected_pod['spec']['dnsPolicy'] = dns_policy
+        self.assertEqual(self.expected_pod['spec'], actual_pod['spec'])
+        self.assertEqual(self.expected_pod['metadata']['labels'], actual_pod['metadata']['labels'])
 
-    def test_pod_dnspolicy(self):
-        dns_policy = "ClusterFirstWithHostNet"
+    def test_pod_schedulername(self):
+        scheduler_name = "default-scheduler"
         k = KubernetesPodOperator(
-            namespace='default',
+            namespace="default",
             image="ubuntu:16.04",
             cmds=["bash", "-cx"],
             arguments=["echo 10"],
@@ -243,13 +241,12 @@ class TestKubernetesPodOperator(unittest.TestCase):
             task_id="task",
             in_cluster=False,
             do_xcom_push=False,
-            hostnetwork=True,
-            dnspolicy=dns_policy,
+            schedulername=scheduler_name
         )
-        k.execute(None)
+        context = create_context(k)
+        k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
-        self.expected_pod['spec']['hostNetwork'] = True
-        self.expected_pod['spec']['dnsPolicy'] = dns_policy
+        self.expected_pod['spec']['schedulerName'] = scheduler_name
         self.assertEqual(self.expected_pod, actual_pod)
 
     def test_pod_node_selectors(self):
@@ -268,7 +265,8 @@ class TestKubernetesPodOperator(unittest.TestCase):
             do_xcom_push=False,
             node_selectors=node_selectors,
         )
-        k.execute(None)
+        context = create_context(k)
+        k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.expected_pod['spec']['nodeSelector'] = node_selectors
         self.assertEqual(self.expected_pod, actual_pod)
@@ -277,8 +275,10 @@ class TestKubernetesPodOperator(unittest.TestCase):
         resources = {
             'limit_cpu': 0.25,
             'limit_memory': '64Mi',
+            'limit_ephemeral_storage': '2Gi',
             'request_cpu': '250m',
             'request_memory': '64Mi',
+            'request_ephemeral_storage': '1Gi',
         }
         k = KubernetesPodOperator(
             namespace='default',
@@ -292,17 +292,20 @@ class TestKubernetesPodOperator(unittest.TestCase):
             do_xcom_push=False,
             resources=resources,
         )
-        k.execute(None)
+        context = create_context(k)
+        k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.expected_pod['spec']['containers'][0]['resources'] = {
             'requests': {
                 'memory': '64Mi',
-                'cpu': '250m'
+                'cpu': '250m',
+                'ephemeral-storage': '1Gi'
             },
             'limits': {
                 'memory': '64Mi',
                 'cpu': 0.25,
-                'nvidia.com/gpu': None
+                'nvidia.com/gpu': None,
+                'ephemeral-storage': '2Gi'
             }
         }
         self.assertEqual(self.expected_pod, actual_pod)
@@ -337,7 +340,8 @@ class TestKubernetesPodOperator(unittest.TestCase):
             do_xcom_push=False,
             affinity=affinity,
         )
-        k.execute(None)
+        context = create_context(k)
+        k.execute(context=context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.expected_pod['spec']['affinity'] = affinity
         self.assertEqual(self.expected_pod, actual_pod)
@@ -357,7 +361,8 @@ class TestKubernetesPodOperator(unittest.TestCase):
             do_xcom_push=False,
             ports=[port],
         )
-        k.execute(None)
+        context = create_context(k)
+        k.execute(context=context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.expected_pod['spec']['containers'][0]['ports'] = [{
             'name': 'http',
@@ -366,7 +371,7 @@ class TestKubernetesPodOperator(unittest.TestCase):
         self.assertEqual(self.expected_pod, actual_pod)
 
     def test_volume_mount(self):
-        with mock.patch.object(PodLauncher, 'log') as mock_logger:
+        with patch.object(PodLauncher, 'log') as mock_logger:
             volume_mount = VolumeMount('test-volume',
                                        mount_path='/tmp/test_volume',
                                        sub_path=None,
@@ -394,7 +399,8 @@ class TestKubernetesPodOperator(unittest.TestCase):
                 in_cluster=False,
                 do_xcom_push=False,
             )
-            k.execute(None)
+            context = create_context(k)
+            k.execute(context=context)
             mock_logger.info.assert_any_call(b"retrieved from mount\n")
             actual_pod = self.api_client.sanitize_for_serialization(k.pod)
             self.expected_pod['spec']['containers'][0]['args'] = args
@@ -429,7 +435,8 @@ class TestKubernetesPodOperator(unittest.TestCase):
             do_xcom_push=False,
             security_context=security_context,
         )
-        k.execute(None)
+        context = create_context(k)
+        k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.expected_pod['spec']['securityContext'] = security_context
         self.assertEqual(self.expected_pod, actual_pod)
@@ -453,7 +460,8 @@ class TestKubernetesPodOperator(unittest.TestCase):
             do_xcom_push=False,
             security_context=security_context,
         )
-        k.execute(None)
+        context = create_context(k)
+        k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.expected_pod['spec']['securityContext'] = security_context
         self.assertEqual(self.expected_pod, actual_pod)
@@ -477,7 +485,8 @@ class TestKubernetesPodOperator(unittest.TestCase):
             do_xcom_push=False,
             security_context=security_context,
         )
-        k.execute(None)
+        context = create_context(k)
+        k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.expected_pod['spec']['securityContext'] = security_context
         self.assertEqual(self.expected_pod, actual_pod)
@@ -494,10 +503,11 @@ class TestKubernetesPodOperator(unittest.TestCase):
             task_id="task",
             in_cluster=False,
             do_xcom_push=False,
-            startup_timeout_seconds=5
+            startup_timeout_seconds=5,
         )
         with self.assertRaises(AirflowException):
-            k.execute(None)
+            context = create_context(k)
+            k.execute(context)
             actual_pod = self.api_client.sanitize_for_serialization(k.pod)
             self.expected_pod['spec']['containers'][0]['image'] = bad_image_name
             self.assertEqual(self.expected_pod, actual_pod)
@@ -518,7 +528,8 @@ class TestKubernetesPodOperator(unittest.TestCase):
             service_account_name=bad_service_account_name,
         )
         with self.assertRaises(ApiException):
-            k.execute(None)
+            context = create_context(k)
+            k.execute(context)
             actual_pod = self.api_client.sanitize_for_serialization(k.pod)
             self.expected_pod['spec']['serviceAccountName'] = bad_service_account_name
             self.assertEqual(self.expected_pod, actual_pod)
@@ -540,7 +551,8 @@ class TestKubernetesPodOperator(unittest.TestCase):
             do_xcom_push=False,
         )
         with self.assertRaises(AirflowException):
-            k.execute(None)
+            context = create_context(k)
+            k.execute(context)
             actual_pod = self.api_client.sanitize_for_serialization(k.pod)
             self.expected_pod['spec']['containers'][0]['args'] = bad_internal_command
             self.assertEqual(self.expected_pod, actual_pod)
@@ -559,20 +571,22 @@ class TestKubernetesPodOperator(unittest.TestCase):
             in_cluster=False,
             do_xcom_push=True,
         )
-        self.assertEqual(k.execute(None), json.loads(return_value))
+        context = create_context(k)
+        self.assertEqual(k.execute(context), json.loads(return_value))
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         volume = self.api_client.sanitize_for_serialization(PodDefaults.VOLUME)
         volume_mount = self.api_client.sanitize_for_serialization(PodDefaults.VOLUME_MOUNT)
         container = self.api_client.sanitize_for_serialization(PodDefaults.SIDECAR_CONTAINER)
         self.expected_pod['spec']['containers'][0]['args'] = args
-        self.expected_pod['spec']['containers'][0]['volumeMounts'].insert(0, volume_mount)
+        self.expected_pod['spec']['containers'][0]['volumeMounts'].insert(0, volume_mount)  # noqa
         self.expected_pod['spec']['volumes'].insert(0, volume)
         self.expected_pod['spec']['containers'].append(container)
         self.assertEqual(self.expected_pod, actual_pod)
 
-    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.run_pod")
-    @mock.patch("airflow.kubernetes.kube_client.get_kube_client")
-    def test_envs_from_configmaps(self, mock_client, mock_launcher):
+    @patch("airflow.kubernetes.pod_launcher.PodLauncher.start_pod")
+    @patch("airflow.kubernetes.pod_launcher.PodLauncher.monitor_pod")
+    @patch("airflow.kubernetes.kube_client.get_kube_client")
+    def test_envs_from_configmaps(self, mock_client, mock_monitor, mock_start):
         # GIVEN
         from airflow.utils.state import State
 
@@ -588,21 +602,23 @@ class TestKubernetesPodOperator(unittest.TestCase):
             task_id="task",
             in_cluster=False,
             do_xcom_push=False,
-            configmaps=[configmap]
+            configmaps=[configmap],
         )
         # THEN
-        mock_launcher.return_value = (State.SUCCESS, None)
-        k.execute(None)
+        mock_monitor.return_value = (State.SUCCESS, None)
+        context = create_context(k)
+        k.execute(context)
         self.assertEqual(
-            mock_launcher.call_args[0][0].spec.containers[0].env_from,
+            mock_start.call_args[0][0].spec.containers[0].env_from,
             [k8s.V1EnvFromSource(config_map_ref=k8s.V1ConfigMapEnvSource(
                 name=configmap
             ))]
         )
 
-    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.run_pod")
-    @mock.patch("airflow.kubernetes.kube_client.get_kube_client")
-    def test_envs_from_secrets(self, mock_client, mock_launcher):
+    @patch("airflow.kubernetes.pod_launcher.PodLauncher.start_pod")
+    @patch("airflow.kubernetes.pod_launcher.PodLauncher.monitor_pod")
+    @patch("airflow.kubernetes.kube_client.get_kube_client")
+    def test_envs_from_secrets(self, mock_client, monitor_mock, start_mock):
         # GIVEN
         from airflow.utils.state import State
         secret_ref = 'secret_name'
@@ -621,16 +637,194 @@ class TestKubernetesPodOperator(unittest.TestCase):
             do_xcom_push=False,
         )
         # THEN
-        mock_launcher.return_value = (State.SUCCESS, None)
-        k.execute(None)
+        monitor_mock.return_value = (State.SUCCESS, None)
+        context = create_context(k)
+        k.execute(context)
         self.assertEqual(
-            mock_launcher.call_args[0][0].spec.containers[0].env_from,
+            start_mock.call_args[0][0].spec.containers[0].env_from,
             [k8s.V1EnvFromSource(secret_ref=k8s.V1SecretEnvSource(
                 name=secret_ref
             ))]
         )
 
+    def test_init_container(self):
+        # GIVEN
+        volume_mounts = [k8s.V1VolumeMount(
+            mount_path='/etc/foo',
+            name='test-volume',
+            sub_path=None,
+            read_only=True
+        )]
+
+        init_environments = [k8s.V1EnvVar(
+            name='key1',
+            value='value1'
+        ), k8s.V1EnvVar(
+            name='key2',
+            value='value2'
+        )]
+
+        init_container = k8s.V1Container(
+            name="init-container",
+            image="ubuntu:16.04",
+            env=init_environments,
+            volume_mounts=volume_mounts,
+            command=["bash", "-cx"],
+            args=["echo 10"]
+        )
+
+        volume_config = {
+            'persistentVolumeClaim':
+                {
+                    'claimName': 'test-volume'
+                }
+        }
+        volume = Volume(name='test-volume', configs=volume_config)
+
+        expected_init_container = {
+            'name': 'init-container',
+            'image': 'ubuntu:16.04',
+            'command': ['bash', '-cx'],
+            'args': ['echo 10'],
+            'env': [{
+                'name': 'key1',
+                'value': 'value1'
+            }, {
+                'name': 'key2',
+                'value': 'value2'
+            }],
+            'volumeMounts': [{
+                'mountPath': '/etc/foo',
+                'name': 'test-volume',
+                'readOnly': True
+            }],
+        }
+
+        k = KubernetesPodOperator(
+            namespace='default',
+            image="ubuntu:16.04",
+            cmds=["bash", "-cx"],
+            arguments=["echo 10"],
+            labels={"foo": "bar"},
+            name="test",
+            task_id="task",
+            volumes=[volume],
+            init_containers=[init_container],
+            in_cluster=False,
+            do_xcom_push=False,
+        )
+        context = create_context(k)
+        k.execute(context)
+        actual_pod = self.api_client.sanitize_for_serialization(k.pod)
+        self.expected_pod['spec']['initContainers'] = [expected_init_container]
+        self.expected_pod['spec']['volumes'] = [{
+            'name': 'test-volume',
+            'persistentVolumeClaim': {
+                'claimName': 'test-volume'
+            }
+        }]
+        self.assertEqual(self.expected_pod, actual_pod)
+
+    @patch("airflow.kubernetes.pod_launcher.PodLauncher.start_pod")
+    @patch("airflow.kubernetes.pod_launcher.PodLauncher.monitor_pod")
+    @patch("airflow.kubernetes.kube_client.get_kube_client")
+    def test_pod_template_file(
+            self,
+            mock_client,
+            monitor_mock,
+            start_mock):  # pylint: disable=unused-argument
+        from airflow.utils.state import State
+        k = KubernetesPodOperator(
+            task_id='task',
+            pod_template_file='tests/kubernetes/pod.yaml',
+            do_xcom_push=True
+        )
+        monitor_mock.return_value = (State.SUCCESS, None)
+        context = create_context(k)
+        k.execute(context)
+        actual_pod = self.api_client.sanitize_for_serialization(k.pod)
+        self.assertEqual({
+            'apiVersion': 'v1',
+            'kind': 'Pod',
+            'metadata': {'name': mock.ANY, 'namespace': 'mem-example'},
+            'spec': {
+                'volumes': [{'name': 'xcom', 'emptyDir': {}}],
+                'containers': [{
+                    'args': ['--vm', '1', '--vm-bytes', '150M', '--vm-hang', '1'],
+                    'command': ['stress'],
+                    'image': 'polinux/stress',
+                    'name': 'memory-demo-ctr',
+                    'resources': {
+                        'limits': {'memory': '200Mi'},
+                        'requests': {'memory': '100Mi'}
+                    },
+                    'volumeMounts': [{
+                        'name': 'xcom',
+                        'mountPath': '/airflow/xcom'
+                    }]
+                }, {
+                    'name': 'airflow-xcom-sidecar',
+                    'image': "alpine",
+                    'command': ['sh', '-c', PodDefaults.XCOM_CMD],
+                    'volumeMounts': [
+                        {
+                            'name': 'xcom',
+                            'mountPath': '/airflow/xcom'
+                        }
+                    ],
+                    'resources': {'requests': {'cpu': '1m'}},
+                }],
+            }
+        }, actual_pod)
+
+    @patch("airflow.kubernetes.pod_launcher.PodLauncher.start_pod")
+    @patch("airflow.kubernetes.pod_launcher.PodLauncher.monitor_pod")
+    @patch("airflow.kubernetes.kube_client.get_kube_client")
+    def test_pod_priority_class_name(
+            self,
+            mock_client,
+            monitor_mock,
+            start_mock):  # pylint: disable=unused-argument
+        """Test ability to assign priorityClassName to pod
+
+        """
+        from airflow.utils.state import State
+
+        priority_class_name = "medium-test"
+        k = KubernetesPodOperator(
+            namespace='default',
+            image="ubuntu:16.04",
+            cmds=["bash", "-cx"],
+            arguments=["echo 10"],
+            labels={"foo": "bar"},
+            name="test",
+            task_id="task",
+            in_cluster=False,
+            do_xcom_push=False,
+            priority_class_name=priority_class_name,
+        )
+
+        monitor_mock.return_value = (State.SUCCESS, None)
+        context = create_context(k)
+        k.execute(context)
+        actual_pod = self.api_client.sanitize_for_serialization(k.pod)
+        self.expected_pod['spec']['priorityClassName'] = priority_class_name
+        self.assertEqual(self.expected_pod, actual_pod)
+
+    def test_pod_name(self):
+        pod_name_too_long = "a" * 221
+        with self.assertRaises(AirflowException):
+            KubernetesPodOperator(
+                namespace='default',
+                image="ubuntu:16.04",
+                cmds=["bash", "-cx"],
+                arguments=["echo 10"],
+                labels={"foo": "bar"},
+                name=pod_name_too_long,
+                task_id="task",
+                in_cluster=False,
+                do_xcom_push=False,
+            )
+
 
 # pylint: enable=unused-argument
-if __name__ == '__main__':
-    unittest.main()
diff --git a/scripts/ci/_utils.sh b/scripts/ci/_utils.sh
index 68778d8..2167229 100644
--- a/scripts/ci/_utils.sh
+++ b/scripts/ci/_utils.sh
@@ -134,7 +134,7 @@ function initialize_common_environment {
     # Add the right volume mount for sources, depending which mount strategy is used
     if [[ ${MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS} == "true" ]]; then
         print_info
-        print_info "Mount whole airflow source directory for static checks (make sure all files are in container)"
+        print_info "Mount whole airflow source directory for static checks"
         print_info
         EXTRA_DOCKER_FLAGS=( \
           "-v" "${AIRFLOW_SOURCES}:/opt/airflow" \
@@ -208,12 +208,11 @@ function initialize_common_environment {
     # Default extras used for building CI image
     export DEFAULT_CI_EXTRAS="devel_ci"
 
-
     # Default extras used for building Production image. The master of this information is in the Dockerfile
     DEFAULT_PROD_EXTRAS=$(grep "ARG AIRFLOW_EXTRAS=" "${AIRFLOW_SOURCES}/Dockerfile"|
             awk 'BEGIN { FS="=" } { print $2 }' | tr -d '"')
 
-    # By default we build CI images  but when we specify `--producton-image` we switch to production image
+    # By default we build CI images  but when we specify `--production-image` we switch to production image
     export PRODUCTION_IMAGE="false"
 
     # The SQLlite URL used for sqlite runs
@@ -221,6 +220,16 @@ function initialize_common_environment {
 
     # Determines if airflow should be installed from a specified reference in GitHub
     export INSTALL_AIRFLOW_REFERENCE=""
+
+    # Version of Kubernetes to run
+    export KUBERNETES_VERSION="${KUBERNETES_VERSION:="v1.15.3"}"
+
+    # Name of the KinD cluster to connect to
+    export KIND_CLUSTER_NAME=${KIND_CLUSTER_NAME:="airflow-python-${PYTHON_MAJOR_MINOR_VERSION}-${KUBERNETES_VERSION}"}
+
+    # Name of the KinD cluster to connect to when referred to via kubectl
+    export KUBECTL_CLUSTER_NAME=kind-${KIND_CLUSTER_NAME}
+
 }
 
 # Prints verbose information in case VERBOSE variable is set
@@ -267,6 +276,7 @@ function generate_local_mounts_list {
         "$prefix"setup.cfg:/opt/airflow/setup.cfg:cached
         "$prefix"setup.py:/opt/airflow/setup.py:cached
         "$prefix"tests:/opt/airflow/tests:cached
+        "$prefix"kubernetes_tests:/opt/airflow/kubernetes_tests:cached
         "$prefix"tmp:/opt/airflow/tmp:cached
     )
 }
@@ -526,7 +536,7 @@ function assert_not_in_container() {
     fi
 }
 
-# Removes the "Forced answer" (yes/no/quit) given previously, unles you specifically want to remember it.
+# Removes the "Forced answer" (yes/no/quit) given previously, unless you specifically want to remember it.
 #
 # This is the default behaviour of all rebuild scripts to ask independently whether you want to
 # rebuild the image or not. Sometimes however we want to remember answer previously given. For
@@ -711,7 +721,7 @@ function get_remote_image_info() {
     # delete container just in case
     verbose_docker rm --force "remote-airflow-manifest" >/dev/null 2>&1
     set -e
-    # Create container out of the manifest image without runnning it
+    # Create container out of the manifest image without running it
     verbose_docker create --name "remote-airflow-manifest" "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}" >/dev/null 2>&1
     # Extract manifest and store it in local file
     verbose_docker cp "remote-airflow-manifest:${AIRFLOW_CI_BASE_TAG}.json" "${TMP_MANIFEST_REMOTE_JSON}" >/dev/null 2>&1
@@ -751,7 +761,7 @@ function compare_layers() {
     if (( NUM_DIFF >= MAGIC_CUT_OFF_NUMBER_OF_LAYERS )); then
         echo
         echo
-        echo "WARNING! Your image and the dockerhub image differ signifcantly"
+        echo "WARNING! Your image and the dockerhub image differ significantly"
         echo
         echo "Forcing pulling the images. It will be faster than rebuilding usually."
         echo "You can avoid it by setting SKIP_CHECK_REMOTE_IMAGE to true"
@@ -986,6 +996,8 @@ function get_ci_environment() {
     export CI_EVENT_TYPE="manual"
     export CI_TARGET_REPO="apache/airflow"
     export CI_TARGET_BRANCH="master"
+    export CI_SOURCE_REPO="apache/airflow"
+    export CI_SOURCE_BRANCH="master"
     export CI_BUILD_ID="default-build-id"
     export CI_JOB_ID="default-job-id"
     if [[ ${CI:=} != "true" ]]; then
@@ -994,29 +1006,50 @@ function get_ci_environment() {
         echo
     else
         if [[ ${TRAVIS:=} == "true" ]]; then
+            export CI_TARGET_REPO="${TRAVIS_REPO_SLUG}"
+            export CI_TARGET_BRANCH="${TRAVIS_BRANCH}"
+            export CI_BUILD_ID="${TRAVIS_BUILD_ID}"
+            export CI_JOB_ID="${TRAVIS_JOB_ID}"
             if [[ "${TRAVIS_PULL_REQUEST:=}" == "true" ]]; then
                 export CI_EVENT_TYPE="pull_request"
+                export CI_SOURCE_REPO="${TRAVIS_PULL_REQUEST_SLUG}"
+                export CI_SOURCE_BRANCH="${TRAVIS_PULL_REQUEST_BRANCH}"
             elif [[ "${TRAVIS_EVENT_TYPE:=}" == "cron" ]]; then
                 export CI_EVENT_TYPE="schedule"
             else
                 export CI_EVENT_TYPE="push"
             fi
-            export CI_TARGET_BRANCH="${TRAVIS_BRANCH}"
-            export CI_TARGET_REPO="${TRAVIS_REPO_SLUG}"
-            export CI_BUILD_ID="${TRAVIS_BUILD_ID}"
-            export CI_JOB_ID="${TRAVIS_JOB_ID}"
         elif [[ ${GITHUB_ACTIONS:=} == "true" ]]; then
+            export CI_TARGET_REPO="${GITHUB_REPOSITORY}"
+            export CI_TARGET_BRANCH="${GITHUB_BASE_REF}"
+            export CI_BUILD_ID="${GITHUB_RUN_ID}"
+            export CI_JOB_ID="${GITHUB_JOB}"
             if [[ ${GITHUB_EVENT_NAME:=} == "pull_request" ]]; then
                 export CI_EVENT_TYPE="pull_request"
+                # default name of the source repo (assuming it's forked without rename)
+                export SOURCE_AIRFLOW_REPO=${SOURCE_AIRFLOW_REPO:="airflow"}
+                # For Pull Requests it's ambiguous to find the PR and we need to
+                # assume that name of repo is airflow but it could be overridden in case it's not
+                export CI_SOURCE_REPO="${GITHUB_ACTOR}/${SOURCE_AIRFLOW_REPO}"
+                export CI_SOURCE_BRANCH="${GITHUB_HEAD_REF}"
+                BRANCH_EXISTS=$(git ls-remote --heads \
+                    "https://github.com/${CI_SOURCE_REPO}.git" "${CI_SOURCE_BRANCH}")
+                if [[ ${BRANCH_EXISTS} == "" ]]; then
+                    echo
+                    echo "https://github.com/${CI_SOURCE_REPO}.git Branch ${CI_SOURCE_BRANCH} does not exist"
+                    echo
+                    echo
+                    echo "Fallback to https://github.com/${CI_TARGET_REPO}.git Branch ${CI_TARGET_BRANCH}"
+                    echo
+                    # Fallback to the target repository if the repo does not exist
+                    export CI_SOURCE_REPO="${CI_TARGET_REPO}"
+                    export CI_SOURCE_BRANCH="${CI_TARGET_BRANCH}"
+                fi
             elif [[ ${GITHUB_EVENT_TYPE:=} == "schedule" ]]; then
                 export CI_EVENT_TYPE="schedule"
             else
                 export CI_EVENT_TYPE="push"
             fi
-            export CI_TARGET_REPO="${GITHUB_REPOSITORY}"
-            export CI_TARGET_BRANCH="${GITHUB_BASE_REF}"
-            export CI_BUILD_ID="${GITHUB_RUN_ID}"
-            export CI_JOB_ID="${GITHUB_JOB}"
         else
             echo
             echo "ERROR! Unknown CI environment. Exiting"
@@ -1029,6 +1062,8 @@ function get_ci_environment() {
     echo "CI_EVENT_TYPE=${CI_EVENT_TYPE}"
     echo "CI_TARGET_REPO=${CI_TARGET_REPO}"
     echo "CI_TARGET_BRANCH=${CI_TARGET_BRANCH}"
+    echo "CI_SOURCE_REPO=${CI_SOURCE_REPO}"
+    echo "CI_SOURCE_BRANCH=${CI_SOURCE_BRANCH}"
     echo "CI_BUILD_ID=${CI_BUILD_ID}"
     echo "CI_JOB_ID=${CI_JOB_ID}"
     echo
@@ -1386,6 +1421,7 @@ function build_prod_image() {
     fi
 }
 
+
 # Removes airflow CI and base images
 function remove_all_images() {
     echo
@@ -1558,6 +1594,11 @@ function prepare_prod_build() {
         export CACHED_AIRFLOW_PROD_BUILD_IMAGE=""
         export CACHED_PYTHON_BASE_IMAGE=""
     fi
+    export AIRFLOW_KUBERNETES_IMAGE=${AIRFLOW_PROD_IMAGE}-kubernetes
+    AIRFLOW_KUBERNETES_IMAGE_NAME=$(echo "${AIRFLOW_KUBERNETES_IMAGE}" | cut -f 1 -d ":")
+    export AIRFLOW_KUBERNETES_IMAGE_NAME
+    AIRFLOW_KUBERNETES_IMAGE_TAG=$(echo "${AIRFLOW_KUBERNETES_IMAGE}" | cut -f 2 -d ":")
+    export AIRFLOW_KUBERNETES_IMAGE_TAG
 
     if [[ "${INSTALL_AIRFLOW_REFERENCE:=}" != "" ]]; then
         # When --install-airflow-reference is used then the image is build from github tag
@@ -1666,3 +1707,437 @@ function set_mysql_encoding() {
 function get_airflow_version_from_production_image() {
      docker run --entrypoint /bin/bash "${AIRFLOW_PROD_IMAGE}" -c 'echo "${AIRFLOW_VERSION}"'
 }
+
+function dump_kind_logs() {
+    echo "###########################################################################################"
+    echo "                   Dumping logs from KIND"
+    echo "###########################################################################################"
+
+    FILE_NAME="${1}"
+    kind --name "${KIND_CLUSTER_NAME}" export logs "${FILE_NAME}"
+}
+
+
+function send_kubernetes_logs_to_file_io() {
+    echo "##############################################################################"
+    echo
+    echo "   DUMPING LOG FILES FROM KIND AND SENDING THEM TO file.io"
+    echo
+    echo "##############################################################################"
+    DUMP_DIR_NAME=$(date "+%Y-%m-%d")_kind_${CI_BUILD_ID:="default"}_${CI_JOB_ID:="default"}
+    DUMP_DIR=/tmp/${DUMP_DIR_NAME}
+    dump_kind_logs "${DUMP_DIR}"
+    tar -cvzf "${DUMP_DIR}.tar.gz" -C /tmp "${DUMP_DIR_NAME}"
+    echo
+    echo "   Logs saved to ${DUMP_DIR}.tar.gz"
+    echo
+    echo "##############################################################################"
+    curl -F "file=@${DUMP_DIR}.tar.gz" https://file.io
+}
+
+function check_kind_and_kubectl_are_installed() {
+    SYSTEM=$(uname -s| tr '[:upper:]' '[:lower:]')
+    KIND_VERSION="v0.7.0"
+    KIND_URL="https://github.com/kubernetes-sigs/kind/releases/download/${KIND_VERSION}/kind-${SYSTEM}-amd64"
+    KIND_PATH="${BUILD_CACHE_DIR}/bin/kind"
+    KUBECTL_VERSION="v1.15.3"
+    KUBECTL_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/${SYSTEM}/amd64/kubectl"
+    KUBECTL_PATH="${BUILD_CACHE_DIR}/bin/kubectl"
+    mkdir -pv "${BUILD_CACHE_DIR}/bin"
+    if [[ ! -f "${KIND_PATH}" ]]; then
+        echo
+        echo "Downloading Kind version ${KIND_VERSION}"
+        echo
+        curl --fail --location "${KIND_URL}" --output "${KIND_PATH}"
+        chmod +x "${KIND_PATH}"
+    fi
+    if [[ ! -f "${KUBECTL_PATH}" ]]; then
+        echo
+        echo "Downloading Kubectl version ${KUBECTL_VERSION}"
+        echo
+        curl --fail --location "${KUBECTL_URL}" --output "${KUBECTL_PATH}"
+        chmod +x "${KUBECTL_PATH}"
+    fi
+    PATH=${PATH}:${BUILD_CACHE_DIR}/bin
+}
+
+function create_cluster() {
+    if [[ "${TRAVIS:="false"}" == "true" ]]; then
+        # Travis CI does not handle the nice output of Kind well, so we need to capture it
+        # And display only if kind fails to start
+        start_output_heartbeat "Creating kubernetes cluster" 10
+        set +e
+        if ! OUTPUT=$(kind create cluster \
+                        --name "${KIND_CLUSTER_NAME}" \
+                        --config "${AIRFLOW_SOURCES}/scripts/ci/kubernetes/kind-cluster-conf.yaml" \
+                        --image "kindest/node:${KUBERNETES_VERSION}" 2>&1); then
+            echo "${OUTPUT}"
+        fi
+        stop_output_heartbeat
+    else
+        kind create cluster \
+            --name "${KIND_CLUSTER_NAME}" \
+            --config "${AIRFLOW_SOURCES}/scripts/ci/kubernetes/kind-cluster-conf.yaml" \
+            --image "kindest/node:${KUBERNETES_VERSION}"
+    fi
+    echo
+    echo "Created cluster ${KIND_CLUSTER_NAME}"
+    echo
+
+}
+
+function delete_cluster() {
+    kind delete cluster --name "${KIND_CLUSTER_NAME}"
+    echo
+    echo "Deleted cluster ${KIND_CLUSTER_NAME}"
+    echo
+    rm -rf "${HOME}/.kube/*"
+}
+
+function perform_kind_cluster_operation() {
+    OPERATION="${1}"
+    ALL_CLUSTERS=$(kind get clusters || true)
+
+    echo
+    echo "Kubernetes mode: ${KUBERNETES_MODE}"
+    echo
+
+    if [[ ${OPERATION} == "status" ]]; then
+        if [[ ${ALL_CLUSTERS} == *"${KIND_CLUSTER_NAME}"* ]]; then
+            echo
+            echo "Cluster name: ${KIND_CLUSTER_NAME}"
+            echo
+            kind get nodes --name "${KIND_CLUSTER_NAME}"
+            echo
+            exit
+        else
+            echo
+            echo "Cluster ${KIND_CLUSTER_NAME} is not running"
+            echo
+            exit
+        fi
+    fi
+    if [[ ${ALL_CLUSTERS} == *"${KIND_CLUSTER_NAME}"* ]]; then
+        if [[ ${OPERATION} == "start" ]]; then
+            echo
+            echo "Cluster ${KIND_CLUSTER_NAME} is already created"
+            echo "Reusing previously created cluster"
+            echo
+        elif [[ ${OPERATION} == "restart" ]]; then
+            echo
+            echo "Recreating cluster"
+            echo
+            delete_cluster
+            create_cluster
+        elif [[ ${OPERATION} == "stop" ]]; then
+            echo
+            echo "Deleting cluster"
+            echo
+            delete_cluster
+            exit
+        elif [[ ${OPERATION} == "deploy" ]]; then
+            echo
+            echo "Deploying Airflow to KinD"
+            echo
+            get_ci_environment
+            check_kind_and_kubectl_are_installed
+            build_kubernetes_image
+            load_image_to_kind_cluster
+            prepare_kubernetes_app_variables
+            prepare_kubernetes_resources
+            apply_kubernetes_resources
+            wait_for_airflow_pods_up_and_running
+            wait_for_airflow_webserver_up_and_running
+        elif [[ ${OPERATION} == "test" ]]; then
+            echo
+            echo "Testing with kind to KinD"
+            echo
+            "${AIRFLOW_SOURCES}/scripts/ci/ci_run_kubernetes_tests.sh"
+        else
+            echo
+            echo "Wrong cluster operation: ${OPERATION}. Should be one of:"
+            echo "${FORMATTED_KIND_OPERATIONS}"
+            echo
+            exit 1
+        fi
+    else
+        if [[ ${OPERATION} == "start" ]]; then
+            echo
+            echo "Creating cluster"
+            echo
+            create_cluster
+        elif [[ ${OPERATION} == "recreate" ]]; then
+            echo
+            echo "Cluster ${KIND_CLUSTER_NAME} does not exist. Creating rather than recreating"
+            echo "Creating cluster"
+            echo
+            create_cluster
+        elif [[ ${OPERATION} == "stop" || ${OEPRATON} == "deploy" || ${OPERATION} == "test" ]]; then
+            echo
+            echo "Cluster ${KIND_CLUSTER_NAME} does not exist. It should exist for ${OPERATION} operation"
+            echo
+            exit 1
+        else
+            echo
+            echo "Wrong cluster operation: ${OPERATION}. Should be one of:"
+            echo "${FORMATTED_KIND_OPERATIONS}"
+            echo
+            exit 1
+        fi
+    fi
+}
+
+function check_cluster_ready_for_airflow() {
+    kubectl cluster-info --cluster "${KUBECTL_CLUSTER_NAME}"
+    kubectl get nodes --cluster "${KUBECTL_CLUSTER_NAME}"
+    echo
+    echo "Showing storageClass"
+    echo
+    kubectl get storageclass --cluster "${KUBECTL_CLUSTER_NAME}"
+    echo
+    echo "Showing kube-system pods"
+    echo
+    kubectl get -n kube-system pods --cluster "${KUBECTL_CLUSTER_NAME}"
+    echo
+    echo "Airflow environment on kubernetes is good to go!"
+    echo
+    kubectl create namespace test-namespace --cluster "${KUBECTL_CLUSTER_NAME}"
+}
+
+
+function build_kubernetes_image() {
+    cd "${AIRFLOW_SOURCES}" || exit 1
+    prepare_prod_build
+    if [[ $(docker images -q "${AIRFLOW_PROD_IMAGE}") == "" ||
+            ${FORCE_BUILD_IMAGES:="false"} == "true" ]]; then
+        build_prod_image
+    else
+        echo
+        echo "Skip rebuilding prod image. Use --force-build-images to rebuild prod image."
+        echo
+    fi
+    echo
+    echo "Adding kubernetes-specific scripts to prod image."
+    echo "Building ${AIRFLOW_KUBERNETES_IMAGE} from ${AIRFLOW_PROD_IMAGE} with latest sources."
+    echo
+    docker build \
+        --build-arg AIRFLOW_PROD_IMAGE="${AIRFLOW_PROD_IMAGE}" \
+        --cache-from "${AIRFLOW_PROD_IMAGE}" \
+        --tag="${AIRFLOW_KUBERNETES_IMAGE}" \
+        -f- . << 'EOF'
+    ARG AIRFLOW_PROD_IMAGE
+    FROM ${AIRFLOW_PROD_IMAGE}
+
+    ARG AIRFLOW_SOURCES=/home/airflow/airflow_sources/
+    ENV AIRFLOW_SOURCES=${AIRFLOW_SOURCES}
+
+    USER root
+
+    COPY --chown=airflow:airflow . ${AIRFLOW_SOURCES}
+
+    COPY scripts/ci/kubernetes/docker/airflow-test-env-init-db.sh /tmp/airflow-test-env-init-db.sh
+    COPY scripts/ci/kubernetes/docker/airflow-test-env-init-dags.sh /tmp/airflow-test-env-init-dags.sh
+    COPY scripts/ci/kubernetes/docker/bootstrap.sh /bootstrap.sh
+
+    RUN chmod +x /bootstrap.sh
+
+
+    USER airflow
+
+
+    ENTRYPOINT ["/bootstrap.sh"]
+EOF
+
+    echo "The ${AIRFLOW_KUBERNETES_IMAGE} is prepared for deployment."
+}
+
+function load_image_to_kind_cluster() {
+    echo
+    echo "Loading ${AIRFLOW_KUBERNETES_IMAGE} to ${KIND_CLUSTER_NAME}"
+    echo
+    kind load docker-image --name "${KIND_CLUSTER_NAME}" "${AIRFLOW_KUBERNETES_IMAGE}"
+}
+
+function prepare_kubernetes_app_variables() {
+    echo
+    echo "Preparing kubernetes variables"
+    echo
+    KUBERNETES_APP_DIR="${AIRFLOW_SOURCES}/scripts/ci/kubernetes/app"
+    TEMPLATE_DIRNAME="${KUBERNETES_APP_DIR}/templates"
+    BUILD_DIRNAME="${KUBERNETES_APP_DIR}/build"
+
+    # shellcheck source=common/_image_variables.sh
+    . "${AIRFLOW_SOURCES}/common/_image_variables.sh"
+
+    # Source branch will be set in DockerHub
+    SOURCE_BRANCH=${SOURCE_BRANCH:=${DEFAULT_BRANCH}}
+    BRANCH_NAME=${BRANCH_NAME:=${SOURCE_BRANCH}}
+
+    if [[ ! -d "${BUILD_DIRNAME}" ]]; then
+        mkdir -p "${BUILD_DIRNAME}"
+    fi
+
+    rm -f "${BUILD_DIRNAME}"/*
+    rm -f "${BUILD_DIRNAME}"/*
+
+    if [[ "${KUBERNETES_MODE}" == "image" ]]; then
+        INIT_DAGS_VOLUME_NAME=airflow-dags
+        POD_AIRFLOW_DAGS_VOLUME_NAME=airflow-dags
+        CONFIGMAP_DAGS_FOLDER=/opt/airflow/dags
+        CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT=
+        CONFIGMAP_DAGS_VOLUME_CLAIM=airflow-dags
+    else
+        INIT_DAGS_VOLUME_NAME=airflow-dags-fake
+        POD_AIRFLOW_DAGS_VOLUME_NAME=airflow-dags-git
+        CONFIGMAP_DAGS_FOLDER=/opt/airflow/dags/repo/airflow/example_dags
+        CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT=/opt/airflow/dags
+        CONFIGMAP_DAGS_VOLUME_CLAIM=
+    fi
+
+
+    CONFIGMAP_GIT_REPO=${CI_SOURCE_REPO}
+    CONFIGMAP_BRANCH=${CI_SOURCE_BRANCH}
+}
+
+function prepare_kubernetes_resources() {
+    echo
+    echo "Preparing kubernetes resources"
+    echo
+    if [[ "${KUBERNETES_MODE}" == "image" ]]; then
+        sed -e "s/{{INIT_GIT_SYNC}}//g" \
+            "${TEMPLATE_DIRNAME}/airflow.template.yaml" >"${BUILD_DIRNAME}/airflow.yaml"
+    else
+        sed -e "/{{INIT_GIT_SYNC}}/{r ${TEMPLATE_DIRNAME}/init_git_sync.template.yaml" -e 'd}' \
+            "${TEMPLATE_DIRNAME}/airflow.template.yaml" >"${BUILD_DIRNAME}/airflow.yaml"
+    fi
+    sed -i "s|{{AIRFLOW_KUBERNETES_IMAGE}}|${AIRFLOW_KUBERNETES_IMAGE}|g" "${BUILD_DIRNAME}/airflow.yaml"
+
+    sed -i "s|{{CONFIGMAP_GIT_REPO}}|${CONFIGMAP_GIT_REPO}|g" "${BUILD_DIRNAME}/airflow.yaml"
+    sed -i "s|{{CONFIGMAP_BRANCH}}|${CONFIGMAP_BRANCH}|g" "${BUILD_DIRNAME}/airflow.yaml"
+    sed -i "s|{{INIT_DAGS_VOLUME_NAME}}|${INIT_DAGS_VOLUME_NAME}|g" "${BUILD_DIRNAME}/airflow.yaml"
+    sed -i "s|{{POD_AIRFLOW_DAGS_VOLUME_NAME}}|${POD_AIRFLOW_DAGS_VOLUME_NAME}|g" \
+        "${BUILD_DIRNAME}/airflow.yaml"
+
+    sed "s|{{CONFIGMAP_DAGS_FOLDER}}|${CONFIGMAP_DAGS_FOLDER}|g" \
+        "${TEMPLATE_DIRNAME}/configmaps.template.yaml" >"${BUILD_DIRNAME}/configmaps.yaml"
+    sed -i "s|{{CONFIGMAP_GIT_REPO}}|${CONFIGMAP_GIT_REPO}|g" "${BUILD_DIRNAME}/configmaps.yaml"
+    sed -i "s|{{CONFIGMAP_BRANCH}}|${CONFIGMAP_BRANCH}|g" "${BUILD_DIRNAME}/configmaps.yaml"
+    sed -i "s|{{CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT}}|${CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT}|g" \
+        "${BUILD_DIRNAME}/configmaps.yaml"
+    sed -i "s|{{CONFIGMAP_DAGS_VOLUME_CLAIM}}|${CONFIGMAP_DAGS_VOLUME_CLAIM}|g" \
+        "${BUILD_DIRNAME}/configmaps.yaml"
+    sed -i "s|{{AIRFLOW_KUBERNETES_IMAGE_NAME}}|${AIRFLOW_KUBERNETES_IMAGE_NAME}|g" \
+        "${BUILD_DIRNAME}/configmaps.yaml"
+    sed -i "s|{{AIRFLOW_KUBERNETES_IMAGE_TAG}}|${AIRFLOW_KUBERNETES_IMAGE_TAG}|g" \
+        "${BUILD_DIRNAME}/configmaps.yaml"
+}
+
+function apply_kubernetes_resources() {
+    echo
+    echo "Apply kubernetes resources."
+    echo
+
+
+    kubectl delete -f "${KUBERNETES_APP_DIR}/postgres.yaml" --cluster "${KUBECTL_CLUSTER_NAME}" \
+        2>&1 | grep -v "NotFound" || true
+    kubectl delete -f "${BUILD_DIRNAME}/airflow.yaml" --cluster "${KUBECTL_CLUSTER_NAME}" \
+        2>&1 | grep -v "NotFound" || true
+    kubectl delete -f "${KUBERNETES_APP_DIR}/secrets.yaml" --cluster "${KUBECTL_CLUSTER_NAME}" \
+        2>&1 | grep -v "NotFound" || true
+
+    set -e
+
+    kubectl apply -f "${KUBERNETES_APP_DIR}/secrets.yaml" --cluster "${KUBECTL_CLUSTER_NAME}"
+    kubectl apply -f "${BUILD_DIRNAME}/configmaps.yaml" --cluster "${KUBECTL_CLUSTER_NAME}"
+    kubectl apply -f "${KUBERNETES_APP_DIR}/postgres.yaml" --cluster "${KUBECTL_CLUSTER_NAME}"
+    kubectl apply -f "${KUBERNETES_APP_DIR}/volumes.yaml" --cluster "${KUBECTL_CLUSTER_NAME}"
+    kubectl apply -f "${BUILD_DIRNAME}/airflow.yaml" --cluster "${KUBECTL_CLUSTER_NAME}"
+}
+
+
+function dump_kubernetes_logs() {
+    POD=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' \
+        --cluster "${KUBECTL_CLUSTER_NAME}" | grep airflow | head -1)
+    echo "------- pod description -------"
+    kubectl describe pod "${POD}" --cluster "${KUBECTL_CLUSTER_NAME}"
+    echo "------- webserver init container logs - init -------"
+    kubectl logs "${POD}" -c init --cluster "${KUBECTL_CLUSTER_NAME}" || true
+    if [[ "${KUBERNETES_MODE}" == "git" ]]; then
+        echo "------- webserver init container logs - git-sync-clone -------"
+        kubectl logs "${POD}" -c git-sync-clone --cluster "${KUBECTL_CLUSTER_NAME}" || true
+    fi
+    echo "------- webserver logs -------"
+    kubectl logs "${POD}" -c webserver --cluster "${KUBECTL_CLUSTER_NAME}" || true
+    echo "------- scheduler logs -------"
+    kubectl logs "${POD}" -c scheduler --cluster "${KUBECTL_CLUSTER_NAME}" || true
+    echo "--------------"
+}
+
+function wait_for_airflow_pods_up_and_running() {
+    set +o pipefail
+    # wait for up to 10 minutes for everything to be deployed
+    PODS_ARE_READY="0"
+    for i in {1..150}; do
+        echo "------- Running kubectl get pods: $i -------"
+        PODS=$(kubectl get pods --cluster "${KUBECTL_CLUSTER_NAME}" | awk 'NR>1 {print $0}')
+        echo "$PODS"
+        NUM_AIRFLOW_READY=$(echo "${PODS}" | grep airflow | awk '{print $2}' | grep -cE '([0-9])\/(\1)' \
+            | xargs)
+        NUM_POSTGRES_READY=$(echo "${PODS}" | grep postgres | awk '{print $2}' | grep -cE '([0-9])\/(\1)' \
+            | xargs)
+        if [[ "${NUM_AIRFLOW_READY}" == "1" && "${NUM_POSTGRES_READY}" == "1" ]]; then
+            PODS_ARE_READY="1"
+            break
+        fi
+        sleep 4
+    done
+    POD=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' \
+        --cluster "${KUBECTL_CLUSTER_NAME}" | grep airflow | head -1)
+
+    if [[ "${PODS_ARE_READY}" == "1" ]]; then
+        echo "PODS are ready."
+        set -o pipefail
+    else
+        echo >&2 "PODS are not ready after waiting for a long time. Exiting..."
+        dump_kubernetes_logs
+        exit 1
+    fi
+}
+
+
+function wait_for_airflow_webserver_up_and_running() {
+    set +o pipefail
+    # Wait until Airflow webserver is up
+    KUBERNETES_HOST=localhost
+    AIRFLOW_WEBSERVER_IS_READY="0"
+    CONSECUTIVE_SUCCESS_CALLS=0
+    for i in {1..30}; do
+        echo "------- Wait until webserver is up: $i -------"
+        PODS=$(kubectl get pods --cluster "${KUBECTL_CLUSTER_NAME}" | awk 'NR>1 {print $0}')
+        echo "$PODS"
+        HTTP_CODE=$(curl -LI "http://${KUBERNETES_HOST}:30809/health" -o /dev/null -w '%{http_code}\n' -sS) \
+            || true
+        if [[ "${HTTP_CODE}" == 200 ]]; then
+            ((CONSECUTIVE_SUCCESS_CALLS += 1))
+        else
+            CONSECUTIVE_SUCCESS_CALLS="0"
+        fi
+        if [[ "${CONSECUTIVE_SUCCESS_CALLS}" == 3 ]]; then
+            AIRFLOW_WEBSERVER_IS_READY="1"
+            break
+        fi
+        sleep 10
+    done
+    set -o pipefail
+    if [[ "${AIRFLOW_WEBSERVER_IS_READY}" == "1" ]]; then
+        echo
+        echo "Airflow webserver is ready."
+        echo
+    else
+        echo >&2
+        echo >&2 "Airflow webserver is not ready after waiting for a long time. Exiting..."
+        echo >&2
+        dump_kubernetes_logs
+        exit 1
+    fi
+}
diff --git a/scripts/ci/in_container/kubernetes/docker/bootstrap.sh b/scripts/ci/ci_deploy_app_to_kubernetes.sh
similarity index 53%
copy from scripts/ci/in_container/kubernetes/docker/bootstrap.sh
copy to scripts/ci/ci_deploy_app_to_kubernetes.sh
index 0a9f34d..6061c1c 100755
--- a/scripts/ci/in_container/kubernetes/docker/bootstrap.sh
+++ b/scripts/ci/ci_deploy_app_to_kubernetes.sh
@@ -15,13 +15,22 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+# shellcheck source=scripts/ci/_script_init.sh
+. "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
-if [[ "$1" = "webserver" ]]
-then
-    exec airflow webserver
-fi
+set -euo pipefail
 
-if [[ "$1" = "scheduler" ]]
-then
-    exec airflow scheduler
-fi
+export KUBERNETES_VERSION=${KUBERNETES_VERSION:="v1.15.3"}
+export PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:="3.6"}
+export KIND_CLUSTER_NAME=${KIND_CLUSTER_NAME:="airflow-python-${PYTHON_MAJOR_MINOR_VERSION}-${KUBERNETES_VERSION}"}
+export KUBERNETES_MODE=${KUBERNETES_MODE:="image"}
+
+get_ci_environment
+check_kind_and_kubectl_are_installed
+build_kubernetes_image
+load_image_to_kind_cluster
+prepare_kubernetes_app_variables
+prepare_kubernetes_resources
+apply_kubernetes_resources
+wait_for_airflow_pods_up_and_running
+wait_for_airflow_webserver_up_and_running
diff --git a/scripts/ci/in_container/run_ci_tests.sh b/scripts/ci/ci_load_image_to_kind.sh
similarity index 53%
copy from scripts/ci/in_container/run_ci_tests.sh
copy to scripts/ci/ci_load_image_to_kind.sh
index b046c08..2fc2570 100755
--- a/scripts/ci/in_container/run_ci_tests.sh
+++ b/scripts/ci/ci_load_image_to_kind.sh
@@ -15,34 +15,20 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
-# shellcheck source=scripts/ci/in_container/_in_container_script_init.sh
-. "$( dirname "${BASH_SOURCE[0]}" )/_in_container_script_init.sh"
+# shellcheck source=scripts/ci/_script_init.sh
+. "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
-# any argument received is overriding the default nose execution arguments:
-PYTEST_ARGS=( "$@" )
+cd "${AIRFLOW_SOURCES}" || exit 1
 
+export KUBERNETES_VERSION=${KUBERNETES_VERSION:="v1.15.3"}
+export PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:="3.6"}
+export KIND_CLUSTER_NAME=${KIND_CLUSTER_NAME:="airflow-python-${PYTHON_MAJOR_MINOR_VERSION}-${KUBERNETES_VERSION}"}
+
+prepare_prod_build
 echo
-echo "Starting the tests with those pytest arguments: ${PYTEST_ARGS[*]}"
+echo "Loading the ${AIRFLOW_KUBERNETES_IMAGE} to cluster ${KIND_CLUSTER_NAME} from docker"
+echo
+"${AIRFLOW_SOURCES}/.build/bin/kind" load docker-image --name "${KIND_CLUSTER_NAME}" "${AIRFLOW_KUBERNETES_IMAGE}"
+echo
+echo "Loaded the ${AIRFLOW_KUBERNETES_IMAGE} to cluster ${KIND_CLUSTER_NAME}"
 echo
-set +e
-
-pytest "${PYTEST_ARGS[@]}"
-
-RES=$?
-
-set +x
-if [[ "${RES}" == "0" && ${CI:="false"} == "true" ]]; then
-    echo "All tests successful"
-    bash <(curl -s https://codecov.io/bash)
-fi
-
-if [[ ${CI} == "true" ]]; then
-    send_docker_logs_to_file_io
-    send_airflow_logs_to_file_io
-fi
-
-if [[ ${CI} == "true" && ${ENABLE_KIND_CLUSTER} == "true" ]]; then
-    send_kubernetes_logs_to_file_io
-fi
-
-exit "${RES}"
diff --git a/scripts/ci/in_container/kubernetes/docker/bootstrap.sh b/scripts/ci/ci_perform_kind_cluster_operation.sh
similarity index 79%
rename from scripts/ci/in_container/kubernetes/docker/bootstrap.sh
rename to scripts/ci/ci_perform_kind_cluster_operation.sh
index 0a9f34d..7963344 100755
--- a/scripts/ci/in_container/kubernetes/docker/bootstrap.sh
+++ b/scripts/ci/ci_perform_kind_cluster_operation.sh
@@ -16,12 +16,11 @@
 # specific language governing permissions and limitations
 # under the License.
 
-if [[ "$1" = "webserver" ]]
-then
-    exec airflow webserver
-fi
+# shellcheck source=scripts/ci/_script_init.sh
+. "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
 
-if [[ "$1" = "scheduler" ]]
-then
-    exec airflow scheduler
-fi
+check_kind_and_kubectl_are_installed
+
+perform_kind_cluster_operation "${@}"
+
+check_cluster_ready_for_airflow
diff --git a/scripts/ci/ci_run_airflow_testing.sh b/scripts/ci/ci_run_airflow_testing.sh
index e3b23ab..f8d5e7e 100755
--- a/scripts/ci/ci_run_airflow_testing.sh
+++ b/scripts/ci/ci_run_airflow_testing.sh
@@ -18,7 +18,7 @@
 export VERBOSE=${VERBOSE:="false"}
 
 function run_airflow_testing_in_docker_with_kubernetes() {
-    export KUBERNETES_MODE=${KUBERNETES_MODE:="git_mode"}
+    export KUBERNETES_MODE=${KUBERNETES_MODE:="image"}
     export KUBERNETES_VERSION=${KUBERNETES_VERSION:="v1.15.3"}
 
     # shellcheck disable=SC2016
@@ -151,3 +151,34 @@ do
     exit ${EXIT_CODE}
 done
 set -u
+
+#TODO FIXME
+if [[ "${INTEGRATIONS[*]}" == *"kubernetes"* ]]; then
+    export KUBERNETES_MODE=${KUBERNETES_MODE:="image"}
+    export KUBERNETES_VERSION=${KUBERNETES_VERSION:="v1.15.3"}
+    export KIND_CLUSTER_NAME="airflow-python-${PYTHON_MAJOR_MINOR_VERSION}-${KUBERNETES_VERSION}"
+
+    if [[ ${ENABLE_KIND_CLUSTER} == "true" ]]; then
+        install_kind_in_host
+        "${MY_DIR}/setup_kind_cluster_in_host.sh"
+    fi
+
+    # adding trap to exiting trap
+    HANDLERS="$( trap -p EXIT | cut -f2 -d \' )"
+    # shellcheck disable=SC2064
+    trap "${HANDLERS}${HANDLERS:+;}send_kubernetes_logs_to_file_io" EXIT
+fi
+
+set +u
+# shellcheck disable=SC2016
+docker-compose --log-level INFO \
+  -f "${MY_DIR}/docker-compose/base.yml" \
+  -f "${MY_DIR}/docker-compose/backend-${BACKEND}.yml" \
+  "${INTEGRATIONS[@]}" \
+  "${DOCKER_COMPOSE_LOCAL[@]}" \
+     run airflow \
+       '/opt/airflow/scripts/ci/in_container/entrypoint_ci.sh "${@}"' \
+       /opt/airflow/scripts/ci/in_container/entrypoint_ci.sh "${@}"
+     # Note the command is there twice (!) because it is passed via bash -c
+     # and bash -c starts passing parameters from $0. TODO: fixme
+set -u
diff --git a/scripts/ci/ci_run_kubernetes_tests.sh b/scripts/ci/ci_run_kubernetes_tests.sh
new file mode 100755
index 0000000..c530112
--- /dev/null
+++ b/scripts/ci/ci_run_kubernetes_tests.sh
@@ -0,0 +1,105 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# shellcheck source=scripts/ci/_script_init.sh
+. "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
+
+INTERACTIVE="false"
+
+declare -a TESTS
+declare -a PYTEST_ARGS
+
+TESTS=()
+
+if [[ $# != 0 ]]; then
+    if [[ $1 == "--help" || $1 == "-h" ]]; then
+        echo
+        echo "Running kubernetes tests"
+        echo
+        echo "    $0                      - runs all kubernetes tests"
+        echo "    $0 TEST [TEST ...]      - runs selected kubernetes tests (from kubernetes_tests folder)"
+        echo "    $0 [-i|--interactive]   - Activates virtual environment ready to run tests and drops you in"
+        echo "    $0 [--help]             - Prints this help message"
+        echo
+        exit
+    elif [[ $1 == "--interactive" || $1 == "-i" ]]; then
+        echo
+        echo "Entering interactive environment for kubernetes testing"
+        echo
+        INTERACTIVE="true"
+    else
+        TESTS=("${@}")
+    fi
+    PYTEST_ARGS=(
+        "--pythonwarnings=ignore::DeprecationWarning"
+        "--pythonwarnings=ignore::PendingDeprecationWarning"
+    )
+else
+    TESTS=("kubernetes_tests")
+    PYTEST_ARGS=(
+        "--verbosity=1"
+        "--strict-markers"
+        "--durations=100"
+        "--cov=airflow/"
+        "--cov-config=.coveragerc"
+        "--cov-report=html:airflow/www/static/coverage/"
+        "--color=yes"
+        "--maxfail=50"
+        "--pythonwarnings=ignore::DeprecationWarning"
+        "--pythonwarnings=ignore::PendingDeprecationWarning"
+        )
+
+fi
+
+get_ci_environment
+
+cd "${AIRFLOW_SOURCES}" || exit 1
+
+VIRTUALENV_PATH="${BUILD_CACHE_DIR}/.kubernetes_venv"
+
+if [[ ! -d ${VIRTUALENV_PATH} ]]; then
+    echo
+    echo "Creating virtualenv at ${VIRTUALENV_PATH}"
+    echo
+    python -m venv "${VIRTUALENV_PATH}"
+fi
+
+. "${VIRTUALENV_PATH}/bin/activate"
+
+pip install pytest freezegun pytest-cov \
+    --constraint "requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"
+
+pip install -e ".[kubernetes]" \
+    --constraint "requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"
+
+if [[ ${INTERACTIVE} == "true" ]]; then
+    echo
+    echo "Activating the virtual environment for kubernetes testing"
+    echo
+    echo "You can run kubernetes testing via 'pytest kubernetes_tests/....'"
+    echo "You can add -s to see the output of your tests on screen"
+    echo
+    echo "The webserver is available at http://localhost:30809/"
+    echo
+    echo "User/password: airflow/airflow"
+    echo
+    echo "You are entering the virtualenv now. Type exit to exit back to the original shell"
+    echo
+    exec "${SHELL}"
+else
+    pytest "${PYTEST_ARGS[@]}" "${TESTS[@]}"
+fi
diff --git a/scripts/ci/docker-compose/base.yml b/scripts/ci/docker-compose/base.yml
index 2505784..34e31ff 100644
--- a/scripts/ci/docker-compose/base.yml
+++ b/scripts/ci/docker-compose/base.yml
@@ -41,7 +41,6 @@ services:
       - VERBOSE
       - VEROSE_COMMANDS
       - AIRFLOW_CI_IMAGE
-      - RUNTIME
       - ENABLE_KIND_CLUSTER
       - ENABLED_INTEGRATIONS
       - RUN_INTEGRATION_TESTS
diff --git a/scripts/ci/docker-compose/local.yml b/scripts/ci/docker-compose/local.yml
index d79b1aa..2991de1 100644
--- a/scripts/ci/docker-compose/local.yml
+++ b/scripts/ci/docker-compose/local.yml
@@ -55,6 +55,7 @@ services:
       - ../../../setup.cfg:/opt/airflow/setup.cfg:cached
       - ../../../setup.py:/opt/airflow/setup.py:cached
       - ../../../tests:/opt/airflow/tests:cached
+      - ../../../kubernetes_tests:/opt/airflow/kubernetes_tests:cached
       - ../../../tmp:/opt/airflow/tmp:cached
       # END automatically generated volumes from LOCAL_MOUNTS in _utils.sh
     environment:
diff --git a/scripts/ci/in_container/_in_container_utils.sh b/scripts/ci/in_container/_in_container_utils.sh
index 9b9814a..67cecfd 100644
--- a/scripts/ci/in_container/_in_container_utils.sh
+++ b/scripts/ci/in_container/_in_container_utils.sh
@@ -232,34 +232,6 @@ function send_airflow_logs_to_file_io() {
     curl -F "file=@${DUMP_FILE}" https://file.io
 }
 
-
-function dump_kind_logs() {
-    echo "###########################################################################################"
-    echo "                   Dumping logs from KIND"
-    echo "###########################################################################################"
-
-    FILE_NAME="${1}"
-    kind --name "${CLUSTER_NAME}" export logs "${FILE_NAME}"
-}
-
-
-function send_kubernetes_logs_to_file_io() {
-    echo "##############################################################################"
-    echo
-    echo "   DUMPING LOG FILES FROM KIND AND SENDING THEM TO file.io"
-    echo
-    echo "##############################################################################"
-    DUMP_DIR_NAME=$(date "+%Y-%m-%d")_kind_${CI_BUILD_ID:="default"}_${CI_JOB_ID:="default"}
-    DUMP_DIR=/tmp/${DUMP_DIR_NAME}
-    dump_kind_logs "${DUMP_DIR}"
-    tar -cvzf "${DUMP_DIR}.tar.gz" -C /tmp "${DUMP_DIR_NAME}"
-    echo
-    echo "   Logs saved to ${DUMP_DIR}.tar.gz"
-    echo
-    echo "##############################################################################"
-    curl -F "file=@${DUMP_DIR}.tar.gz" https://file.io
-}
-
 function install_released_airflow_version() {
     pip uninstall -y apache-airflow || true
     find /root/airflow/ -type f -print0 | xargs -0 rm -f --
diff --git a/scripts/ci/in_container/entrypoint_ci.sh b/scripts/ci/in_container/entrypoint_ci.sh
index 7a2ab15..111e7ca 100755
--- a/scripts/ci/in_container/entrypoint_ci.sh
+++ b/scripts/ci/in_container/entrypoint_ci.sh
@@ -29,7 +29,6 @@ BACKEND=${BACKEND:=sqlite}
 KUBERNETES_MODE=${KUBERNETES_MODE:=""}
 KUBERNETES_VERSION=${KUBERNETES_VERSION:=""}
 ENABLE_KIND_CLUSTER=${ENABLE_KIND_CLUSTER:="false"}
-RUNTIME=${RUNTIME:=""}
 
 export AIRFLOW_HOME=${AIRFLOW_HOME:=${HOME}}
 
@@ -143,54 +142,31 @@ if [[ ${INTEGRATION_KERBEROS:="false"} == "true" ]]; then
 fi
 
 
-if [[ "${RUNTIME}" == "" ]]; then
-    # Start MiniCluster
-    java -cp "/opt/minicluster-1.1-SNAPSHOT/*" com.ing.minicluster.MiniCluster \
-        >"${AIRFLOW_HOME}/logs/minicluster.log" 2>&1 &
+# Start MiniCluster
+java -cp "/opt/minicluster-1.1-SNAPSHOT/*" com.ing.minicluster.MiniCluster \
+    >"${AIRFLOW_HOME}/logs/minicluster.log" 2>&1 &
 
-    # Set up ssh keys
-    echo 'yes' | ssh-keygen -t rsa -C your_email@youremail.com -m PEM -P '' -f ~/.ssh/id_rsa \
-        >"${AIRFLOW_HOME}/logs/ssh-keygen.log" 2>&1
+# Set up ssh keys
+echo 'yes' | ssh-keygen -t rsa -C your_email@youremail.com -m PEM -P '' -f ~/.ssh/id_rsa \
+    >"${AIRFLOW_HOME}/logs/ssh-keygen.log" 2>&1
 
-    cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
-    ln -s -f ~/.ssh/authorized_keys ~/.ssh/authorized_keys2
-    chmod 600 ~/.ssh/*
+cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
+ln -s -f ~/.ssh/authorized_keys ~/.ssh/authorized_keys2
+chmod 600 ~/.ssh/*
 
-    # SSH Service
-    sudo service ssh restart >/dev/null 2>&1
+# SSH Service
+sudo service ssh restart >/dev/null 2>&1
 
-    # Sometimes the server is not quick enough to load the keys!
-    while [[ $(ssh-keyscan -H localhost 2>/dev/null | wc -l) != "3" ]] ; do
-        echo "Not all keys yet loaded by the server"
-        sleep 0.05
-    done
-
-    ssh-keyscan -H localhost >> ~/.ssh/known_hosts 2>/dev/null
-fi
+# Sometimes the server is not quick enough to load the keys!
+while [[ $(ssh-keyscan -H localhost 2>/dev/null | wc -l) != "3" ]] ; do
+    echo "Not all keys yet loaded by the server"
+    sleep 0.05
+done
 
+ssh-keyscan -H localhost >> ~/.ssh/known_hosts 2>/dev/null
 
-export KIND_CLUSTER_OPERATION="${KIND_CLUSTER_OPERATION:="start"}"
 export KUBERNETES_VERSION=${KUBERNETES_VERSION:=""}
 
-if [[ ${RUNTIME:=""} == "kubernetes" ]]; then
-    unset KRB5_CONFIG
-    unset KRB5_KTNAME
-    export AIRFLOW_KUBERNETES_IMAGE=${AIRFLOW_CI_IMAGE}-kubernetes
-    AIRFLOW_KUBERNETES_IMAGE_NAME=$(echo "${AIRFLOW_KUBERNETES_IMAGE}" | cut -f 1 -d ":")
-    export AIRFLOW_KUBERNETES_IMAGE_NAME
-    AIRFLOW_KUBERNETES_IMAGE_TAG=$(echo "${AIRFLOW_KUBERNETES_IMAGE}" | cut -f 2 -d ":")
-    export AIRFLOW_KUBERNETES_IMAGE_TAG
-fi
-
-
-if [[ "${ENABLE_KIND_CLUSTER}" == "true" ]]; then
-    export CLUSTER_NAME="airflow-python-${PYTHON_MAJOR_MINOR_VERSION}-${KUBERNETES_VERSION}"
-    "${MY_DIR}/kubernetes/setup_kind_cluster.sh"
-    if [[ ${KIND_CLUSTER_OPERATION} == "stop" ]]; then
-        exit 1
-    fi
-fi
-
 # shellcheck source=scripts/ci/in_container/configure_environment.sh
 . "${MY_DIR}/configure_environment.sh"
 
@@ -240,9 +216,9 @@ if [[ ${#@} -gt 0 && -n "$1" ]]; then
 fi
 
 if [[ -n ${RUN_INTEGRATION_TESTS:=""} ]]; then
-    for INTEGRATION in ${RUN_INTEGRATION_TESTS}
+    for INT in ${RUN_INTEGRATION_TESTS}
     do
-        CI_ARGS+=("--integration" "${INTEGRATION}")
+        CI_ARGS+=("--integration" "${INT}")
     done
     CI_ARGS+=("-rpfExX")
 elif [[ ${ONLY_RUN_LONG_RUNNING_TESTS:=""} == "true" ]]; then
@@ -261,17 +237,6 @@ elif [[ ${ONLY_RUN_QUARANTINED_TESTS:=""} == "true" ]]; then
         "--timeout" "90")
 fi
 
-
-if [[ -n ${RUNTIME} ]]; then
-    CI_ARGS+=("--runtime" "${RUNTIME}" "-rpfExX")
-    TESTS_TO_RUN="tests/runtime"
-    if [[ ${RUNTIME} == "kubernetes" ]]; then
-        export SKIP_INIT_DB=true
-        "${MY_DIR}/deploy_airflow_to_kubernetes.sh"
-    fi
-fi
-
-
 ARGS=("${CI_ARGS[@]}" "${TESTS_TO_RUN}")
 
 if [[ ${RUN_SYSTEM_TESTS:="false"} == "true" ]]; then
diff --git a/scripts/ci/in_container/kubernetes/app/deploy_app.sh b/scripts/ci/in_container/kubernetes/app/deploy_app.sh
deleted file mode 100755
index 1f7832d..0000000
--- a/scripts/ci/in_container/kubernetes/app/deploy_app.sh
+++ /dev/null
@@ -1,210 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#   http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied.  See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-set -euo pipefail
-
-MY_DIR=$(cd "$(dirname "$0")" && pwd)
-
-AIRFLOW_SOURCES=$(
-    cd "${MY_DIR}/../../../../../" || exit 1
-    pwd
-)
-export AIRFLOW_SOURCES
-
-# We keep _utils here because we are not in the in_container directory
-# shellcheck source=scripts/ci/in_container/_in_container_utils.sh
-. "${MY_DIR}/../../_in_container_utils.sh"
-
-assert_in_container
-
-in_container_script_start
-
-function end_and_dump_logs() {
-    dump_logs
-    in_container_script_end
-}
-
-trap in_container_script_end EXIT
-
-export TEMPLATE_DIRNAME="${MY_DIR}/templates"
-export BUILD_DIRNAME="${MY_DIR}/build"
-
-# shellcheck source=common/_image_variables.sh
-. "${AIRFLOW_SOURCES}/common/_image_variables.sh"
-
-if [[ ! -d "${BUILD_DIRNAME}" ]]; then
-    mkdir -p "${BUILD_DIRNAME}"
-fi
-
-rm -f "${BUILD_DIRNAME}"/*
-
-if [[ "${KUBERNETES_MODE}" == "persistent_mode" ]]; then
-    INIT_DAGS_VOLUME_NAME=airflow-dags
-    POD_AIRFLOW_DAGS_VOLUME_NAME=airflow-dags
-    CONFIGMAP_DAGS_FOLDER=/opt/airflow/dags
-    CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT=
-    CONFIGMAP_DAGS_VOLUME_CLAIM=airflow-dags
-else
-    INIT_DAGS_VOLUME_NAME=airflow-dags-fake
-    POD_AIRFLOW_DAGS_VOLUME_NAME=airflow-dags-git
-    CONFIGMAP_DAGS_FOLDER=/opt/airflow/dags/repo/airflow/example_dags
-    CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT=/opt/airflow/dags
-    CONFIGMAP_DAGS_VOLUME_CLAIM=
-fi
-
-CONFIGMAP_GIT_REPO=${GITHUB_REPOSITORY:-apache/airflow}
-CONFIGMAP_BRANCH=${GITHUB_BASE_REF:=master}
-
-if [[ "${KUBERNETES_MODE}" == "persistent_mode" ]]; then
-    sed -e "s/{{INIT_GIT_SYNC}}//g" \
-        "${TEMPLATE_DIRNAME}/airflow.template.yaml" >"${BUILD_DIRNAME}/airflow.yaml"
-else
-    sed -e "/{{INIT_GIT_SYNC}}/{r ${TEMPLATE_DIRNAME}/init_git_sync.template.yaml" -e 'd}' \
-        "${TEMPLATE_DIRNAME}/airflow.template.yaml" >"${BUILD_DIRNAME}/airflow.yaml"
-fi
-sed -i "s|{{AIRFLOW_KUBERNETES_IMAGE}}|${AIRFLOW_KUBERNETES_IMAGE}|g" "${BUILD_DIRNAME}/airflow.yaml"
-
-sed -i "s|{{CONFIGMAP_GIT_REPO}}|${CONFIGMAP_GIT_REPO}|g" "${BUILD_DIRNAME}/airflow.yaml"
-sed -i "s|{{CONFIGMAP_BRANCH}}|${CONFIGMAP_BRANCH}|g" "${BUILD_DIRNAME}/airflow.yaml"
-sed -i "s|{{INIT_DAGS_VOLUME_NAME}}|${INIT_DAGS_VOLUME_NAME}|g" "${BUILD_DIRNAME}/airflow.yaml"
-sed -i "s|{{POD_AIRFLOW_DAGS_VOLUME_NAME}}|${POD_AIRFLOW_DAGS_VOLUME_NAME}|g" \
-    "${BUILD_DIRNAME}/airflow.yaml"
-
-sed "s|{{CONFIGMAP_DAGS_FOLDER}}|${CONFIGMAP_DAGS_FOLDER}|g" \
-    "${TEMPLATE_DIRNAME}/configmaps.template.yaml" >"${BUILD_DIRNAME}/configmaps.yaml"
-sed -i "s|{{CONFIGMAP_GIT_REPO}}|${CONFIGMAP_GIT_REPO}|g" "${BUILD_DIRNAME}/configmaps.yaml"
-sed -i "s|{{CONFIGMAP_BRANCH}}|${CONFIGMAP_BRANCH}|g" "${BUILD_DIRNAME}/configmaps.yaml"
-sed -i "s|{{CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT}}|${CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT}|g" \
-    "${BUILD_DIRNAME}/configmaps.yaml"
-sed -i "s|{{CONFIGMAP_DAGS_VOLUME_CLAIM}}|${CONFIGMAP_DAGS_VOLUME_CLAIM}|g" \
-    "${BUILD_DIRNAME}/configmaps.yaml"
-sed -i "s|{{AIRFLOW_KUBERNETES_IMAGE_NAME}}|${AIRFLOW_KUBERNETES_IMAGE_NAME}|g" \
-    "${BUILD_DIRNAME}/configmaps.yaml"
-sed -i "s|{{AIRFLOW_KUBERNETES_IMAGE_TAG}}|${AIRFLOW_KUBERNETES_IMAGE_TAG}|g" \
-    "${BUILD_DIRNAME}/configmaps.yaml"
-
-cat "${BUILD_DIRNAME}/airflow.yaml"
-cat "${BUILD_DIRNAME}/configmaps.yaml"
-
-kubectl delete -f "${MY_DIR}/postgres.yaml" || true
-kubectl delete -f "${BUILD_DIRNAME}/airflow.yaml" || true
-kubectl delete -f "${MY_DIR}/secrets.yaml" || true
-kubectl apply -f "${MY_DIR}/secrets.yaml" -n test-namespace
-
-set -e
-
-kubectl apply -f "${MY_DIR}/secrets.yaml"
-kubectl apply -f "${MY_DIR}/secrets.yaml" -n test-namespace
-kubectl apply -f "${BUILD_DIRNAME}/configmaps.yaml"
-kubectl apply -f "${BUILD_DIRNAME}/configmaps.yaml" -n test-namespace
-kubectl apply -f "${MY_DIR}/postgres.yaml"
-kubectl apply -f "${MY_DIR}/volumes.yaml"
-kubectl apply -f "${MY_DIR}/volumes.yaml" -n test-namespace
-
-set +x
-set +o pipefail
-PODS_ARE_READY="0"
-for i in {1..150}; do
-    echo "------- Running kubectl get pods: $i -------"
-    PODS=$(kubectl get pods | awk 'NR>1 {print $0}')
-    echo "$PODS"
-    NUM_POSTGRES_READY=$(echo "${PODS}" | grep postgres | awk '{print $2}' | grep -cE '([1-9])\/(\1)' | xargs)
-    if [[ "${NUM_POSTGRES_READY}" == "1" ]]; then
-        PODS_ARE_READY="1"
-        break
-    fi
-    sleep 4
-done
-
-sleep 7
-
-kubectl apply -f "${BUILD_DIRNAME}/airflow.yaml"
-
-dump_logs() {
-    echo "dumping logs"
-    POD=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | grep airflow | head -1)
-    echo "------- pod description -------"
-    kubectl describe pod "${POD}"
-    echo "------- webserver init container logs - init -------"
-    kubectl logs "${POD}" -c init || true
-    if [[ "${KUBERNETES_MODE}" != "persistent_mode" ]]; then
-        echo "------- webserver init container logs - git-sync-clone -------"
-        kubectl logs "${POD}" -c git-sync-clone || true
-    fi
-    echo "------- webserver logs -------"
-    kubectl logs "${POD}" -c webserver || true
-    echo "------- scheduler logs -------"
-    kubectl logs "${POD}" -c scheduler || true
-    echo "--------------"
-}
-
-set +x
-set +o pipefail
-# wait for up to 10 minutes for everything to be deployed
-PODS_ARE_READY="0"
-for i in {1..20}; do
-    echo "------- Running kubectl get pods: $i -------"
-    PODS=$(kubectl get pods | awk 'NR>1 {print $0}')
-    echo "$PODS"
-    NUM_AIRFLOW_READY=$(echo "${PODS}" | grep airflow | awk '{print $2}' | grep -cE '([2-9])\/(\1)' | xargs)
-    NUM_POSTGRES_READY=$(echo "${PODS}" | grep postgres | awk '{print $2}' | grep -cE '([1-9])\/(\1)' | xargs)
-    if [[ "${NUM_AIRFLOW_READY}" == "1" && "${NUM_POSTGRES_READY}" == "1" ]]; then
-        PODS_ARE_READY="1"
-        break
-    fi
-    sleep 4
-done
-POD=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | grep airflow | head -1)
-dump_logs
-if [[ "${PODS_ARE_READY}" == "1" ]]; then
-    echo "PODS are ready."
-else
-    echo >&2 "PODS are not ready after waiting for a long time. Exiting..."
-    dump_logs
-    exit 1
-fi
-
-# Wait until Airflow webserver is up
-KUBERNETES_HOST=${CLUSTER_NAME}-worker
-AIRFLOW_WEBSERVER_IS_READY="0"
-CONSECUTIVE_SUCCESS_CALLS=0
-for i in {1..20}; do
-    echo "------- Wait until webserver is up: $i -------"
-    PODS=$(kubectl get pods | awk 'NR>1 {print $0}')
-    echo "$PODS"
-    HTTP_CODE=$(curl -LI "http://${KUBERNETES_HOST}:30809/health" -o /dev/null -w '%{http_code}\n' -sS) || true
-    if [[ "${HTTP_CODE}" == 200 ]]; then
-        ((CONSECUTIVE_SUCCESS_CALLS += 1))
-    else
-        CONSECUTIVE_SUCCESS_CALLS="0"
-    fi
-    if [[ "${CONSECUTIVE_SUCCESS_CALLS}" == 3 ]]; then
-        AIRFLOW_WEBSERVER_IS_READY="1"
-        break
-    fi
-    sleep 10
-done
-set -o pipefail
-
-if [[ "${AIRFLOW_WEBSERVER_IS_READY}" == "1" ]]; then
-    echo "Airflow webserver is ready."
-else
-    echo >&2 "Airflow webserver is not ready after waiting for a long time. Exiting..."
-    dump_logs
-    exit 1
-fi
diff --git a/scripts/ci/in_container/kubernetes/docker/rebuild_airflow_image.sh b/scripts/ci/in_container/kubernetes/docker/rebuild_airflow_image.sh
deleted file mode 100755
index 1df822e..0000000
--- a/scripts/ci/in_container/kubernetes/docker/rebuild_airflow_image.sh
+++ /dev/null
@@ -1,74 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#   http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied.  See the License for the
-# specific language governing permissions and limitations
-# under the License.
-MY_DIR=$(cd "$(dirname "$0")" && pwd)
-AIRFLOW_SOURCES=$(cd "${MY_DIR}/../../../../../" || exit 1 ; pwd)
-export AIRFLOW_SOURCES
-
-# We keep _utils here because we are not in the in_container directory
-# shellcheck source=scripts/ci/in_container/_in_container_utils.sh
-. "${MY_DIR}/../../_in_container_utils.sh"
-
-export OUTPUT_LOG=${AIRFLOW_SOURCES}/logs/rebuild_airflow_image.log
-
-assert_in_container
-
-in_container_script_start
-
-cd "${AIRFLOW_SOURCES}" || exit 1
-
-# Required to rebuild images from inside container
-mkdir -pv scripts/docker/
-cp /entrypoint.sh scripts/docker/
-
-echo
-echo "Building image from ${AIRFLOW_CI_IMAGE} with latest sources"
-echo
-#export AIRFLOW_PROD_BASE_TAG="${DEFAULT_BRANCH}-python${PYTHON_MAJOR_MINOR_VERSION}"
-#export AIRFLOW_PROD_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_PROD_BASE_TAG}"
-export AIRFLOW_PROD_IMAGE="apache/airflow:v1-10-test-python3.6"
-echo "Adding kubernetes-specific scripts to basic CI image."
-echo "Building ${AIRFLOW_KUBERNETES_IMAGE} from ${AIRFLOW_PROD_IMAGE}"
-echo
-docker build \
-    --build-arg AIRFLOW_PROD_IMAGE="${AIRFLOW_PROD_IMAGE}" \
-    --cache-from "${AIRFLOW_PROD_IMAGE}" \
-    --tag="${AIRFLOW_KUBERNETES_IMAGE}" \
-    -f- .  <<EOF
-ARG AIRFLOW_PROD_IMAGE
-FROM ${AIRFLOW_PROD_IMAGE}
-
-COPY scripts/ci/in_container/kubernetes/docker/airflow-test-env-init.sh /tmp/airflow-test-env-init.sh
-
-ENV AIRFLOW__CORE__LOAD_EXAMPLES="true"
-ENTRYPOINT ["/usr/bin/dumb-init", "--", "/entrypoint"]
-EOF
-
-echo
-echo "Loading the ${AIRFLOW_KUBERNETES_IMAGE} to cluster ${CLUSTER_NAME} from docker"
-echo
-kind load docker-image --name "${CLUSTER_NAME}" "${AIRFLOW_KUBERNETES_IMAGE}"
-echo
-echo "Loaded the ${AIRFLOW_KUBERNETES_IMAGE} to cluster ${CLUSTER_NAME}"
-echo
-
-echo
-echo "Stopping output heartbeat"
-echo
-
-
-in_container_script_end
diff --git a/scripts/ci/in_container/kubernetes/setup_kind_cluster.sh b/scripts/ci/in_container/kubernetes/setup_kind_cluster.sh
deleted file mode 100755
index fc033da..0000000
--- a/scripts/ci/in_container/kubernetes/setup_kind_cluster.sh
+++ /dev/null
@@ -1,187 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#   http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied.  See the License for the
-# specific language governing permissions and limitations
-# under the License.
-MY_DIR=$(cd "$(dirname "$0")" && pwd)
-
-AIRFLOW_SOURCES=$(
-    cd "${MY_DIR}/../../../../" || exit 1
-    pwd
-)
-export AIRFLOW_SOURCES
-
-# We keep _utils here because we are not in the in_container directory
-# shellcheck source=scripts/ci/in_container/_in_container_utils.sh
-. "${MY_DIR}/../_in_container_utils.sh"
-
-assert_in_container
-
-in_container_script_start
-
-# just in case it's not set
-export KUBERNETES_VERSION=${KUBERNETES_VERSION:="v1.15.3"}
-
-echo
-echo "Kubernetes version = ${KUBERNETES_VERSION}"
-echo
-
-function create_cluster() {
-    if [[ "${CI:="false"}" == "true" ]]; then
-        # Travis CI does not handle the nice output of Kind well, so we need to capture it
-        # And display only if kind fails to start
-        start_output_heartbeat "Creating kubernetes cluster" 10
-        set +e
-        if ! OUTPUT=$(kind create cluster \
-                                    --name "${CLUSTER_NAME}" \
-                                    --config "${MY_DIR}/kind-cluster-conf.yaml" \
-                                    --image "kindest/node:${KUBERNETES_VERSION}" 2>&1); then
-            echo "${OUTPUT}"
-        fi
-        stop_output_heartbeat
-    else
-        kind create cluster \
-            --name "${CLUSTER_NAME}" \
-            --config "${MY_DIR}/kind-cluster-conf.yaml" \
-            --image "kindest/node:${KUBERNETES_VERSION}"
-    fi
-    echo
-    echo "Created cluster ${CLUSTER_NAME}"
-    echo
-
-    echo
-    echo "Connecting Kubernetes' worker and ccontrol plane to the network used by Breeze"
-    echo
-    docker network connect docker-compose_default "${CLUSTER_NAME}-control-plane"
-    docker network connect docker-compose_default "${CLUSTER_NAME}-worker"
-    echo
-    echo "Connected Kubernetes' worker and ccontrol plane to the network used by Breeze"
-    echo
-
-
-    echo
-    echo "Replacing cluster host address with https://${CLUSTER_NAME}-control-plane:6443"
-    echo
-    kubectl config set "clusters.kind-${CLUSTER_NAME}.server" "https://${CLUSTER_NAME}-control-plane:6443"
-
-    echo
-    echo "Replaced cluster host address with https://${CLUSTER_NAME}-control-plane:6443"
-    echo
-
-    echo
-    echo "Patching CoreDNS to avoid loop and to use 8.8.8.8 DNS as forward address."
-    echo
-    echo "============================================================================"
-    echo "      Original coredns configmap:"
-    echo "============================================================================"
-    kubectl get configmaps --namespace=kube-system coredns -o yaml
-    kubectl get configmaps --namespace=kube-system coredns -o yaml | \
-        sed 's/forward \. .*$/forward . 8.8.8.8/' | kubectl apply -f -
-
-    echo
-    echo "============================================================================"
-    echo "      Updated coredns configmap with new forward directive:"
-    echo "============================================================================"
-    kubectl get configmaps --namespace=kube-system coredns -o yaml
-
-
-    echo
-    echo "Restarting CoreDNS"
-    echo
-    kubectl scale deployment --namespace=kube-system coredns --replicas=0
-    kubectl scale deployment --namespace=kube-system coredns --replicas=2
-    echo
-    echo "Restarted CoreDNS"
-    echo
-}
-
-function delete_cluster() {
-    kind delete cluster --name "${CLUSTER_NAME}"
-    echo
-    echo "Deleted cluster ${CLUSTER_NAME}"
-    echo
-    rm -rf "${HOME}/.kube/*"
-}
-
-ALL_CLUSTERS=$(kind get clusters || true)
-if [[ ${ALL_CLUSTERS} == *"${CLUSTER_NAME}"* ]]; then
-    echo
-    echo "Cluster ${CLUSTER_NAME} is already created"
-    echo
-    if [[ ${KIND_CLUSTER_OPERATION} == "start" ]]; then
-        echo
-        echo "Reusing previously created cluster"
-        echo
-    elif [[ ${KIND_CLUSTER_OPERATION} == "recreate" ]]; then
-        echo
-        echo "Recreating cluster"
-        echo
-        delete_cluster
-        create_cluster
-    elif [[ ${KIND_CLUSTER_OPERATION} == "stop" ]]; then
-        echo
-        echo "Deleting cluster"
-        echo
-        delete_cluster
-        exit
-    else
-        echo
-        echo "Wrong cluster operation: ${KIND_CLUSTER_OPERATION}. Should be one of start/stop/recreate"
-        echo
-        exit 1
-    fi
-else
-    if [[ ${KIND_CLUSTER_OPERATION} == "start" ]]; then
-        echo
-        echo "Creating cluster"
-        echo
-        create_cluster
-    elif [[ ${KIND_CLUSTER_OPERATION} == "restart" ]]; then
-        echo
-        echo "Cluster ${CLUSTER_NAME} does not exist. Creating rather than recreating"
-        echo
-        echo "Creating cluster"
-        echo
-        create_cluster
-    elif [[ ${KIND_CLUSTER_OPERATION} == "stop" ]]; then
-        echo
-        echo "Cluster ${CLUSTER_NAME} does not exist. It should exist for stop operation"
-        echo
-        exit 1
-    else
-        echo
-        echo "Wrong cluster operation: ${KIND_CLUSTER_OPERATION}. Should be one of start/stop/recreate"
-        echo
-        exit 1
-    fi
-fi
-
-kubectl cluster-info
-
-kubectl get nodes
-echo
-echo "Showing storageClass"
-echo
-kubectl get storageclass
-echo
-echo "Showing kube-system pods"
-echo
-kubectl get -n kube-system pods
-echo
-echo "Airflow environment on kubernetes is good to go!"
-echo
-kubectl create namespace test-namespace
-
-in_container_script_end
diff --git a/scripts/ci/in_container/run_ci_tests.sh b/scripts/ci/in_container/run_ci_tests.sh
index b046c08..f5f4340 100755
--- a/scripts/ci/in_container/run_ci_tests.sh
+++ b/scripts/ci/in_container/run_ci_tests.sh
@@ -41,8 +41,4 @@ if [[ ${CI} == "true" ]]; then
     send_airflow_logs_to_file_io
 fi
 
-if [[ ${CI} == "true" && ${ENABLE_KIND_CLUSTER} == "true" ]]; then
-    send_kubernetes_logs_to_file_io
-fi
-
 exit "${RES}"
diff --git a/scripts/ci/in_container/kubernetes/app/postgres.yaml b/scripts/ci/kubernetes/app/postgres.yaml
similarity index 100%
rename from scripts/ci/in_container/kubernetes/app/postgres.yaml
rename to scripts/ci/kubernetes/app/postgres.yaml
diff --git a/scripts/ci/in_container/kubernetes/app/secrets.yaml b/scripts/ci/kubernetes/app/secrets.yaml
similarity index 100%
rename from scripts/ci/in_container/kubernetes/app/secrets.yaml
rename to scripts/ci/kubernetes/app/secrets.yaml
diff --git a/scripts/ci/in_container/kubernetes/app/templates/airflow.template.yaml b/scripts/ci/kubernetes/app/templates/airflow.template.yaml
similarity index 70%
rename from scripts/ci/in_container/kubernetes/app/templates/airflow.template.yaml
rename to scripts/ci/kubernetes/app/templates/airflow.template.yaml
index 0ee9492..3f2465d 100644
--- a/scripts/ci/in_container/kubernetes/app/templates/airflow.template.yaml
+++ b/scripts/ci/kubernetes/app/templates/airflow.template.yaml
@@ -46,31 +46,53 @@ spec:
         runAsGroup: 50000
         fsGroup: 50000
       initContainers:
-      - name: "init"
-        image: {{AIRFLOW_KUBERNETES_IMAGE}}
-        imagePullPolicy: IfNotPresent
-        volumeMounts:
-        - name: airflow-configmap
-          mountPath: /opt/airflow/airflow.cfg
-          subPath: airflow.cfg
-        - name: {{INIT_DAGS_VOLUME_NAME}}
-          mountPath: /opt/airflow/dags
-        env:
-        - name: SQL_ALCHEMY_CONN
-          valueFrom:
-            secretKeyRef:
-              name: airflow-secrets
-              key: sql_alchemy_conn
-        command:
-          - "bash"
-        args:
-          - "-cx"
-          - "/tmp/airflow-test-env-init.sh"
+        - name: "init-dags"
+          image: {{AIRFLOW_KUBERNETES_IMAGE}}
+          imagePullPolicy: Never
+          securityContext:
+            runAsUser: 0
+          volumeMounts:
+            - name: airflow-configmap
+              mountPath: /opt/airflow/airflow.cfg
+              subPath: airflow.cfg
+            - name: {{INIT_DAGS_VOLUME_NAME}}
+              mountPath: /opt/airflow/dags
+          env:
+            - name: SQL_ALCHEMY_CONN
+              valueFrom:
+                secretKeyRef:
+                  name: airflow-secrets
+                  key: sql_alchemy_conn
+          command:
+            - "bash"
+          args:
+            - "-cx"
+            - "/tmp/airflow-test-env-init-dags.sh"
+        - name: "init-db"
+          image: {{AIRFLOW_KUBERNETES_IMAGE}}
+          imagePullPolicy: Never
+          volumeMounts:
+          - name: airflow-configmap
+            mountPath: /opt/airflow/airflow.cfg
+            subPath: airflow.cfg
+          - name: {{INIT_DAGS_VOLUME_NAME}}
+            mountPath: /opt/airflow/dags
+          env:
+          - name: SQL_ALCHEMY_CONN
+            valueFrom:
+              secretKeyRef:
+                name: airflow-secrets
+                key: sql_alchemy_conn
+          command:
+            - "bash"
+          args:
+            - "-cx"
+            - "/tmp/airflow-test-env-init-db.sh"
 {{INIT_GIT_SYNC}}
       containers:
       - name: webserver
         image: {{AIRFLOW_KUBERNETES_IMAGE}}
-        imagePullPolicy: IfNotPresent
+        imagePullPolicy: Never
         ports:
         - name: webserver
           containerPort: 8080
@@ -95,7 +117,7 @@ spec:
           mountPath: /opt/airflow/logs
       - name: scheduler
         image: {{AIRFLOW_KUBERNETES_IMAGE}}
-        imagePullPolicy: IfNotPresent
+        imagePullPolicy: Never
         args: ["scheduler"]
         env:
         - name: AIRFLOW__KUBERNETES__NAMESPACE
diff --git a/scripts/ci/in_container/kubernetes/app/templates/configmaps.template.yaml b/scripts/ci/kubernetes/app/templates/configmaps.template.yaml
similarity index 100%
rename from scripts/ci/in_container/kubernetes/app/templates/configmaps.template.yaml
rename to scripts/ci/kubernetes/app/templates/configmaps.template.yaml
diff --git a/scripts/ci/in_container/kubernetes/app/templates/init_git_sync.template.yaml b/scripts/ci/kubernetes/app/templates/init_git_sync.template.yaml
similarity index 55%
rename from scripts/ci/in_container/kubernetes/app/templates/init_git_sync.template.yaml
rename to scripts/ci/kubernetes/app/templates/init_git_sync.template.yaml
index 343ad8c..940b6ec 100644
--- a/scripts/ci/in_container/kubernetes/app/templates/init_git_sync.template.yaml
+++ b/scripts/ci/kubernetes/app/templates/init_git_sync.template.yaml
@@ -15,22 +15,22 @@
 # specific language governing permissions and limitations
 # under the License.
 
-      - name: git-sync-clone
-        env:
-        - name: GIT_SYNC_REPO
-          value: https://github.com/{{CONFIGMAP_GIT_REPO}}.git
-        - name: GIT_SYNC_BRANCH
-          value: {{CONFIGMAP_BRANCH}}
-        - name: GIT_SYNC_ROOT
-          value: /git
-        - name: GIT_SYNC_DEST
-          value: repo
-        - name: GIT_SYNC_ONE_TIME
-          value: "true"
-        image: gcr.io/google-containers/git-sync-amd64:v2.0.5
-        imagePullPolicy: IfNotPresent
-        securityContext:
-          runAsUser: 0
-        volumeMounts:
-        - mountPath: /git
-          name: airflow-dags-git
+        - name: "git-sync-clone"
+          env:
+          - name: GIT_SYNC_REPO
+            value: https://github.com/{{CONFIGMAP_GIT_REPO}}.git
+          - name: GIT_SYNC_BRANCH
+            value: {{CONFIGMAP_BRANCH}}
+          - name: GIT_SYNC_ROOT
+            value: /git
+          - name: GIT_SYNC_DEST
+            value: repo
+          - name: GIT_SYNC_ONE_TIME
+            value: "true"
+          image: gcr.io/google-containers/git-sync-amd64:v2.0.5
+          imagePullPolicy: IfNotPresent
+          securityContext:
+            runAsUser: 0
+          volumeMounts:
+          - mountPath: /git
+            name: airflow-dags-git
diff --git a/scripts/ci/in_container/kubernetes/app/volumes.yaml b/scripts/ci/kubernetes/app/volumes.yaml
similarity index 100%
rename from scripts/ci/in_container/kubernetes/app/volumes.yaml
rename to scripts/ci/kubernetes/app/volumes.yaml
diff --git a/scripts/ci/in_container/kubernetes/docker/airflow-test-env-init.sh b/scripts/ci/kubernetes/docker/airflow-test-env-init-dags.sh
similarity index 70%
rename from scripts/ci/in_container/kubernetes/docker/airflow-test-env-init.sh
rename to scripts/ci/kubernetes/docker/airflow-test-env-init-dags.sh
index 8e15f0f..9722958 100755
--- a/scripts/ci/in_container/kubernetes/docker/airflow-test-env-init.sh
+++ b/scripts/ci/kubernetes/docker/airflow-test-env-init-dags.sh
@@ -16,8 +16,21 @@
 # specific language governing permissions and limitations
 # under the License.
 
-set -x
+set -euo pipefail
 
-cd /opt/airflow && \
-airflow initdb && \
-(airflow create_user -u airflow -l airflow -f jon -e airflow@apache.org -r Admin -p airflow || true)
+echo
+echo "Copying airflow dags"
+echo
+
+
+# Create DAGS folder if it does not exist
+mkdir -pv "${AIRFLOW_HOME}/dags"
+ls -la "${AIRFLOW_HOME}/dags/"
+rm -rvf "${AIRFLOW_HOME}/dags/*"
+
+# Copy DAGS from current sources
+cp -Rv "${AIRFLOW_SOURCES}"/airflow/example_dags/* "${AIRFLOW_HOME}/dags/"
+
+echo
+echo "Copied airflow dags"
+echo
diff --git a/scripts/ci/docker-compose/runtime-kubernetes.yml b/scripts/ci/kubernetes/docker/airflow-test-env-init-db.sh
old mode 100644
new mode 100755
similarity index 60%
rename from scripts/ci/docker-compose/runtime-kubernetes.yml
rename to scripts/ci/kubernetes/docker/airflow-test-env-init-db.sh
index 3b96f46..c790160
--- a/scripts/ci/docker-compose/runtime-kubernetes.yml
+++ b/scripts/ci/kubernetes/docker/airflow-test-env-init-db.sh
@@ -1,3 +1,4 @@
+#!/usr/bin/env bash
 # Licensed to the Apache Software Foundation (ASF) under one
 # or more contributor license agreements.  See the NOTICE file
 # distributed with this work for additional information
@@ -14,12 +15,32 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
----
-version: "2.2"
-services:
-  airflow:
-    environment:
-      - RUNTIME=kubernetes
-      - KUBERNETES_MODE
-      - KUBERNETES_VERSION
-      - KIND_CLUSTER_OPERATION
+
+set -euo pipefail
+
+echo
+echo "Initializing the Airflow db"
+echo
+
+
+# Init and upgrade the database to latest heads
+cd "${AIRFLOW_SOURCES}"/airflow || exit 1
+
+airflow db init
+alembic upgrade heads
+
+echo
+echo "Initialized the database"
+echo
+
+# Create Airflow User if it does not exist
+airflow create_user \
+    --username airflow \
+    --lastname airflow \
+    --firstname jon \
+    --email airflow@apache.org \
+    --role Admin --password airflow || true
+
+echo
+echo "Created airflow user"
+echo
diff --git a/scripts/ci/kubernetes/docker/bootstrap.sh b/scripts/ci/kubernetes/docker/bootstrap.sh
new file mode 100755
index 0000000..9099b6b
--- /dev/null
+++ b/scripts/ci/kubernetes/docker/bootstrap.sh
@@ -0,0 +1,74 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+set -euo pipefail
+
+# This part is to allow fast iteration with the kubernetes tests
+# Pre-installed Airflow from the production image is removed and Airflow is re-installed from the
+# sources added during preparing the kubernetes image. This way when we deploy the image
+# to KinD we only add latest sources and only that most recent layer is sent to kind
+# and airflow always runs with compiled dist web files from pre-compiled dist installed in prod image
+
+
+echo
+echo "Save minimised web files"
+echo
+
+mv "$(python -m site | grep ^USER_SITE | awk '{print $2}' | tr -d "'")/airflow/www/static/dist/" \
+    "/tmp"
+
+echo
+echo "Uninstalling pre-installed airflow"
+echo
+
+# Uninstall preinstalled Apache Airlfow
+pip uninstall -y apache-airflow
+
+
+echo
+echo "Installing airflow from the sources"
+echo
+
+# Installing airflow from the sources copied to the Kubernetes image
+pip install --user "${AIRFLOW_SOURCES}"
+
+echo
+echo "Restore minimised web files"
+echo
+
+mv "/tmp/dist" "$(python -m site | grep ^USER_SITE | awk '{print $2}' | tr -d "'")/airflow/www/static/"
+
+echo
+echo "Airflow prepared. Running ${1}"
+echo
+
+
+if [[ "$1" = "webserver" ]]
+then
+    exec airflow webserver
+fi
+
+if [[ "$1" = "scheduler" ]]
+then
+    exec airflow scheduler
+fi
+
+echo
+echo "Entering bash"
+echo
+
+exec /bin/bash
diff --git a/scripts/ci/in_container/kubernetes/kind-cluster-conf.yaml b/scripts/ci/kubernetes/kind-cluster-conf.yaml
similarity index 100%
rename from scripts/ci/in_container/kubernetes/kind-cluster-conf.yaml
rename to scripts/ci/kubernetes/kind-cluster-conf.yaml
diff --git a/tests/conftest.py b/tests/conftest.py
index f39eadd..e7de7fc 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -83,12 +83,6 @@ def pytest_addoption(parser):
         help="only run tests matching the backend: [sqlite,postgres,mysql].",
     )
     group.addoption(
-        "--runtime",
-        action="store",
-        metavar="RUNTIME",
-        help="only run tests matching the runtime: [kubernetes].",
-    )
-    group.addoption(
         "--system",
         action="append",
         metavar="SYSTEMS",
@@ -174,9 +168,6 @@ def pytest_configure(config):
         "markers", "backend(name): mark test to run with named backend"
     )
     config.addinivalue_line(
-        "markers", "runtime(name): mark test to run with named runtime"
-    )
-    config.addinivalue_line(
         "markers", "system(name): mark test to run with named system"
     )
     config.addinivalue_line(
@@ -212,16 +203,6 @@ def skip_if_not_marked_with_backend(selected_backend, item):
                 format(backend=selected_backend, item=item))
 
 
-def skip_if_not_marked_with_runtime(selected_runtime, item):
-    for marker in item.iter_markers(name="runtime"):
-        runtime_name = marker.args[0]
-        if runtime_name == selected_runtime:
-            return
-    pytest.skip("The test is skipped because it has not been selected via --runtime switch. "
-                "Only tests marked with pytest.mark.runtime('{runtime}') are run: {item}".
-                format(runtime=selected_runtime, item=item))
-
-
 def skip_if_not_marked_with_system(selected_systems, item):
     for marker in item.iter_markers(name="system"):
         systems_name = marker.args[0]
@@ -268,19 +249,6 @@ def skip_if_integration_disabled(marker, item):
                            integration_name=integration_name, item=item))
 
 
-def skip_if_runtime_disabled(marker, item):
-    runtime_name = marker.args[0]
-    environment_variable_name = "RUNTIME"
-    environment_variable_value = os.environ.get(environment_variable_name)
-    if not environment_variable_value or environment_variable_value != runtime_name:
-        pytest.skip("The test requires {runtime_name} integration started and "
-                    "{name} environment variable to be set to true (it is '{value}')."
-                    " It can be set by specifying '--environment {runtime_name}' at breeze startup"
-                    ": {item}".
-                    format(name=environment_variable_name, value=environment_variable_value,
-                           runtime_name=runtime_name, item=item))
-
-
 def skip_if_wrong_backend(marker, item):
     valid_backend_names = marker.args
     environment_variable_name = "BACKEND"
@@ -323,11 +291,6 @@ def pytest_runtest_setup(item):
     selected_backend = item.config.getoption("--backend")
     if selected_backend:
         skip_if_not_marked_with_backend(selected_backend, item)
-    for marker in item.iter_markers(name="runtime"):
-        skip_if_runtime_disabled(marker, item)
-    selected_runtime = item.config.getoption("--runtime")
-    if selected_runtime:
-        skip_if_not_marked_with_runtime(selected_runtime, item)
     if not include_long_running:
         skip_long_running_test(item)
     if not include_quarantined:
diff --git a/tests/providers/cncf/kubernetes/operators/test_kubernetes_pod.py b/tests/providers/cncf/kubernetes/operators/test_kubernetes_pod.py
new file mode 100644
index 0000000..266ce05
--- /dev/null
+++ b/tests/providers/cncf/kubernetes/operators/test_kubernetes_pod.py
@@ -0,0 +1,134 @@
+# pylint: disable=unused-argument
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+from unittest import mock
+
+import kubernetes.client.models as k8s
+import pendulum
+
+from airflow.exceptions import AirflowException
+from airflow.models import DAG, TaskInstance
+from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
+from airflow.utils import timezone
+
+
+# noinspection PyUnusedLocal
+class TestKubernetesPodOperator(unittest.TestCase):
+
+    # noinspection DuplicatedCode
+    @staticmethod
+    def create_context(task):
+        dag = DAG(dag_id="dag")
+        tzinfo = pendulum.timezone("Europe/Amsterdam")
+        execution_date = timezone.datetime(2016, 1, 1, 1, 0, 0, tzinfo=tzinfo)
+        task_instance = TaskInstance(task=task,
+                                     execution_date=execution_date)
+        return {
+            "dag": dag,
+            "ts": execution_date.isoformat(),
+            "task": task,
+            "ti": task_instance,
+        }
+
+    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.start_pod")
+    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.monitor_pod")
+    @mock.patch("airflow.kubernetes.kube_client.get_kube_client")
+    def test_config_path(self, client_mock, monitor_mock, start_mock):  # pylint: disable=unused-argument
+        from airflow.utils.state import State
+
+        file_path = "/tmp/fake_file"
+        k = KubernetesPodOperator(
+            namespace='default',
+            image="ubuntu:16.04",
+            cmds=["bash", "-cx"],
+            arguments=["echo 10"],
+            labels={"foo": "bar"},
+            name="test",
+            task_id="task",
+            in_cluster=False,
+            do_xcom_push=False,
+            config_file=file_path,
+            cluster_context='default',
+        )
+        monitor_mock.return_value = (State.SUCCESS, None)
+        client_mock.list_namespaced_pod.return_value = []
+        context = self.create_context(k)
+        k.execute(context=context)
+        client_mock.assert_called_once_with(
+            in_cluster=False,
+            cluster_context='default',
+            config_file=file_path,
+        )
+
+    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.start_pod")
+    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.monitor_pod")
+    @mock.patch("airflow.kubernetes.kube_client.get_kube_client")
+    def test_image_pull_secrets_correctly_set(self, mock_client, monitor_mock, start_mock):
+        from airflow.utils.state import State
+
+        fake_pull_secrets = "fakeSecret"
+        k = KubernetesPodOperator(
+            namespace='default',
+            image="ubuntu:16.04",
+            cmds=["bash", "-cx"],
+            arguments=["echo 10"],
+            labels={"foo": "bar"},
+            name="test",
+            task_id="task",
+            in_cluster=False,
+            do_xcom_push=False,
+            image_pull_secrets=fake_pull_secrets,
+            cluster_context='default',
+        )
+        monitor_mock.return_value = (State.SUCCESS, None)
+        context = self.create_context(k)
+        k.execute(context=context)
+        self.assertEqual(
+            start_mock.call_args[0][0].spec.image_pull_secrets,
+            [k8s.V1LocalObjectReference(name=fake_pull_secrets)]
+        )
+
+    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.start_pod")
+    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.monitor_pod")
+    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.delete_pod")
+    @mock.patch("airflow.kubernetes.kube_client.get_kube_client")
+    def test_pod_delete_even_on_launcher_error(
+            self,
+            mock_client,
+            delete_pod_mock,
+            monitor_pod_mock,
+            start_pod_mock):
+        k = KubernetesPodOperator(
+            namespace='default',
+            image="ubuntu:16.04",
+            cmds=["bash", "-cx"],
+            arguments=["echo 10"],
+            labels={"foo": "bar"},
+            name="test",
+            task_id="task",
+            in_cluster=False,
+            do_xcom_push=False,
+            cluster_context='default',
+            is_delete_operator_pod=True,
+        )
+        monitor_pod_mock.side_effect = AirflowException('fake failure')
+        with self.assertRaises(AirflowException):
+            context = self.create_context(k)
+            k.execute(context=context)
+        assert delete_pod_mock.called
diff --git a/tests/runtime/__init__.py b/tests/runtime/__init__.py
deleted file mode 100644
index 114d189..0000000
--- a/tests/runtime/__init__.py
+++ /dev/null
@@ -1,18 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#   http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied.  See the License for the
-# specific language governing permissions and limitations
-# under the License.


[airflow] 06/36: Useful help information in test-target and docker-compose commands (#8796)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit ff5b33ea1f2e86a75b10ca435b6207f4a431dbc1
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Mon May 11 11:58:25 2020 +0200

    Useful help information in test-target and docker-compose commands (#8796)
    
    There was no useful information printed in test-target and
    docker-compose commands. It's fixed now
    
    (cherry picked from commit d5c4001f8db198be31e366a39b0faa8543e45844)
---
 .pre-commit-config.yaml |  11 -----
 BREEZE.rst              |  30 ++++++++------
 breeze                  | 104 ++++++++++++++++++++++++++++++++----------------
 breeze-complete         |  35 ++++++++++++++--
 4 files changed, 120 insertions(+), 60 deletions(-)

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 3621795..68ddcd5 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -242,17 +242,6 @@ repos:
         files: \.py$
         exclude: ^airflow/_vendor|^dev/
         pass_filenames: true
-      - id: check-providers-package
-        language: pygrep
-        name: Find providers package added in 2.0.*
-        entry: >
-          (?x)
-            (
-                ^.*airflow\.providers.*     # Matches import airflow.providers
-            )$
-        files: \.py$
-        exclude: ^airflow/_vendor
-        pass_filenames: true
       - id: python2-compile
         name: Compile code using python2
         language: system
diff --git a/BREEZE.rst b/BREEZE.rst
index 7212d0e..44af4e5 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -954,7 +954,7 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: docker-compose
 
-  breeze [FLAGS] docker-compose -- <EXTRA_ARGS>
+  breeze [FLAGS] docker-compose <DOCKER_COMPOSE_COMMAND> -- <EXTRA_ARGS>
 
         Run docker-compose command instead of entering the environment. Use 'help' as command
         to see available commands. The <EXTRA_ARGS> passed after -- are treated
@@ -1046,15 +1046,18 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: static-check
 
-  breeze [FLAGS] static-check -- <EXTRA_ARGS>
+  breeze [FLAGS] static-check <STATIC_CHECK> -- <EXTRA_ARGS>
 
         Run selected static checks for currently changed files. You should specify static check that
         you would like to run or 'all' to run all checks. One of:
 
-                 all bat-tests check-apache-license check-executables-have-shebangs check-hooks-apply
-                 check-merge-conflict check-xml debug-statements doctoc detect-private-key
-                 end-of-file-fixer flake8 forbid-tabs insert-license lint-dockerfile
-                 mixed-line-ending mypy setup-order shellcheck
+                 all airflow-config-yaml bat-tests build check-apache-license
+                 check-executables-have-shebangs check-hooks-apply check-integrations
+                 check-merge-conflict check-xml debug-statements detect-private-key doctoc
+                 end-of-file-fixer fix-encoding-pragma flake8 forbid-tabs insert-license
+                 lint-dockerfile mixed-line-ending mypy pydevd python2-compile python2-fastcheck
+                 python-no-log-warn rst-backticks setup-order shellcheck trailing-whitespace
+                 update-breeze-file update-extras update-local-yml-file yamllint
 
         You can pass extra arguments including options to to the pre-commit framework as
         <EXTRA_ARGS> passed after --. For example:
@@ -1072,15 +1075,18 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: static-check-all-files
 
-  breeze [FLAGS] static-check-all-files -- <EXTRA_ARGS>
+  breeze [FLAGS] static-check-all-files <STATIC_CHECK> -- <EXTRA_ARGS>
 
         Run selected static checks for all applicable files. You should specify static check that
         you would like to run or 'all' to run all checks. One of:
 
-                 all bat-tests check-apache-license check-executables-have-shebangs check-hooks-apply
-                 check-merge-conflict check-xml debug-statements doctoc detect-private-key
-                 end-of-file-fixer flake8 forbid-tabs insert-license lint-dockerfile
-                 mixed-line-ending mypy setup-order shellcheck
+                 all airflow-config-yaml bat-tests build check-apache-license
+                 check-executables-have-shebangs check-hooks-apply check-integrations
+                 check-merge-conflict check-xml debug-statements detect-private-key doctoc
+                 end-of-file-fixer fix-encoding-pragma flake8 forbid-tabs insert-license
+                 lint-dockerfile mixed-line-ending mypy pydevd python2-compile python2-fastcheck
+                 python-no-log-warn rst-backticks setup-order shellcheck trailing-whitespace
+                 update-breeze-file update-extras update-local-yml-file yamllint
 
         You can pass extra arguments including options to the pre-commit framework as
         <EXTRA_ARGS> passed after --. For example:
@@ -1098,7 +1104,7 @@ This is the current syntax for  `./breeze <./breeze>`_:
 
   Detailed usage for command: test-target
 
-  breeze [FLAGS] test-target -- <EXTRA_ARGS>
+  breeze [FLAGS] test-target <TEST_TARGET> -- <EXTRA_ARGS>
 
         Run the specified unit test target. There might be multiple
         targets specified separated with comas. The <EXTRA_ARGS> passed after -- are treated
diff --git a/breeze b/breeze
index 4ec67c6..7f5958b 100755
--- a/breeze
+++ b/breeze
@@ -837,14 +837,17 @@ function parse_arguments() {
           COMMAND_TO_RUN="cleanup_image"
           shift ;;
         docker-compose)
-          if [[ "$#" -lt 2 ]]; then
+          LAST_SUBCOMMAND="${1}"
+          if [[ $# -lt 2 ]]; then
             echo "You should specify docker compose command to run"
-            exit 1
+            shift
+            RUN_HELP="true"
+          else
+            DOCKER_COMPOSE_COMMAND="${2}"
+            shift 2
           fi
-          LAST_SUBCOMMAND="${1}"
-          DOCKER_COMPOSE_COMMAND="${2}"
           COMMAND_TO_RUN="run_docker_compose"
-          shift 2 ;;
+          ;;
         execute-command)
           LAST_SUBCOMMAND="${1}"
           COMMAND_TO_RUN="run_in_bash"
@@ -869,34 +872,56 @@ function parse_arguments() {
           LAST_SUBCOMMAND="${1}"
           COMMAND_TO_RUN="perform_static_checks"
           if [[ "$#" -lt 2 ]]; then
-            echo "You should specify static check that you would like to run or 'all' to run all checks."
-            echo "One of [${_BREEZE_ALLOWED_STATIC_CHECKS:=}]."
-            echo
-            echo "For example:"
-            echo "${CMDNAME} static-check MyPy"
-            exit 1
+            if [[ ${RUN_HELP} != "true" ]]; then
+              echo "You should specify static check that you would like to run or 'all' to run all checks."
+              echo
+              echo "One of :"
+              echo
+              echo "${_BREEZE_ALLOWED_STATIC_CHECKS:=}"
+              echo
+              echo "For example:"
+              echo
+              echo "${CMDNAME} static-check mypy"
+              echo
+              exit 1
+            else
+              shift
+            fi
+          else
+            export PYTHON_MAJOR_MINOR_VERSION=${STATIC_CHECK_PYTHON_MAJOR_MINOR_VERSION}
+            export STATIC_CHECK="${2:-}"
+            export STATIC_CHECK_ALL_FILES="false"
+            EXTRA_STATIC_CHECK_OPTIONS+=("--show-diff-on-failure")
+            shift 2
           fi
-          export PYTHON_MAJOR_MINOR_VERSION=${STATIC_CHECK_PYTHON_MAJOR_MINOR_VERSION}
-          export STATIC_CHECK="${2}"
-          export STATIC_CHECK_ALL_FILES="false"
-          EXTRA_STATIC_CHECK_OPTIONS+=("--show-diff-on-failure")
-          shift 2 ;;
+          ;;
         static-check-all-files)
           LAST_SUBCOMMAND="${1}"
+          COMMAND_TO_RUN="perform_static_checks"
           if [[ "$#" -lt 2 ]]; then
-            echo "You should specify static check that you would like to run or 'all' to run all checks."
-            echo "One of [${_BREEZE_ALLOWED_STATIC_CHECKS:=}]."
-            echo
-            echo "For example:"
-            echo "${CMDNAME} static-check-all-files MyPy"
-            exit 1
+            if [[ ${RUN_HELP} != "true" ]]; then
+              echo "You should specify static check that you would like to run or 'all' to run all checks."
+              echo
+              echo "One of :"
+              echo
+              echo "${_BREEZE_ALLOWED_STATIC_CHECKS:=}"
+              echo
+              echo "For example:"
+              echo
+              echo "${CMDNAME} static-check-all-files mypy"
+              echo
+              exit 1
+            else
+              shift
+            fi
+          else
+            export PYTHON_MAJOR_MINOR_VERSION=${STATIC_CHECK_PYTHON_MAJOR_MINOR_VERSION}
+            export STATIC_CHECK="${2:-}"
+            export STATIC_CHECK_ALL_FILES="true"
+            EXTRA_STATIC_CHECK_OPTIONS+=("--all-files" "--show-diff-on-failure")
+            shift 2
           fi
-          COMMAND_TO_RUN="perform_static_checks"
-          export PYTHON_MAJOR_MINOR_VERSION=${STATIC_CHECK_PYTHON_MAJOR_MINOR_VERSION}
-          export STATIC_CHECK="${2}"
-          export STATIC_CHECK_ALL_FILES="true"
-          EXTRA_STATIC_CHECK_OPTIONS+=("--all-files" "--show-diff-on-failure")
-          shift 2 ;;
+          ;;
         stop)
           LAST_SUBCOMMAND="${1}"
           COMMAND_TO_RUN="run_docker_compose"
@@ -913,13 +938,14 @@ function parse_arguments() {
           shift ;;
         test-target)
           LAST_SUBCOMMAND="${1}"
-          if [[ "${TEST_TARGET}" == "." ]]; then
-              export TEST_TARGET=""
+          if [[ $# -lt 2 ]]; then
+            RUN_HELP="true"
+            shift
           else
-              export TEST_TARGET="${2}"
+            export TEST_TARGET="${2}"
+            shift 2
           fi
-          COMMAND_TO_RUN="run_tests"
-          shift 2 ;;
+          COMMAND_TO_RUN="run_tests" ;;
         toggle-suppress-cheatsheet)
           LAST_SUBCOMMAND="${1}"
           if [[ -f "${SUPPRESS_CHEATSHEET_FILE}" ]]; then
@@ -1289,11 +1315,21 @@ Help commands:
 # Prints detailed usage for command specified
 function detailed_usage() {
       SUBCOMMAND=${1}
+      COMMAND_PARAMETER=""
+      if [[ ${SUBCOMMAND} == "static-check" ]]; then
+        COMMAND_PARAMETER="<STATIC_CHECK> "
+      elif [[ ${SUBCOMMAND} == "static-check-all-files" ]]; then
+        COMMAND_PARAMETER="<STATIC_CHECK> "
+      elif [[ ${SUBCOMMAND} == "test-target" ]]; then
+        COMMAND_PARAMETER="<TEST_TARGET> "
+      elif [[ ${SUBCOMMAND} == "docker-compose" ]]; then
+        COMMAND_PARAMETER="<DOCKER_COMPOSE_COMMAND> "
+      fi
       echo "
 
 Detailed usage for command: ${SUBCOMMAND}
 
-${CMDNAME} [FLAGS] ${SUBCOMMAND} -- <EXTRA_ARGS>
+${CMDNAME} [FLAGS] ${SUBCOMMAND} ${COMMAND_PARAMETER}-- <EXTRA_ARGS>
 $(get_detailed_usage "${SUBCOMMAND}")
 
 "
diff --git a/breeze-complete b/breeze-complete
index 4fea00f..3164851 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -42,9 +42,38 @@ EOF
 
 _BREEZE_ALLOWED_STATIC_CHECKS=$(cat <<-EOF
 all
-bat-tests check-apache-license check-executables-have-shebangs check-hooks-apply check-merge-conflict
-check-xml debug-statements doctoc detect-private-key end-of-file-fixer flake8 forbid-tabs
-insert-license lint-dockerfile mixed-line-ending mypy setup-order shellcheck
+airflow-config-yaml
+bat-tests
+build
+check-apache-license
+check-executables-have-shebangs
+check-hooks-apply
+check-integrations
+check-merge-conflict
+check-xml
+debug-statements
+detect-private-key
+doctoc
+end-of-file-fixer
+fix-encoding-pragma
+flake8
+forbid-tabs
+insert-license
+lint-dockerfile
+mixed-line-ending
+mypy
+pydevd
+python2-compile
+python2-fastcheck
+python-no-log-warn
+rst-backticks
+setup-order
+shellcheck
+trailing-whitespace
+update-breeze-file
+update-extras
+update-local-yml-file
+yamllint
 EOF
 )
 


[airflow] 16/36: Adds hive as extra in pyhive (#9075)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 374a3e89ac160b1494bfaa13508f6e833da41e9e
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sat May 30 20:59:04 2020 +0200

    Adds hive as extra in pyhive (#9075)
    
    Seems that apache hive needs to install [hive] extra of pyhive
    in order to be usable ¯\_(ツ)_/¯.
    
    Fixes: #8933
    (cherry picked from commit db4297406d82406d52510ce8e5376b07b037db15)
---
 setup.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/setup.py b/setup.py
index 9166279..5b279f0 100644
--- a/setup.py
+++ b/setup.py
@@ -279,7 +279,7 @@ hdfs = [
 ]
 hive = [
     'hmsclient>=0.1.0',
-    'pyhive>=0.6.0',
+    'pyhive[hive]>=0.6.0',
 ]
 jdbc = [
     'JPype1==0.7.1',


[airflow] 34/36: Add PR/issue note in Contribution Workflow Example (#9177)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit ec123341ee40d0e5a65726e8d96e45b810f5486c
Author: Kamil Breguła <mi...@users.noreply.github.com>
AuthorDate: Mon Jun 8 13:24:56 2020 +0200

    Add PR/issue note in Contribution Workflow Example (#9177)
    
    * Add PR/issue note in Contribution Workflow Example
    
    * Update CONTRIBUTING.rst
    
    Co-authored-by: Eric Lopes <nu...@users.noreply.github.com>
    
    * Update CONTRIBUTING.rst
    
    Co-authored-by: Eric Lopes <nu...@users.noreply.github.com>
    
    * Update CONTRIBUTING.rst
    
    Co-authored-by: Eric Lopes <nu...@users.noreply.github.com>
    (cherry picked from commit 2038b6957c48de8b2ad580d4ba7f51c1d1c98252)
---
 CONTRIBUTING.rst | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index 11199df..1b8ce2f 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -665,6 +665,9 @@ Contribution Workflow Example
 Typically, you start your first contribution by reviewing open tickets
 at `Apache JIRA <https://issues.apache.org/jira/browse/AIRFLOW>`__.
 
+If you create pull-request, you don't have to create an issue first, but if you want, you can do it.
+Creating an issue will allow you to collect feedback or share plans with other people.
+
 For example, you want to have the following sample ticket assigned to you:
 `AIRFLOW-5934: Add extra CC: to the emails sent by Aiflow <https://issues.apache.org/jira/browse/AIRFLOW-5934>`_.
 


[airflow] 14/36: Additional python extras and deps can be set in breeze (#9035)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 62c97ed6fa6739fb94797be871b1e703501fadea
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Wed May 27 17:09:11 2020 +0200

    Additional python extras and deps can be set in breeze (#9035)
    
    Closes #8604
    Closes #8866
    
    (cherry picked from commit 738667082d32d3ef93ec2cd6c3735ff3691ba1cc)
---
 BREEZE.rst           | 12 ++++++++++++
 Dockerfile           |  4 ++--
 Dockerfile.ci        |  3 ++-
 breeze               | 14 ++++++++++++++
 breeze-complete      |  1 +
 scripts/ci/_utils.sh | 12 +++++++++++-
 6 files changed, 42 insertions(+), 4 deletions(-)

diff --git a/BREEZE.rst b/BREEZE.rst
index 44af4e5..262ddcd 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -774,6 +774,12 @@ This is the current syntax for  `./breeze <./breeze>`_:
                  async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,
                  ssh,statsd,virtualenv
 
+  --additional-extras
+          Additional extras to pass to build images The default is no additional extras.
+
+  --additional-python-deps
+          Additional python dependencies to use when building the images.
+
   -C, --force-clean-images
           Force build images with cache disabled. This will remove the pulled or build images
           and start building images from scratch. This might take a long time.
@@ -1312,6 +1318,12 @@ This is the current syntax for  `./breeze <./breeze>`_:
                  async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,
                  ssh,statsd,virtualenv
 
+  --additional-extras
+          Additional extras to pass to build images The default is no additional extras.
+
+  --additional-python-deps
+          Additional python dependencies to use when building the images.
+
   -C, --force-clean-images
           Force build images with cache disabled. This will remove the pulled or build images
           and start building images from scratch. This might take a long time.
diff --git a/Dockerfile b/Dockerfile
index ef3a988..ac27823 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -169,10 +169,10 @@ ARG AIRFLOW_VERSION
 ENV AIRFLOW_VERSION=${AIRFLOW_VERSION}
 
 ARG AIRFLOW_EXTRAS
-ARG ADDITIONAL_AIRFLOW_EXTRAS
+ARG ADDITIONAL_AIRFLOW_EXTRAS=""
 ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}${ADDITIONAL_AIRFLOW_EXTRAS:+,}${ADDITIONAL_AIRFLOW_EXTRAS}
 
-ARG ADDITIONAL_PYTHON_DEPS
+ARG ADDITIONAL_PYTHON_DEPS=""
 ENV ADDITIONAL_PYTHON_DEPS=${ADDITIONAL_PYTHON_DEPS}
 
 ARG AIRFLOW_INSTALL_SOURCES="."
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 15f3732..2d107aa 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -295,7 +295,8 @@ ENV AIRFLOW_BRANCH=${AIRFLOW_BRANCH}
 
 # Airflow Extras installed
 ARG AIRFLOW_EXTRAS="all"
-ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}
+ARG ADDITIONAL_AIRFLOW_EXTRAS=""
+ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}${ADDITIONAL_AIRFLOW_EXTRAS:+,}${ADDITIONAL_AIRFLOW_EXTRAS}
 
 RUN echo "Installing with extras: ${AIRFLOW_EXTRAS}."
 
diff --git a/breeze b/breeze
index 7f5958b..b2e3c67 100755
--- a/breeze
+++ b/breeze
@@ -772,6 +772,14 @@ function parse_arguments() {
           export AIRFLOW_EXTRAS="${2}"
           echo "Extras : ${AIRFLOW_EXTRAS}"
           shift 2 ;;
+        --additional-extras)
+          export ADDITIONAL_AIRFLOW_EXTRAS="${2}"
+          echo "Additional extras : ${ADDITIONAL_AIRFLOW_EXTRAS}"
+          shift 2 ;;
+        --additional-python-deps)
+          export ADDITIONAL_PYTHON_DEPS="${2}"
+          echo "Additional python dependencies: ${ADDITIONAL_PYTHON_DEPS}"
+          shift 2 ;;
         -D|--dockerhub-user)
           export DOCKERHUB_USER="${2}"
           echo "Dockerhub user ${DOCKERHUB_USER}"
@@ -1527,6 +1535,12 @@ ${FORMATTED_DEFAULT_CI_EXTRAS}
         Production image:
 ${FORMATTED_DEFAULT_PROD_EXTRAS}
 
+--additional-extras
+        Additional extras to pass to build images The default is no additional extras.
+
+--additional-python-deps
+        Additional python dependencies to use when building the images.
+
 -C, --force-clean-images
         Force build images with cache disabled. This will remove the pulled or build images
         and start building images from scratch. This might take a long time.
diff --git a/breeze-complete b/breeze-complete
index 3164851..8afe550 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -99,6 +99,7 @@ verbose assume-yes assume-no assume-quit forward-credentials
 force-build-images force-pull-images production-image extras: force-clean-images use-local-cache
 dockerhub-user: dockerhub-repo: push-images
 postgres-version: mysql-version:
+additional-extras: additional-python-deps:
 "
 
 export BREEZE_COMMANDS="
diff --git a/scripts/ci/_utils.sh b/scripts/ci/_utils.sh
index 7bbd2dd..85aeb91 100644
--- a/scripts/ci/_utils.sh
+++ b/scripts/ci/_utils.sh
@@ -1311,6 +1311,8 @@ Docker building ${AIRFLOW_CI_IMAGE}.
             --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
         --build-arg AIRFLOW_BRANCH="${BRANCH_NAME}" \
         --build-arg AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS}" \
+        --build-arg ADDITIONAL_AIRFLOW_EXTRAS="${ADDITIONAL_AIRFLOW_EXTRAS}" \
+        --build-arg ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS}" \
         --build-arg AIRFLOW_CONTAINER_CI_OPTIMISED_BUILD="${AIRFLOW_CONTAINER_CI_OPTIMISED_BUILD}" \
         --build-arg UPGRADE_TO_LATEST_REQUIREMENTS="${UPGRADE_TO_LATEST_REQUIREMENTS}" \
         "${DOCKER_CACHE_CI_DIRECTIVE[@]}" \
@@ -1364,6 +1366,8 @@ function build_prod_image() {
         --build-arg PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION}" \
         --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
         --build-arg AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS}" \
+        --build-arg ADDITIONAL_AIRFLOW_EXTRAS="${ADDITIONAL_AIRFLOW_EXTRAS}" \
+        --build-arg ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS}" \
         "${DOCKER_CACHE_PROD_BUILD_DIRECTIVE[@]}" \
         -t "${AIRFLOW_PROD_BUILD_IMAGE}" \
         --target "airflow-build-image" \
@@ -1372,6 +1376,8 @@ function build_prod_image() {
         "${EXTRA_DOCKER_PROD_BUILD_FLAGS[@]}" \
         --build-arg PYTHON_BASE_IMAGE="${PYTHON_BASE_IMAGE}" \
         --build-arg PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION}" \
+        --build-arg ADDITIONAL_AIRFLOW_EXTRAS="${ADDITIONAL_AIRFLOW_EXTRAS}" \
+        --build-arg ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS}" \
         --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
         --build-arg AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS}" \
         "${DOCKER_CACHE_PROD_DIRECTIVE[@]}" \
@@ -1473,8 +1479,9 @@ function prepare_ci_build() {
     export IMAGE_DESCRIPTION="Airflow CI"
     export AIRFLOW_CONTAINER_CI_OPTIMISED_BUILD="true"
     export AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS:="${DEFAULT_CI_EXTRAS}"}"
+    export ADDITIONAL_AIRFLOW_EXTRAS="${ADDITIONAL_AIRFLOW_EXTRAS:=""}"
+    export ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS:=""}"
     export AIRFLOW_IMAGE="${AIRFLOW_CI_IMAGE}"
-
     go_to_airflow_sources
     fix_group_permissions
 }
@@ -1532,7 +1539,10 @@ function prepare_prod_build() {
     export THE_IMAGE_TYPE="PROD"
     export IMAGE_DESCRIPTION="Airflow production"
     export AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS:="${DEFAULT_PROD_EXTRAS}"}"
+    export ADDITIONAL_AIRFLOW_EXTRAS="${ADDITIONAL_AIRFLOW_EXTRAS:=""}"
+    export ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS:=""}"
     export AIRFLOW_IMAGE="${AIRFLOW_PROD_IMAGE}"
+
     if [[ ${CACHE_REGISTRY=""} != "" ]]; then
         echo "${CACHE_REGISTRY_PASSWORD}" | docker login \
             --username "${CACHE_REGISTRY_USERNAME}" \


[airflow] 01/36: Fix --forward-credentials flag in Breeze (#8554)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 863bed7c924e55136059b5e31de89a10389133b5
Author: Felix Uellendall <fe...@users.noreply.github.com>
AuthorDate: Sat Apr 25 15:51:25 2020 +0200

    Fix --forward-credentials flag in Breeze (#8554)
    
    
    (cherry picked from commit 1ea9fa758a8df8460c9dcd6ad0fe6b1f4d9ee3c3)
---
 TESTING.rst                                       | 1 -
 scripts/ci/docker-compose/forward-credentials.yml | 1 -
 2 files changed, 2 deletions(-)

diff --git a/TESTING.rst b/TESTING.rst
index 76dd589..3c53fdd 100644
--- a/TESTING.rst
+++ b/TESTING.rst
@@ -500,7 +500,6 @@ Currently forwarded credentials are:
   * credentials stored in ``${HOME}/.aws``, ``${HOME}/.boto``, and ``${HOME}/.s3`` (for AWS authentication)
   * credentials stored in ``${HOME}/.docker`` for docker
   * credentials stored in ``${HOME}/.kube`` for kubectl
-  * credentials stored in ``${HOME}/.ssh`` for SSH
 
 
 Adding a New System Test
diff --git a/scripts/ci/docker-compose/forward-credentials.yml b/scripts/ci/docker-compose/forward-credentials.yml
index a417a5b..e2d5f75 100644
--- a/scripts/ci/docker-compose/forward-credentials.yml
+++ b/scripts/ci/docker-compose/forward-credentials.yml
@@ -29,4 +29,3 @@ services:
       - ${HOME}/.gsutil:/root/.gsutil:cached
       - ${HOME}/.kube:/root/.kube:cached
       - ${HOME}/.s3:/root/.s3:cached
-      - ${HOME}/.ssh:/root/.ssh:cached


[airflow] 25/36: Fixes a bug where `build-image` command did not calculate md5 (#9130)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit c34cd6de8b6cc9b925d593e7b51035a6a3126a79
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Thu Jun 4 10:00:14 2020 +0200

    Fixes a bug where `build-image` command did not calculate md5 (#9130)
    
    Even if image was rebuilt, the md5 files were not updated
    
    (cherry picked from commit d6b954d4728de97e9bbd2ba01dd4f5c19c8441bb)
---
 breeze               |  1 +
 scripts/ci/_utils.sh | 35 +++++++++++++++++++----------------
 2 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/breeze b/breeze
index 7fa6e28..18a06fd 100755
--- a/breeze
+++ b/breeze
@@ -1922,6 +1922,7 @@ function run_build_command {
                 build_prod_image
             else
                 prepare_ci_build
+                calculate_md5sum_for_all_files
                 build_ci_image
                 update_all_md5_files
                 build_ci_image_manifest
diff --git a/scripts/ci/_utils.sh b/scripts/ci/_utils.sh
index 2167229..c782ab6 100644
--- a/scripts/ci/_utils.sh
+++ b/scripts/ci/_utils.sh
@@ -347,7 +347,7 @@ function remove_cache_directory() {
 # The md5sum files are stored in .build directory - you can delete this directory
 # If you want to rebuild everything from the scratch
 #
-function check_file_md5sum {
+function calculate_file_md5sum {
     local FILE="${1}"
     local MD5SUM
     local MD5SUM_CACHE_DIR="${BUILD_CACHE_DIR}/${DEFAULT_BRANCH}/${PYTHON_MAJOR_MINOR_VERSION}/${THE_IMAGE_TYPE}"
@@ -408,6 +408,19 @@ function update_all_md5_files() {
     touch "${BUILT_IMAGE_FLAG_FILE}"
 }
 
+
+function calculate_md5sum_for_all_files() {
+    FILES_MODIFIED="false"
+    set +e
+    for FILE in "${FILES_FOR_REBUILD_CHECK[@]}"
+    do
+        if ! calculate_file_md5sum "${AIRFLOW_SOURCES}/${FILE}"; then
+            FILES_MODIFIED="true"
+        fi
+    done
+    set +e
+}
+
 #
 # Checks md5sum of all important files in order to optimise speed of running various operations
 # That mount sources of Airflow to container and require docker image built with latest dependencies.
@@ -433,23 +446,13 @@ function check_if_docker_build_is_needed() {
     print_info
     if [[ ${FORCE_BUILD_IMAGES:=""} == "true" ]]; then
         echo "Docker image build is forced for ${THE_IMAGE_TYPE} image"
-        set +e
-        for FILE in "${FILES_FOR_REBUILD_CHECK[@]}"
-        do
-            # Just store md5sum for all files in md5sum.new - do not check if it is different
-            check_file_md5sum "${AIRFLOW_SOURCES}/${FILE}"
-        done
-        set -e
+        calculate_md5sum_for_all_files
         export NEEDS_DOCKER_BUILD="true"
     else
-        set +e
-        for FILE in "${FILES_FOR_REBUILD_CHECK[@]}"
-        do
-            if ! check_file_md5sum "${AIRFLOW_SOURCES}/${FILE}"; then
-                export NEEDS_DOCKER_BUILD="true"
-            fi
-        done
-        set -e
+        calculate_md5sum_for_all_files
+        if [[ ${FILES_MODIFIED} == "true" ]]; then
+            export NEEDS_DOCKER_BUILD="true"
+        fi
         if [[ ${NEEDS_DOCKER_BUILD} == "true" ]]; then
             echo "Docker image build is needed for ${THE_IMAGE_TYPE} image!"
         else


[airflow] 02/36: Fixed optimistions of non-py-code builds (#8601)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 0d8a238cd8df59d6ba96ae1172618f53bebb3abc
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Tue Apr 28 12:48:30 2020 +0200

    Fixed optimistions of non-py-code builds (#8601)
    
    Count of changed files returned exit code 1 not 0 when there
    was no files matching the pattern, so the whole optimisation
    did not work as intended.
    
    (cherry picked from commit 00bbf324acf543e18571f89a760430b5730d1c40)
---
 scripts/ci/ci_count_changed_files.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/ci/ci_count_changed_files.sh b/scripts/ci/ci_count_changed_files.sh
index 262fb96..0713dc1 100755
--- a/scripts/ci/ci_count_changed_files.sh
+++ b/scripts/ci/ci_count_changed_files.sh
@@ -31,7 +31,7 @@ git remote add target "https://github.com/${CI_TARGET_REPO}"
 
 git fetch target "${CI_TARGET_BRANCH}:${CI_TARGET_BRANCH}" --depth=1
 
-CHANGED_FILES=$(git diff-tree --no-commit-id --name-only -r "${1}" "${CI_TARGET_BRANCH}")
+CHANGED_FILES=$(git diff-tree --no-commit-id --name-only -r "${1}" "${CI_TARGET_BRANCH}" || true)
 
 echo
 echo "Changed files:"
@@ -48,7 +48,7 @@ echo
 echo
 echo "Count changed files matching the ${2} pattern"
 echo
-COUNT_CHANGED_FILES=$(echo "${CHANGED_FILES}" | grep -c "${2}")
+COUNT_CHANGED_FILES=$(echo "${CHANGED_FILES}" | grep -c "${2}" || true)
 echo "${COUNT_CHANGED_FILES}"
 echo
 


[airflow] 15/36: detect incompatible docker server version in breeze (#9042)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit daca50789b6641c4d8d68bdf68d1fdac7b8b367f
Author: QP Hou <qp...@scribd.com>
AuthorDate: Thu May 28 04:50:24 2020 -0700

    detect incompatible docker server version in breeze (#9042)
    
    
    (cherry picked from commit 14e241f05c19208090a96744ed5d7d0b5abff365)
---
 scripts/ci/in_container/check_environment.sh | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/scripts/ci/in_container/check_environment.sh b/scripts/ci/in_container/check_environment.sh
index 0950ef7..a7ed4f8 100755
--- a/scripts/ci/in_container/check_environment.sh
+++ b/scripts/ci/in_container/check_environment.sh
@@ -186,6 +186,12 @@ function resetdb_if_requested() {
     return $?
 }
 
+function check_docker_client_server_version_compatibility() {
+    docker version || exit 1
+}
+
+check_docker_client_server_version_compatibility
+
 if [[ -n ${BACKEND:=} ]]; then
     echo "==============================================================================================="
     echo "             Checking backend: ${BACKEND}"


[airflow] 07/36: The librabbitmq library stopped installing for python3.7 (#8853)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit ffbde530c9778412218f037671a40a2ba1c46f9e
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Wed May 13 11:16:15 2020 +0200

    The librabbitmq library stopped installing for python3.7 (#8853)
    
    When preparing backport relases I found that rabbitmq was not
    included in the "devel_ci" extras. It turned out that librabbitmq was
    not installing in python3.7 and the reason it turned out to be
    that librabbitmq is not maintained for 2 years already and it
    has been replaced by py-amqp library. The pythhon py-amqp
    library has been improved using cython compilation, so it
    became production ready and librabbitmq has been abandoned.
    
    We are switching to the py-amqp library here and adding
    rabbitmq back to "devel_ci" dependencies.
    
    Details in: https://github.com/celery/librabbitmq/issues/153
---
 setup.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/setup.py b/setup.py
index 8c8d9a4..9166279 100644
--- a/setup.py
+++ b/setup.py
@@ -341,7 +341,7 @@ qds = [
     'qds-sdk>=1.10.4',
 ]
 rabbitmq = [
-    'librabbitmq>=1.6.1',
+    'amqp',
 ]
 redis = [
     'redis~=3.2',
@@ -450,7 +450,7 @@ devel_all = (all_dbs + atlas + aws +
              celery + cgroups + crypto + datadog + devel + doc + docker +
              elasticsearch + gcp + grpc + hashicorp + jdbc + jenkins + kerberos + kubernetes + ldap +
              oracle + papermill + password +
-             redis + samba + segment + sendgrid + sentry + slack + snowflake + ssh +
+             rabbitmq + redis + samba + segment + sendgrid + sentry + slack + snowflake + ssh +
              virtualenv + webhdfs + zendesk)
 
 # Snakebite is not Python 3 compatible :'(


[airflow] 31/36: Remove vendored nvd3 and slugify libraries (#9136)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit f16cae37031918feeba227b7e378de98c9fbaebc
Author: Ash Berlin-Taylor <as...@firemirror.com>
AuthorDate: Thu Jun 4 15:45:29 2020 +0100

    Remove vendored nvd3 and slugify libraries (#9136)
    
    We pulled in them because slugify _used_ to default to the GPL'd
    `unidecode` module, but since Slugify 3.0[1] it has used text-unidecode
    by first (and only installs the GPL library by an optional extra, not by
    default) so we can now use it.
    
    This lets us upgreade text-unidecode from 1.2 to the latest 1.3, which
    is the version one of dbt's dependencies needs.
    
    [1]: https://github.com/un33k/python-slugify/blob/4.0.0/CHANGELOG.md#300
    
    (cherry picked from commit 2580b049cf3e2f3990e862a1d0a71f9f6784bd39)
---
 .coveragerc                                        |   1 -
 .flake8                                            |   2 +-
 .pre-commit-config.yaml                            |  27 +-
 .rat-excludes                                      |   5 +-
 LICENSE                                            |   2 +-
 MANIFEST.in                                        |   1 -
 airflow/_vendor/README                             |  13 -
 airflow/_vendor/__init__.py                        |  18 -
 airflow/_vendor/nvd3/LICENSE                       |  24 -
 airflow/_vendor/nvd3/NVD3Chart.py                  | 506 ---------------------
 airflow/_vendor/nvd3/__init__.py                   |  29 --
 airflow/_vendor/nvd3/cumulativeLineChart.py        | 104 -----
 airflow/_vendor/nvd3/discreteBarChart.py           |  91 ----
 airflow/_vendor/nvd3/ipynb.py                      |  91 ----
 airflow/_vendor/nvd3/lineChart.py                  | 120 -----
 airflow/_vendor/nvd3/linePlusBarChart.py           | 131 ------
 airflow/_vendor/nvd3/lineWithFocusChart.py         | 105 -----
 airflow/_vendor/nvd3/multiBarChart.py              |  95 ----
 airflow/_vendor/nvd3/multiBarHorizontalChart.py    | 100 ----
 airflow/_vendor/nvd3/pieChart.py                   | 101 ----
 airflow/_vendor/nvd3/scatterChart.py               | 121 -----
 airflow/_vendor/nvd3/stackedAreaChart.py           |  99 ----
 airflow/_vendor/nvd3/templates/base.html           |  35 --
 airflow/_vendor/nvd3/templates/content.html        | 123 -----
 .../nvd3/templates/cumulativelinechart.html        |  10 -
 .../_vendor/nvd3/templates/discretebarchart.html   |  31 --
 .../_vendor/nvd3/templates/linebarwfocuschart.html |  60 ---
 airflow/_vendor/nvd3/templates/linechart.html      |  46 --
 .../_vendor/nvd3/templates/lineplusbarchart.html   |  43 --
 .../_vendor/nvd3/templates/linewfocuschart.html    |  10 -
 airflow/_vendor/nvd3/templates/multibarchart.html  |  10 -
 .../nvd3/templates/multibarcharthorizontal.html    |  10 -
 airflow/_vendor/nvd3/templates/page.html           |  12 -
 airflow/_vendor/nvd3/templates/piechart.html       |  80 ----
 airflow/_vendor/nvd3/templates/scatterchart.html   |  52 ---
 .../_vendor/nvd3/templates/stackedareachart.html   |   7 -
 airflow/_vendor/nvd3/translator.py                 |  71 ---
 airflow/_vendor/slugify/LICENSE                    |  21 -
 airflow/_vendor/slugify/__init__.py                |   6 -
 airflow/_vendor/slugify/slugify.py                 | 185 --------
 airflow/www/views.py                               |   7 +-
 requirements/requirements-python3.6.txt            |   4 +-
 requirements/requirements-python3.7.txt            |   4 +-
 scripts/ci/in_container/_in_container_utils.sh     |   2 +-
 setup.cfg                                          |  28 +-
 setup.py                                           |   3 +-
 46 files changed, 53 insertions(+), 2593 deletions(-)

diff --git a/.coveragerc b/.coveragerc
index 9a07159..8380515 100644
--- a/.coveragerc
+++ b/.coveragerc
@@ -24,4 +24,3 @@ omit =
     airflow/migrations/*
     airflow/www/node_modules/**
     airflow/www_rbac/node_modules/**
-    airflow/_vendor/*
diff --git a/.flake8 b/.flake8
index 4a2ca5b..099ff70 100644
--- a/.flake8
+++ b/.flake8
@@ -1,5 +1,5 @@
 [flake8]
 max-line-length = 110
 ignore = E731,W504,I001,W503
-exclude = .svn,CVS,.bzr,.hg,.git,__pycache__,.eggs,*.egg,*/_vendor/*,node_modules
+exclude = .svn,CVS,.bzr,.hg,.git,__pycache__,.eggs,*.egg,node_modules
 format = ${cyan}%(path)s${reset}:${yellow_bold}%(row)d${reset}:${green_bold}%(col)d${reset}: ${red_bold}%(code)s${reset} %(text)s
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 68ddcd5..fe543d1 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -25,11 +25,11 @@ repos:
     rev: v1.1.7
     hooks:
       - id: forbid-tabs
-        exclude: ^airflow/_vendor/.*$|^docs/Makefile$
+        exclude: ^docs/Makefile$
       - id: insert-license
         name: Add license for all SQL files
         files: \.sql$
-        exclude: ^\.github/.*$|^airflow/_vendor/.*$
+        exclude: ^\.github/.*$
         args:
           - --comment-style
           - "/*||*/"
@@ -38,7 +38,7 @@ repos:
           - --fuzzy-match-generates-todo
       - id: insert-license
         name: Add license for all other files
-        exclude: ^\.github/.*$"|^airflow/_vendor/.*$
+        exclude: ^\.github/.*$
         args:
           - --comment-style
           - "|#|"
@@ -49,7 +49,7 @@ repos:
           \.properties$|\.cfg$|\.conf$|\.ini$|\.ldif$|\.readthedocs$|\.service$|^Dockerfile.*$
       - id: insert-license
         name: Add license for all rst files
-        exclude: ^\.github/.*$"|^airflow/_vendor/.*$
+        exclude: ^\.github/.*$
         args:
           - --comment-style
           - "||"
@@ -60,7 +60,7 @@ repos:
       - id: insert-license
         name: Add license for all JS/CSS files
         files: \.(js|css)$
-        exclude: ^\.github/.*$|^airflow/_vendor/.*$|^airflow/www/static/.*|^airflow/www_rbac/static/.*$
+        exclude: ^\.github/.*$
         args:
           - --comment-style
           - "/**| *| */"
@@ -70,7 +70,7 @@ repos:
       - id: insert-license
         name: Add license for all JINJA template files
         files: ^airflow/www/templates/.*\.html$|^docs/templates/.*\.html$|^airflow/contrib/plugins/metastore_browser/templates/.*\.html$  # yamllint disable-line rule:line-length
-        exclude: ^\.github/.*$|^airflow/_vendor/.*$
+        exclude: ^\.github/.*$
         args:
           - --comment-style
           - "{#||#}"
@@ -79,7 +79,7 @@ repos:
           - --fuzzy-match-generates-todo
       - id: insert-license
         name: Add license for all shell files
-        exclude: ^\.github/.*$"|^airflow/_vendor/.*$
+        exclude: ^\.github/.*$
         files: ^breeze$|^breeze-complete$|\.sh$|\.bash$|\.bats$
         args:
           - --comment-style
@@ -89,7 +89,7 @@ repos:
           - --fuzzy-match-generates-todo
       - id: insert-license
         name: Add license for all python files
-        exclude: ^\.github/.*$"|^airflow/_vendor/.*$
+        exclude: ^\.github/.*$
         types: [python]
         args:
           - --comment-style
@@ -99,7 +99,7 @@ repos:
           - --fuzzy-match-generates-todo
       - id: insert-license
         name: Add license for all XML files
-        exclude: ^\.github/.*$"|^airflow/_vendor/.*$
+        exclude: ^\.github/.*$
         types: [xml]
         args:
           - --comment-style
@@ -109,7 +109,7 @@ repos:
           - --fuzzy-match-generates-todo
       - id: insert-license
         name: Add license for all yaml files
-        exclude: ^\.github/.*$"|^airflow/_vendor/.*$
+        exclude: ^\.github/.*$
         types: [yaml]
         args:
           - --comment-style
@@ -120,7 +120,7 @@ repos:
       - id: insert-license
         name: Add license for all md files
         files: \.md$
-        exclude: ^\.github/.*$|^airflow/_vendor/.*$
+        exclude: ^\.github/.*$
         args:
           - --comment-style
           - "<!--|| -->"
@@ -169,7 +169,6 @@ repos:
         language: docker_image
         entry: koalaman/shellcheck:stable -x -a
         files: ^breeze$|^breeze-complete$|\.sh$|^hooks/build$|^hooks/push$|\.bash$|\.bats$
-        exclude: ^airflow/_vendor/.*$
         ##
         ## Dear committer.
         ##
@@ -240,7 +239,7 @@ repos:
                 ^\sdef\s*\S*\(.*\):\s*\-\>\s*\S*.*  # Matches -> return value syntax from Python3
             )$
         files: \.py$
-        exclude: ^airflow/_vendor|^dev/
+        exclude: ^dev/
         pass_filenames: true
       - id: python2-compile
         name: Compile code using python2
@@ -288,7 +287,7 @@ repos:
         language: system
         entry: "./scripts/ci/pre_commit_mypy.sh"
         files: \.py$
-        exclude: ^airflow/_vendor/.*$|^dev
+        exclude: ^dev
         require_serial: true
       - id: flake8
         name: Run flake8
diff --git a/.rat-excludes b/.rat-excludes
index 906d804..f50f7d3 100644
--- a/.rat-excludes
+++ b/.rat-excludes
@@ -74,11 +74,12 @@ apache-airflow-.*\+source.tar.gz.*
 apache-airflow-.*\+bin.tar.gz.*
 PULL_REQUEST_TEMPLATE.md
 
-# vendored modules
-_vendor/*
 
 # Locally mounted files
 .*egg-info/*
 .bash_history
 .bash_aliases
 .inputrc
+
+# the example notebook is ASF 2 licensed but RAT cannot read this
+input_notebook.ipynb
diff --git a/LICENSE b/LICENSE
index 4cac7bd..a0663a6 100644
--- a/LICENSE
+++ b/LICENSE
@@ -241,7 +241,7 @@ The text of each license is also included at licenses/LICENSE-[project].txt.
     (MIT License) ElasticMock v1.3.2 (https://github.com/vrcmarcos/elasticmock)
     (MIT License) MomentJS v2.22.2 (http://momentjs.com/)
     (MIT License) moment-strftime v0.5.0 (https://github.com/benjaminoakes/moment-strftime)
-    (MIT License) python-slugify v2.0.1 (https://github.com/un33k/python-slugify)
+    (MIT License) python-slugify v4.0.0 (https://github.com/un33k/python-slugify)
     (MIT License) python-nvd3 v0.15.0 (https://github.com/areski/python-nvd3)
     (MIT License) eonasdan-bootstrap-datetimepicker v4.17.37 (https://github.com/eonasdan/bootstrap-datetimepicker/)
 
diff --git a/MANIFEST.in b/MANIFEST.in
index e4afd00..dae0f3e 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -27,7 +27,6 @@ graft airflow/www_rbac
 graft airflow/www_rbac/static
 graft airflow/www_rbac/templates
 graft airflow/www_rbac/translations
-graft airflow/_vendor/
 include airflow/alembic.ini
 include airflow/git_version
 include airflow/serialization/schema.json
diff --git a/airflow/_vendor/README b/airflow/_vendor/README
deleted file mode 100644
index a79ea89..0000000
--- a/airflow/_vendor/README
+++ /dev/null
@@ -1,13 +0,0 @@
-Original files in this directory were created with the following commands::
-
-    mkdir -p slugify/
-    curl -fsSL -O https://files.pythonhosted.org/packages/1f/9c/8b07d625e9c9df567986d887f0375075abb1923e49d074a7803cd1527dae/python-slugify-2.0.1.tar.gz
-    tar -xzf python-slugify-*.tar.gz --strip-components=2 -C slugify/ '*/slugify/*'
-    tar -xzf python-slugify-*.tar.gz --strip-components=1 -C slugify/ '*/LICENSE'
-    rm *.tar.gz
-
-    mkdir -p nvd3/
-    curl -fsSL -O https://files.pythonhosted.org/packages/0b/aa/97165daa6e319409c5c2582e62736a7353bda3c90d90fdcb0b11e116dd2d/python-nvd3-0.15.0.tar.gz
-    tar -xzf python-nvd3-*.tar.gz --strip-components=2 -C nvd3/ '*/nvd3/*'
-    tar -xzf python-nvd3-*.tar.gz --strip-components=1 -C nvd3/ '*/LICENSE'
-    rm *.tar.gz
diff --git a/airflow/_vendor/__init__.py b/airflow/_vendor/__init__.py
deleted file mode 100644
index 114d189..0000000
--- a/airflow/_vendor/__init__.py
+++ /dev/null
@@ -1,18 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#   http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied.  See the License for the
-# specific language governing permissions and limitations
-# under the License.
diff --git a/airflow/_vendor/nvd3/LICENSE b/airflow/_vendor/nvd3/LICENSE
deleted file mode 100644
index 1add624..0000000
--- a/airflow/_vendor/nvd3/LICENSE
+++ /dev/null
@@ -1,24 +0,0 @@
-The MIT License (MIT)
-
-Python-nvd3
-
-Copyright (c) 2013 Arezqui Belaid <ar...@gmail.com> and other contributors
-
-Permission is hereby granted, free of charge, to any person obtaining
-a copy of this software and associated documentation files (the
-"Software"), to deal in the Software without restriction, including
-without limitation the rights to use, copy, modify, merge, publish,
-distribute, sublicense, and/or sell copies of the Software, and to
-permit persons to whom the Software is furnished to do so, subject to
-the following conditions:
-
-The above copyright notice and this permission notice shall be
-included in all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
-LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
-OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
-WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
diff --git a/airflow/_vendor/nvd3/NVD3Chart.py b/airflow/_vendor/nvd3/NVD3Chart.py
deleted file mode 100644
index faefe5d..0000000
--- a/airflow/_vendor/nvd3/NVD3Chart.py
+++ /dev/null
@@ -1,506 +0,0 @@
-#!/usr/bin/python
-# -*- coding: utf-8 -*-
-
-"""
-Python-nvd3 is a Python wrapper for NVD3 graph library.
-NVD3 is an attempt to build re-usable charts and chart components
-for d3.js without taking away the power that d3.js gives you.
-
-Project location : https://github.com/areski/python-nvd3
-"""
-
-from __future__ import unicode_literals
-from optparse import OptionParser
-from jinja2 import Environment, PackageLoader
-from airflow._vendor.slugify import slugify
-
-try:
-    import simplejson as json
-except ImportError:
-    import json
-
-CONTENT_FILENAME = "./content.html"
-PAGE_FILENAME = "./page.html"
-
-
-pl = PackageLoader('airflow._vendor.nvd3', 'templates')
-jinja2_env = Environment(lstrip_blocks=True, trim_blocks=True, loader=pl)
-
-template_content = jinja2_env.get_template(CONTENT_FILENAME)
-template_page = jinja2_env.get_template(PAGE_FILENAME)
-
-
-def stab(tab=1):
-    """
-    create space tabulation
-    """
-    return ' ' * 4 * tab
-
-
-class NVD3Chart(object):
-    """
-    NVD3Chart Base class.
-    """
-    #: chart count
-    count = 0
-    #:  directory holding the assets (bower_components)
-    assets_directory = './bower_components/'
-
-    # this attribute is overriden by children of this
-    # class
-    CHART_FILENAME = None
-    template_environment = Environment(lstrip_blocks=True, trim_blocks=True,
-                                       loader=pl)
-
-    def __init__(self, **kwargs):
-        """
-        This is the base class for all the charts. The following keywords are
-        accepted:
-
-        :keyword: **display_container** - default: ``True``
-        :keyword: **jquery_on_ready** - default: ``False``
-        :keyword: **charttooltip_dateformat** - default: ``'%d %b %Y'``
-        :keyword: **name** - default: the class name
-                    ``model`` - set the model (e.g. ``pieChart``, `
-                    ``LineWithFocusChart``, ``MultiBarChart``).
-        :keyword: **color_category** - default - ``None``
-        :keyword: **color_list** - default - ``None``
-                  used by pieChart (e.g. ``['red', 'blue', 'orange']``)
-        :keyword: **margin_bottom** - default - ``20``
-        :keyword: **margin_left** - default - ``60``
-        :keyword: **margin_right** - default - ``60``
-        :keyword: **margin_top** - default - ``30``
-        :keyword: **height** - default - ``''``
-        :keyword: **width** - default - ``''``
-        :keyword: **stacked** - default - ``False``
-        :keyword: **focus_enable** - default - ``False``
-        :keyword: **resize** - define - ``False``
-        :keyword: **show_legend** - default - ``True``
-        :keyword: **show_labels** - default - ``True``
-        :keyword: **tag_script_js** - default - ``True``
-        :keyword: **use_interactive_guideline** - default - ``False``
-        :keyword: **chart_attr** - default - ``None``
-        :keyword: **extras** - default - ``None``
-
-            Extra chart modifiers. Use this to modify different attributes of
-            the chart.
-        :keyword: **x_axis_date** - default - False
-            Signal that x axis is a date axis
-        :keyword: **date_format** - default - ``%x``
-                  see https://github.com/mbostock/d3/wiki/Time-Formatting
-        :keyword: **x_axis_format** - default - ``''``.
-        :keyword: **y_axis_format** - default - ``''``.
-        :keyword: **style** - default - ``''``
-            Style modifiers for the DIV container.
-        :keyword: **color_category** - default - ``category10``
-
-            Acceptable values are nvd3 categories such as
-            ``category10``, ``category20``, ``category20c``.
-        """
-        # set the model
-        self.model = self.__class__.__name__  #: The chart model,
-
-        #: an Instance of Jinja2 template
-        self.template_page_nvd3 = template_page
-        self.template_content_nvd3 = template_content
-        self.series = []
-        self.axislist = {}
-        # accepted keywords
-        self.display_container = kwargs.get('display_container', True)
-        self.charttooltip_dateformat = kwargs.get('charttooltip_dateformat',
-                                                  '%d %b %Y')
-        self._slugify_name(kwargs.get('name', self.model))
-        self.jquery_on_ready = kwargs.get('jquery_on_ready', False)
-        self.color_category = kwargs.get('color_category', None)
-        self.color_list = kwargs.get('color_list', None)
-        self.margin_bottom = kwargs.get('margin_bottom', 20)
-        self.margin_left = kwargs.get('margin_left', 60)
-        self.margin_right = kwargs.get('margin_right', 60)
-        self.margin_top = kwargs.get('margin_top', 30)
-        self.height = kwargs.get('height', '')
-        self.width = kwargs.get('width', '')
-        self.stacked = kwargs.get('stacked', False)
-        self.focus_enable = kwargs.get('focus_enable', False)
-        self.resize = kwargs.get('resize', False)
-        self.show_legend = kwargs.get('show_legend', True)
-        self.show_labels = kwargs.get('show_labels', True)
-        self.tag_script_js = kwargs.get('tag_script_js', True)
-        self.use_interactive_guideline = kwargs.get("use_interactive_guideline",
-                                                    False)
-        self.chart_attr = kwargs.get("chart_attr", {})
-        self.extras = kwargs.get('extras', None)
-        self.style = kwargs.get('style', '')
-        self.date_format = kwargs.get('date_format', '%x')
-        self.x_axis_date = kwargs.get('x_axis_date', False)
-        #: x-axis contain date format or not
-        # possible duplicate of x_axis_date
-        self.date_flag = kwargs.get('date_flag', False)
-        self.x_axis_format = kwargs.get('x_axis_format', '')
-        # Load remote JS assets or use the local bower assets?
-        self.remote_js_assets = kwargs.get('remote_js_assets', True)
-
-        # None keywords attribute that should be modified by methods
-        # We should change all these to _attr
-
-        self.htmlcontent = ''  #: written by buildhtml
-        self.htmlheader = ''
-        #: Place holder for the graph (the HTML div)
-        #: Written by ``buildcontainer``
-        self.container = u''
-        #: Header for javascript code
-        self.containerheader = u''
-        # CDN http://cdnjs.com/libraries/nvd3/ needs to make sure it's up to
-        # date
-        self.header_css = [
-            '<link href="%s" rel="stylesheet" />' % h for h in
-            (
-                'https://cdnjs.cloudflare.com/ajax/libs/nvd3/1.7.0/nv.d3.min.css' if self.remote_js_assets else self.assets_directory + 'nvd3/src/nv.d3.css',
-            )
-        ]
-
-        self.header_js = [
-            '<script src="%s"></script>' % h for h in
-            (
-                'https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js' if self.remote_js_assets else self.assets_directory + 'd3/d3.min.js',
-                'https://cdnjs.cloudflare.com/ajax/libs/nvd3/1.7.0/nv.d3.min.js' if self.remote_js_assets else self.assets_directory + 'nvd3/nv.d3.min.js'
-            )
-        ]
-
-        #: Javascript code as string
-        self.jschart = None
-        self.custom_tooltip_flag = False
-        self.tooltip_condition_string = ''
-        self.charttooltip = ''
-        self.serie_no = 1
-
-    def _slugify_name(self, name):
-        """Slufigy name with underscore"""
-        self.name = slugify(name, separator='_')
-
-    def add_serie(self, y, x, name=None, extra=None, **kwargs):
-        """
-        add serie - Series are list of data that will be plotted
-        y {1, 2, 3, 4, 5} / x {1, 2, 3, 4, 5}
-
-        **Attributes**:
-
-            * ``name`` - set Serie name
-            * ``x`` - x-axis data
-            * ``y`` - y-axis data
-
-            kwargs:
-
-            * ``shape`` - for scatterChart, you can set different shapes
-                          (circle, triangle etc...)
-            * ``size`` - for scatterChart, you can set size of different shapes
-            * ``type`` - for multiChart, type should be bar
-            * ``bar`` - to display bars in Chart
-            * ``color_list`` - define list of colors which will be
-                               used by pieChart
-            * ``color`` - set axis color
-            * ``disabled`` -
-
-            extra:
-
-            * ``tooltip`` - set tooltip flag
-            * ``date_format`` - set date_format for tooltip if x-axis is in
-              date format
-
-        """
-        if not name:
-            name = "Serie %d" % (self.serie_no)
-
-        # For scatterChart shape & size fields are added in serie
-        if 'shape' in kwargs or 'size' in kwargs:
-            csize = kwargs.get('size', 1)
-            cshape = kwargs.get('shape', 'circle')
-
-            serie = [{
-                'x': x[i],
-                'y': j,
-                'shape': cshape,
-                'size': csize[i] if isinstance(csize, list) else csize
-            } for i, j in enumerate(y)]
-        else:
-            if self.model == 'pieChart':
-                serie = [{'label': x[i], 'value': y} for i, y in enumerate(y)]
-            else:
-                serie = [{'x': x[i], 'y': y} for i, y in enumerate(y)]
-
-        data_keyvalue = {'values': serie, 'key': name}
-
-        # multiChart
-        # Histogram type='bar' for the series
-        if 'type' in kwargs and kwargs['type']:
-            data_keyvalue['type'] = kwargs['type']
-
-        # Define on which Y axis the serie is related
-        # a chart can have 2 Y axis, left and right, by default only one Y Axis is used
-        if 'yaxis' in kwargs and kwargs['yaxis']:
-            data_keyvalue['yAxis'] = kwargs['yaxis']
-        else:
-            if self.model != 'pieChart':
-                data_keyvalue['yAxis'] = '1'
-
-        if 'bar' in kwargs and kwargs['bar']:
-            data_keyvalue['bar'] = 'true'
-
-        if 'disabled' in kwargs and kwargs['disabled']:
-            data_keyvalue['disabled'] = 'true'
-
-        if 'color' in kwargs and kwargs['color']:
-            data_keyvalue['color'] = kwargs['color']
-
-        if extra:
-            if self.model == 'pieChart':
-                if 'color_list' in extra and extra['color_list']:
-                    self.color_list = extra['color_list']
-
-            if extra.get('date_format'):
-                self.charttooltip_dateformat = extra['date_format']
-
-            if extra.get('tooltip'):
-                self.custom_tooltip_flag = True
-
-                if self.model != 'pieChart':
-                    _start = extra['tooltip']['y_start']
-                    _end = extra['tooltip']['y_end']
-                    _start = ("'" + str(_start) + "' + ") if _start else ''
-                    _end = (" + '" + str(_end) + "'") if _end else ''
-
-                    if self.model == 'linePlusBarChart':
-                        if self.tooltip_condition_string:
-                            self.tooltip_condition_string += stab(5)
-                        self.tooltip_condition_string += stab(0) + "if(key.indexOf('" + name + "') > -1 ){\n" +\
-                            stab(6) + "var y = " + _start + " String(graph.point.y) " + _end + ";\n" +\
-                            stab(5) + "}\n"
-                    elif self.model == 'cumulativeLineChart':
-                        self.tooltip_condition_string += stab(0) + "if(key == '" + name + "'){\n" +\
-                            stab(6) + "var y = " + _start + " String(e) " + _end + ";\n" +\
-                            stab(5) + "}\n"
-                    else:
-                        self.tooltip_condition_string += stab(5) + "if(key == '" + name + "'){\n" +\
-                            stab(6) + "var y = " + _start + " String(graph.point.y) " + _end + ";\n" +\
-                            stab(5) + "}\n"
-
-                if self.model == 'pieChart':
-                    _start = extra['tooltip']['y_start']
-                    _end = extra['tooltip']['y_end']
-                    _start = ("'" + str(_start) + "' + ") if _start else ''
-                    _end = (" + '" + str(_end) + "'") if _end else ''
-                    self.tooltip_condition_string += "var y = " + _start + " String(y) " + _end + ";\n"
-
-        # Increment series counter & append
-        self.serie_no += 1
-        self.series.append(data_keyvalue)
-
-    def add_chart_extras(self, extras):
-        """
-        Use this method to add extra d3 properties to your chart.
-        For example, you want to change the text color of the graph::
-
-            chart = pieChart(name='pieChart', color_category='category20c', height=400, width=400)
-
-            xdata = ["Orange", "Banana", "Pear", "Kiwi", "Apple", "Strawberry", "Pineapple"]
-            ydata = [3, 4, 0, 1, 5, 7, 3]
-
-            extra_serie = {"tooltip": {"y_start": "", "y_end": " cal"}}
-            chart.add_serie(y=ydata, x=xdata, extra=extra_serie)
-
-        The above code will create graph with a black text, the following will change it::
-
-            text_white="d3.selectAll('#pieChart text').style('fill', 'white');"
-            chart.add_chart_extras(text_white)
-
-        The above extras will be appended to the java script generated.
-
-        Alternatively, you can use the following initialization::
-
-            chart = pieChart(name='pieChart',
-                             color_category='category20c',
-                             height=400, width=400,
-                             extras=text_white)
-        """
-        self.extras = extras
-
-    def set_graph_height(self, height):
-        """Set Graph height"""
-        self.height = str(height)
-
-    def set_graph_width(self, width):
-        """Set Graph width"""
-        self.width = str(width)
-
-    def set_containerheader(self, containerheader):
-        """Set containerheader"""
-        self.containerheader = containerheader
-
-    def set_date_flag(self, date_flag=False):
-        """Set date flag"""
-        self.date_flag = date_flag
-
-    def set_custom_tooltip_flag(self, custom_tooltip_flag):
-        """Set custom_tooltip_flag & date_flag"""
-        self.custom_tooltip_flag = custom_tooltip_flag
-
-    def __str__(self):
-        """return htmlcontent"""
-        self.buildhtml()
-        return self.htmlcontent
-
-    def buildcontent(self):
-        """Build HTML content only, no header or body tags. To be useful this
-        will usually require the attribute `juqery_on_ready` to be set which
-        will wrap the js in $(function(){<regular_js>};)
-        """
-        self.buildcontainer()
-        # if the subclass has a method buildjs this method will be
-        # called instead of the method defined here
-        # when this subclass method is entered it does call
-        # the method buildjschart defined here
-        self.buildjschart()
-        self.htmlcontent = self.template_content_nvd3.render(chart=self)
-
-    def buildhtml(self):
-        """Build the HTML page
-        Create the htmlheader with css / js
-        Create html page
-        Add Js code for nvd3
-        """
-        self.buildcontent()
-        self.content = self.htmlcontent
-        self.htmlcontent = self.template_page_nvd3.render(chart=self)
-
-    # this is used by django-nvd3
-    def buildhtmlheader(self):
-        """generate HTML header content"""
-        self.htmlheader = ''
-        # If the JavaScript assets have already been injected, don't bother re-sourcing them.
-        global _js_initialized
-        if '_js_initialized' not in globals() or not _js_initialized:
-            for css in self.header_css:
-                self.htmlheader += css
-            for js in self.header_js:
-                self.htmlheader += js
-
-    def buildcontainer(self):
-        """generate HTML div"""
-        if self.container:
-            return
-
-        # Create SVG div with style
-        if self.width:
-            if self.width[-1] != '%':
-                self.style += 'width:%spx;' % self.width
-            else:
-                self.style += 'width:%s;' % self.width
-        if self.height:
-            if self.height[-1] != '%':
-                self.style += 'height:%spx;' % self.height
-            else:
-                self.style += 'height:%s;' % self.height
-        if self.style:
-            self.style = 'style="%s"' % self.style
-
-        self.container = self.containerheader + \
-            '<div id="%s"><svg %s></svg></div>\n' % (self.name, self.style)
-
-    def buildjschart(self):
-        """generate javascript code for the chart"""
-        self.jschart = ''
-
-        # add custom tooltip string in jschart
-        # default condition (if build_custom_tooltip is not called explicitly with date_flag=True)
-        if self.tooltip_condition_string == '':
-            self.tooltip_condition_string = 'var y = String(graph.point.y);\n'
-
-        # Include data
-        self.series_js = json.dumps(self.series)
-
-    def create_x_axis(self, name, label=None, format=None, date=False, custom_format=False):
-        """Create X-axis"""
-        axis = {}
-        if custom_format and format:
-            axis['tickFormat'] = format
-        elif format:
-            if format == 'AM_PM':
-                axis['tickFormat'] = "function(d) { return get_am_pm(parseInt(d)); }"
-            else:
-                axis['tickFormat'] = "d3.format(',%s')" % format
-
-        if label:
-            axis['axisLabel'] = "'" + label + "'"
-
-        # date format : see https://github.com/mbostock/d3/wiki/Time-Formatting
-        if date:
-            self.dateformat = format
-            axis['tickFormat'] = ("function(d) { return d3.time.format('%s')"
-                                  "(new Date(parseInt(d))) }\n"
-                                  "" % self.dateformat)
-            # flag is the x Axis is a date
-            if name[0] == 'x':
-                self.x_axis_date = True
-
-        # Add new axis to list of axis
-        self.axislist[name] = axis
-
-        # Create x2Axis if focus_enable
-        if name == "xAxis" and self.focus_enable:
-            self.axislist['x2Axis'] = axis
-
-    def create_y_axis(self, name, label=None, format=None, custom_format=False):
-        """
-        Create Y-axis
-        """
-        axis = {}
-
-        if custom_format and format:
-            axis['tickFormat'] = format
-        elif format:
-            axis['tickFormat'] = "d3.format(',%s')" % format
-
-        if label:
-            axis['axisLabel'] = "'" + label + "'"
-
-        # Add new axis to list of axis
-        self.axislist[name] = axis
-
-
-class TemplateMixin(object):
-    """
-    A mixin that override buildcontent. Instead of building the complex
-    content template we exploit Jinja2 inheritance. Thus each chart class
-    renders it's own chart template which inherits from content.html
-    """
-    def buildcontent(self):
-        """Build HTML content only, no header or body tags. To be useful this
-        will usually require the attribute `juqery_on_ready` to be set which
-        will wrap the js in $(function(){<regular_js>};)
-        """
-        self.buildcontainer()
-        # if the subclass has a method buildjs this method will be
-        # called instead of the method defined here
-        # when this subclass method is entered it does call
-        # the method buildjschart defined here
-        self.buildjschart()
-        self.htmlcontent = self.template_chart_nvd3.render(chart=self)
-
-
-def _main():
-    """
-    Parse options and process commands
-    """
-    # Parse arguments
-    usage = "usage: nvd3.py [options]"
-    parser = OptionParser(usage=usage,
-                          version=("python-nvd3 - Charts generator with "
-                                   "nvd3.js and d3.js"))
-    parser.add_option("-q", "--quiet",
-                      action="store_false", dest="verbose", default=True,
-                      help="don't print messages to stdout")
-
-    (options, args) = parser.parse_args()
-
-
-if __name__ == '__main__':
-    _main()
diff --git a/airflow/_vendor/nvd3/__init__.py b/airflow/_vendor/nvd3/__init__.py
deleted file mode 100755
index 5b737b4..0000000
--- a/airflow/_vendor/nvd3/__init__.py
+++ /dev/null
@@ -1,29 +0,0 @@
-#!/usr/bin/python
-# -*- coding: utf-8 -*-
-
-"""
-Python-nvd3 is a Python wrapper for NVD3 graph library.
-NVD3 is an attempt to build re-usable charts and chart components
-for d3.js without taking away the power that d3.js gives you.
-
-Project location : https://github.com/areski/python-nvd3
-"""
-
-__version__ = '0.15.0'
-__all__ = ['lineChart', 'pieChart', 'lineWithFocusChart',
-           'stackedAreaChart', 'multiBarHorizontalChart',
-           'linePlusBarChart', 'cumulativeLineChart',
-           'scatterChart', 'discreteBarChart', 'multiBarChart']
-
-
-from .lineChart import lineChart
-from .pieChart import pieChart
-from .lineWithFocusChart import lineWithFocusChart
-from .stackedAreaChart import stackedAreaChart
-from .multiBarHorizontalChart import multiBarHorizontalChart
-from .linePlusBarChart import linePlusBarChart
-from .cumulativeLineChart import cumulativeLineChart
-from .scatterChart import scatterChart
-from .discreteBarChart import discreteBarChart
-from .multiBarChart import multiBarChart
-from . import ipynb
diff --git a/airflow/_vendor/nvd3/cumulativeLineChart.py b/airflow/_vendor/nvd3/cumulativeLineChart.py
deleted file mode 100644
index d98d086..0000000
--- a/airflow/_vendor/nvd3/cumulativeLineChart.py
+++ /dev/null
@@ -1,104 +0,0 @@
-#!/usr/bin/python
-# -*- coding: utf-8 -*-
-
-"""
-Python-nvd3 is a Python wrapper for NVD3 graph library.
-NVD3 is an attempt to build re-usable charts and chart components
-for d3.js without taking away the power that d3.js gives you.
-
-Project location : https://github.com/areski/python-nvd3
-"""
-
-from .NVD3Chart import NVD3Chart, TemplateMixin
-
-
-class cumulativeLineChart(TemplateMixin, NVD3Chart):
-    """
-    A cumulative line chart is used when you have one important grouping representing
-    an ordered set of data and one value to show, summed over time.
-
-    Python example::
-
-        from nvd3 import cumulativeLineChart
-        chart = cumulativeLineChart(name='cumulativeLineChart', x_is_date=True)
-        xdata = [1365026400000000, 1365026500000000, 1365026600000000]
-        ydata = [6, 5, 1]
-        y2data = [36, 55, 11]
-
-        extra_serie = {"tooltip": {"y_start": "There are ", "y_end": " calls"}}
-        chart.add_serie(name="Serie 1", y=ydata, x=xdata, extra=extra_serie)
-
-        extra_serie = {"tooltip": {"y_start": "", "y_end": " mins"}}
-        chart.add_serie(name="Serie 2", y=y2data, x=xdata, extra=extra_serie)
-        chart.buildhtml()
-
-    Javascript generated:
-
-    .. raw:: html
-
-        <div id="cumulativeLineChart"><svg style="height:450px; width:100%"></svg></div>
-        <script>
-            data_cumulativeLineChart=[{"values": [{"y": 6, "x": 1365026400000000},
-            {"y": 5, "x": 1365026500000000},
-            {"y": 1, "x": 1365026600000000}],
-            "key": "Serie 1", "yAxis": "1"},
-            {"values": [{"y": 36, "x": 1365026400000000},
-            {"y": 55, "x": 1365026500000000},
-            {"y": 11, "x": 1365026600000000}], "key": "Serie 2", "yAxis": "1"}];
-            nv.addGraph(function() {
-                var chart = nv.models.cumulativeLineChart();
-                chart.margin({top: 30, right: 60, bottom: 20, left: 60});
-                var datum = data_cumulativeLineChart;
-
-                        chart.xAxis
-                            .tickFormat(function(d) { return d3.time.format('%d %b %Y')(new Date(parseInt(d))) });
-                        chart.yAxis
-                            .tickFormat(d3.format(',.1%'));
-
-                    chart.tooltipContent(function(key, y, e, graph) {
-                        var x = d3.time.format("%d %b %Y")(new Date(parseInt(graph.point.x)));
-                        var y = String(graph.point.y);
-                        if(key == 'Serie 1'){
-                                var y = 'There are ' +  String(e)  + ' calls';
-                            }if(key == 'Serie 2'){
-                                var y =  String(e)  + ' mins';
-                            }
-                        tooltip_str = '<center><b>'+key+'</b></center>' + y + ' on ' + x;
-                        return tooltip_str;
-                    });
-                    chart.showLegend(true);
-
-                d3.select('#cumulativeLineChart svg')
-                    .datum(datum)
-                    .transition().duration(500)
-                    .attr('height', 450)
-                    .call(chart); });
-        </script>
-
-    """
-
-    CHART_FILENAME = "./cumulativelinechart.html"
-    template_chart_nvd3 = NVD3Chart.template_environment.get_template(CHART_FILENAME)
-
-    def __init__(self, **kwargs):
-        super(cumulativeLineChart, self).__init__(**kwargs)
-        self.model = 'cumulativeLineChart'
-
-        height = kwargs.get('height', 450)
-        width = kwargs.get('width', None)
-
-        if kwargs.get('x_is_date', False):
-            self.set_date_flag(True)
-            self.create_x_axis('xAxis',
-                               format=kwargs.get('x_axis_format', '%d %b %Y'),
-                               date=True)
-            self.set_custom_tooltip_flag(True)
-        else:
-            self.create_x_axis('xAxis', format=kwargs.get(
-                'x_axis_format', '.2f'))
-
-        self.create_y_axis('yAxis', format=kwargs.get('y_axis_format', '.1%'))
-
-        self.set_graph_height(height)
-        if width:
-            self.set_graph_width(width)
diff --git a/airflow/_vendor/nvd3/discreteBarChart.py b/airflow/_vendor/nvd3/discreteBarChart.py
deleted file mode 100644
index cf6c8a4..0000000
--- a/airflow/_vendor/nvd3/discreteBarChart.py
+++ /dev/null
@@ -1,91 +0,0 @@
-#!/usr/bin/python
-# -*- coding: utf-8 -*-
-
-"""
-Python-nvd3 is a Python wrapper for NVD3 graph library.
-NVD3 is an attempt to build re-usable charts and chart components
-for d3.js without taking away the power that d3.js gives you.
-
-Project location : https://github.com/areski/python-nvd3
-"""
-
-from .NVD3Chart import NVD3Chart, TemplateMixin
-
-
-class discreteBarChart(TemplateMixin, NVD3Chart):
-    """
-    A discrete bar chart or bar graph is a chart with rectangular bars with
-    lengths proportional to the values that they represent.
-
-    Python example::
-
-        from nvd3 import discreteBarChart
-        chart = discreteBarChart(name='discreteBarChart', height=400, width=400)
-
-        xdata = ["A", "B", "C", "D", "E", "F"]
-        ydata = [3, 4, 0, -3, 5, 7]
-
-        chart.add_serie(y=ydata, x=xdata)
-        chart.buildhtml()
-
-    Javascript generated:
-
-    .. raw:: html
-
-        <div id="discreteBarChart"><svg style="height:450px; width:100%"></svg></div>
-        <script>
-            data_discreteBarChart=[{"values": [{"y": 3, "x": "A"}, {"y": 4, "x": "B"}, {"y": 0, "x": "C"}, {"y": -3, "x": "D"}, {"y": 5, "x": "E"}, {"y": 7, "x": "F"}], "key": "Serie 1", "yAxis": "1"}];
-
-            nv.addGraph(function() {
-                var chart = nv.models.discreteBarChart();
-
-                chart.margin({top: 30, right: 60, bottom: 20, left: 60});
-
-                var datum = data_discreteBarChart;
-                        chart.yAxis
-                            .tickFormat(d3.format(',.0f'));
-                        chart.tooltipContent(function(key, y, e, graph) {
-                            var x = String(graph.point.x);
-                            var y = String(graph.point.y);
-                            var y = String(graph.point.y);
-
-                            tooltip_str = '<center><b>'+key+'</b></center>' + y + ' at ' + x;
-                            return tooltip_str;
-                        });
-
-                d3.select('#discreteBarChart svg')
-                    .datum(datum)
-                    .transition().duration(500)
-                    .attr('width', 400)
-                    .attr('height', 400)
-                    .call(chart);
-            });
-        </script>
-
-
-    """
-    CHART_FILENAME = "./discretebarchart.html"
-    template_chart_nvd3 = NVD3Chart.template_environment.get_template(CHART_FILENAME)
-
-    def __init__(self, **kwargs):
-        super(discreteBarChart, self).__init__(**kwargs)
-        self.model = 'discreteBarChart'
-        height = kwargs.get('height', 450)
-        width = kwargs.get('width', None)
-
-        if kwargs.get('x_is_date', False):
-            self.set_date_flag(True)
-            self.create_x_axis('xAxis',
-                               format=kwargs.get('x_axis_format',
-                                                 "%d %b %Y %H %S"),
-                               date=True)
-        else:
-            self.create_x_axis('xAxis', format=None)
-
-        self.create_y_axis('yAxis', format=kwargs.get('y_axis_format', ".0f"))
-
-        self.set_custom_tooltip_flag(True)
-
-        self.set_graph_height(height)
-        if width:
-            self.set_graph_width(width)
diff --git a/airflow/_vendor/nvd3/ipynb.py b/airflow/_vendor/nvd3/ipynb.py
deleted file mode 100644
index f421afc..0000000
--- a/airflow/_vendor/nvd3/ipynb.py
+++ /dev/null
@@ -1,91 +0,0 @@
-'''
-ipython compatability module for nvd3-python
-This adds simple ipython compatibility to the nvd3-python package, without making any
-major modifications to how the main package is structured.  It utilizes the IPython
-display-formatter functionality, as described at:
-http://nbviewer.ipython.org/github/ipython/ipython/blob/master/examples/notebooks/Custom%20Display%20Logic.ipynb
-For additional examples, see:
-https://github.com/sympy/sympy/blob/master/sympy/interactive/printing.py
-'''
-
-try:
-    _ip = get_ipython()
-except:
-    _ip = None
-if _ip and _ip.__module__.lower().startswith('ipy'):
-    global _js_initialized
-    _js_initialized = False
-
-    def _print_html(chart):
-        '''Function to return the HTML code for the div container plus the javascript
-        to generate the chart.  This function is bound to the ipython formatter so that
-        charts are displayed inline.'''
-        global _js_initialized
-        if not _js_initialized:
-            print('js not initialized - pausing to allow time for it to load...')
-            initialize_javascript()
-            import time
-            time.sleep(5)
-        chart.buildhtml()
-        return chart.htmlcontent
-
-    def _setup_ipython_formatter(ip):
-        ''' Set up the ipython formatter to display HTML formatted output inline'''
-        from IPython import __version__ as IPython_version
-        from nvd3 import __all__ as nvd3_all
-
-        if IPython_version >= '0.11':
-            html_formatter = ip.display_formatter.formatters['text/html']
-            for chart_type in nvd3_all:
-                html_formatter.for_type_by_name('nvd3.' + chart_type, chart_type, _print_html)
-
-    def initialize_javascript(d3_js_url='https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js',
-                              nvd3_js_url='https://cdnjs.cloudflare.com/ajax/libs/nvd3/1.7.0/nv.d3.min.js',
-                              nvd3_css_url='https://cdnjs.cloudflare.com/ajax/libs/nvd3/1.7.0/nv.d3.min.css',
-                              use_remote=False):
-        '''Initialize the ipython notebook to be able to display nvd3 results.
-        by instructing IPython to load the nvd3 JS and css files, and the d3 JS file.
-
-        by default, it looks for the files in your IPython Notebook working directory.
-
-        Takes the following options:
-
-        use_remote: use remote hosts for d3.js, nvd3.js, and nv.d3.css (default False)
-        * Note:  the following options are ignored if use_remote is False:
-        nvd3_css_url: location of nvd3 css file (default https://cdnjs.cloudflare.com/ajax/libs/nvd3/1.7.0/nv.d3.min.css)
-        nvd3_js_url: location of nvd3 javascript file (default  https://cdnjs.cloudflare.com/ajax/libs/nvd3/1.7.0/nv.d3.min.css)
-        d3_js_url: location of d3 javascript file (default https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js)
-        '''
-        from IPython.display import display, Javascript, HTML
-
-        if not use_remote:
-            # these file locations are for IPython 1.x, and will probably change when 2.x is released
-            d3_js_url = 'files/d3.v3.js'
-            nvd3_js_url = 'files/nv.d3.js'
-            nvd3_css_url = 'files/nv.d3.css'
-
-        # load the required javascript files
-
-        #display(Javascript('''$.getScript("%s")''' %(d3_js_url)))
-        display(HTML('''<link media="all" href="%s" type="text/css"
-                        rel="stylesheet"/>''' % (nvd3_css_url)))
-        # The following two methods for loading the script file are redundant.
-        # This is intentional.
-        # Ipython's loading of javscript in version 1.x is a bit squirrely, especially
-        # when creating demos to view in nbviewer.
-        # by trying twice, in two different ways (one using jquery and one using plain old
-        # HTML), we maximize our chances of successfully loading the script.
-        display(Javascript('''$.getScript("%s")''' % (nvd3_js_url)))
-        display(Javascript('''$.getScript("%s", function() {
-                              $.getScript("%s", function() {})});''' % (d3_js_url, nvd3_js_url)))
-        display(HTML('<script src="%s"></script>' % (d3_js_url)))
-        display(HTML('<script src="%s"></script>' % (nvd3_js_url)))
-
-        global _js_initialized
-        _js_initialized = True
-
-    print('loaded nvd3 IPython extension\n'
-          'run nvd3.ipynb.initialize_javascript() to set up the notebook\n'
-          'help(nvd3.ipynb.initialize_javascript) for options')
-
-    _setup_ipython_formatter(_ip)
diff --git a/airflow/_vendor/nvd3/lineChart.py b/airflow/_vendor/nvd3/lineChart.py
deleted file mode 100644
index c237d06..0000000
--- a/airflow/_vendor/nvd3/lineChart.py
+++ /dev/null
@@ -1,120 +0,0 @@
-#!/usr/bin/python
-# -*- coding: utf-8 -*-
-
-"""
-Python-nvd3 is a Python wrapper for NVD3 graph library.
-NVD3 is an attempt to build re-usable charts and chart components
-for d3.js without taking away the power that d3.js gives you.
-
-Project location : https://github.com/areski/python-nvd3
-"""
-
-from .NVD3Chart import NVD3Chart, TemplateMixin
-
-
-class lineChart(TemplateMixin, NVD3Chart):
-
-    """
-    A line chart or line graph is a type of chart which displays information
-    as a series of data points connected by straight line segments.
-
-    Python example::
-
-        from nvd3 import lineChart
-        chart = lineChart(name="lineChart", x_is_date=False, x_axis_format="AM_PM")
-
-        xdata = range(24)
-        ydata = [0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 4, 3, 3, 5, 7, 5, 3, 16, 6, 9, 15, 4, 12]
-        ydata2 = [9, 8, 11, 8, 3, 7, 10, 8, 6, 6, 9, 6, 5, 4, 3, 10, 0, 6, 3, 1, 0, 0, 0, 1]
-
-        extra_serie = {"tooltip": {"y_start": "There are ", "y_end": " calls"}}
-        chart.add_serie(y=ydata, x=xdata, name='sine', extra=extra_serie, **kwargs1)
-        extra_serie = {"tooltip": {"y_start": "", "y_end": " min"}}
-        chart.add_serie(y=ydata2, x=xdata, name='cose', extra=extra_serie, **kwargs2)
-        chart.buildhtml()
-
-    Javascript renderd to:
-
-    .. raw:: html
-
-        <div id="lineChart"><svg style="height:450px; width:100%"></svg></div>
-        <script>
-
-            data_lineChart=[{"values": [{"y": 0, "x": 0}, {"y": 0, "x": 1}, {"y": 1, "x": 2}, {"y": 1, "x": 3}, {"y": 0, "x": 4}, {"y": 0, "x": 5}, {"y": 0, "x": 6}, {"y": 0, "x": 7}, {"y": 1, "x": 8}, {"y": 0, "x": 9}, {"y": 0, "x": 10}, {"y": 4, "x": 11}, {"y": 3, "x": 12}, {"y": 3, "x": 13}, {"y": 5, "x": 14}, {"y": 7, "x": 15}, {"y": 5, "x": 16}, {"y": 3, "x": 17}, {"y": 16, "x": 18}, {"y": 6, "x": 19}, {"y": 9, "x": 20}, {"y": 15, "x": 21}, {"y": 4, "x": 22}, {"y": 12, "x": 23}], "k [...]
-
-            nv.addGraph(function() {
-                var chart = nv.models.lineChart();
-                chart.margin({top: 30, right: 60, bottom: 20, left: 60});
-                var datum = data_lineChart;
-                        chart.xAxis
-                            .tickFormat(function(d) { return get_am_pm(parseInt(d)); });
-                        chart.yAxis
-                            .tickFormat(d3.format(',.02f'));
-
-                        chart.tooltipContent(function(key, y, e, graph) {
-                            var x = String(graph.point.x);
-                            var y = String(graph.point.y);
-                                                if(key == 'sine'){
-                                var y = 'There is ' +  String(graph.point.y)  + ' calls';
-                            }
-                            if(key == 'cose'){
-                                var y =  String(graph.point.y)  + ' min';
-                            }
-
-                            tooltip_str = '<center><b>'+key+'</b></center>' + y + ' at ' + x;
-                            return tooltip_str;
-                        });
-                    chart.showLegend(true);
-                        function get_am_pm(d){
-                    if (d > 12) {
-                        d = d - 12; return (String(d) + 'PM');
-                    }
-                    else {
-                        return (String(d) + 'AM');
-                    }
-                };
-
-                d3.select('#lineChart svg')
-                    .datum(datum)
-                    .transition().duration(500)
-                    .attr('height', 200)
-                    .call(chart);
-            return chart;
-        });
-        </script>
-
-    See the source code of this page, to see the underlying javascript.
-    """
-    CHART_FILENAME = "./linechart.html"
-    template_chart_nvd3 = NVD3Chart.template_environment.get_template(CHART_FILENAME)
-
-    def __init__(self, **kwargs):
-        super(lineChart, self).__init__(**kwargs)
-        self.model = 'lineChart'
-
-        height = kwargs.get('height', 450)
-        width = kwargs.get('width', None)
-
-        if kwargs.get('x_is_date', False):
-            self.set_date_flag(True)
-            self.create_x_axis('xAxis',
-                               format=kwargs.get('x_axis_format', '%d %b %Y'),
-                               date=True)
-            self.set_custom_tooltip_flag(True)
-        else:
-            if kwargs.get('x_axis_format') == 'AM_PM':
-                self.x_axis_format = format = 'AM_PM'
-            else:
-                format = kwargs.get('x_axis_format', 'r')
-            self.create_x_axis('xAxis', format=format,
-                               custom_format=kwargs.get('x_custom_format',
-                                                        False))
-        self.create_y_axis(
-            'yAxis',
-            format=kwargs.get('y_axis_format', '.02f'),
-            custom_format=kwargs.get('y_custom_format', False))
-
-        # must have a specified height, otherwise it superimposes both chars
-        self.set_graph_height(height)
-        if width:
-            self.set_graph_width(width)
diff --git a/airflow/_vendor/nvd3/linePlusBarChart.py b/airflow/_vendor/nvd3/linePlusBarChart.py
deleted file mode 100644
index 4eaa5fc..0000000
--- a/airflow/_vendor/nvd3/linePlusBarChart.py
+++ /dev/null
@@ -1,131 +0,0 @@
-#!/usr/bin/python
-# -*- coding: utf-8 -*-
-
-"""
-Python-nvd3 is a Python wrapper for NVD3 graph library.
-NVD3 is an attempt to build re-usable charts and chart components
-for d3.js without taking away the power that d3.js gives you.
-
-Project location : https://github.com/areski/python-nvd3
-"""
-
-from .NVD3Chart import NVD3Chart, TemplateMixin
-
-
-class linePlusBarChart(TemplateMixin, NVD3Chart):
-
-    """
-    A linePlusBarChart Chart is a type of chart which displays information
-    as a series of data points connected by straight line segments
-    and with some series with rectangular bars with lengths proportional
-    to the values that they represent.
-
-    Python example::
-
-        from nvd3 import linePlusBarChart
-        chart = linePlusBarChart(name="linePlusBarChart",
-                             width=500, height=400, x_axis_format="%d %b %Y",
-                             x_is_date=True, focus_enable=True,
-                             yaxis2_format="function(d) { return d3.format(',0.3f')(d) }")
-
-        xdata = [1338501600000, 1345501600000, 1353501600000]
-        ydata = [6, 5, 1]
-        y2data = [0.002, 0.003, 0.004]
-
-        extra_serie = {"tooltip": {"y_start": "There are ", "y_end": " calls"},
-                       "date_format": "%d %b %Y %H:%S" }
-        chart.add_serie(name="Serie 1", y=ydata, x=xdata, extra=extra_serie,
-                        bar=True)
-
-        extra_serie = {"tooltip": {"y_start": "There are ", "y_end": " min"}}
-        chart.add_serie(name="Serie 2", y=y2data, x=xdata, extra=extra_serie)
-        chart.buildcontent()
-
-    Note that in case you have two data serie with extreme different numbers,
-    that you would like to format in different ways,
-    you can pass a keyword *yaxis1_format* or *yaxis2_format* when
-    creating the graph.
-
-    In the example above the graph created presents the values of the second
-    data series with three digits right  of the decimal point.
-
-    Javascript generated:
-
-    .. raw:: html
-
-        <div id="linePlusBarChart"><svg style="height:450px; width:100%"></svg></div>
-        <script>
-            data_linePlusBarChart=[{"bar": "true", "values": [{"y": 6, "x": 1338501600000}, {"y": 5, "x": 1345501600000}, {"y": 1, "x": 1353501600000}], "key": "Serie 1", "yAxis": "1"}, {"values": [{"y": 0.002, "x": 1338501600000}, {"y": 0.003, "x": 1345501600000}, {"y": 0.004, "x": 1353501600000}], "key": "Serie 2", "yAxis": "1"}];
-            nv.addGraph(function() {
-                var chart = nv.models.linePlusBarChart();
-                chart.margin({top: 30, right: 60, bottom: 20, left: 60});
-                var datum = data_linePlusBarChart;
-
-                    chart.y2Axis
-                        .tickFormat(function(d) { return d3.format(',0.3f')(d) });
-                    chart.xAxis
-                        .tickFormat(function(d) { return d3.time.format('%d %b %Y')(new Date(parseInt(d))) });
-                    chart.y1Axis
-                        .tickFormat(function(d) { return d3.format(',f')(d) });
-
-                    chart.tooltipContent(function(key, y, e, graph) {
-                        var x = d3.time.format("%d %b %Y %H:%S")(new Date(parseInt(graph.point.x)));
-                        var y = String(graph.point.y);
-                        if(key.indexOf('Serie 1') > -1 ){
-                                var y = 'There are ' +  String(graph.point.y)  + ' calls';
-                            }
-                            if(key.indexOf('Serie 2') > -1 ){
-                                var y = 'There are ' +  String(graph.point.y)  + ' min';
-                            }
-                        tooltip_str = '<center><b>'+key+'</b></center>' + y + ' on ' + x;
-                        return tooltip_str;
-                    });
-                    chart.showLegend(true);
-                d3.select('#linePlusBarChart svg')
-                    .datum(datum)
-                    .transition().duration(500)
-                    .attr('width', 500)
-                    .attr('height', 400)
-                    .call(chart);
-            });
-        </script>
-
-    """
-    CHART_FILENAME = "./lineplusbarchart.html"
-    template_chart_nvd3 = NVD3Chart.template_environment.get_template(CHART_FILENAME)
-
-    def __init__(self, **kwargs):
-        super(linePlusBarChart, self).__init__(**kwargs)
-        self.model = 'linePlusBarChart'
-
-        height = kwargs.get('height', 450)
-        width = kwargs.get('width', None)
-        self.yaxis1_format = kwargs.get('yaxis1_format',
-                                        "function(d) { return d3.format(',f')(d) }")
-        self.yaxis2_format = kwargs.get('yaxis2_format',
-                                        "function(d) { return d3.format(',f')(d) }")
-
-        if kwargs.get('x_is_date', False):
-            self.set_date_flag(True)
-            self.create_x_axis('xAxis',
-                               format=kwargs.get('x_axis_format',
-                                                 '%d %b %Y %H %S'),
-                               date=True)
-            self.create_x_axis('x2Axis', format=kwargs.get('x_axis_format',
-                                                           '%d %b %Y %H %S'),
-                               date=True)
-            self.set_custom_tooltip_flag(True)
-        else:
-            self.create_x_axis('xAxis', format=kwargs.get('x_axis_format',
-                                                          '.2f'))
-            self.create_x_axis('x2Axis', format=kwargs.get('x_axis_format',
-                                                           '.2f'))
-
-        self.create_y_axis('y1Axis', format=self.yaxis1_format,
-                           custom_format=True)
-        self.create_y_axis('y2Axis', format=self.yaxis2_format,
-                           custom_format=True)
-
-        self.set_graph_height(height)
-        if width:
-            self.set_graph_width(width)
diff --git a/airflow/_vendor/nvd3/lineWithFocusChart.py b/airflow/_vendor/nvd3/lineWithFocusChart.py
deleted file mode 100644
index cd26cd4..0000000
--- a/airflow/_vendor/nvd3/lineWithFocusChart.py
+++ /dev/null
@@ -1,105 +0,0 @@
-#!/usr/bin/python
-# -*- coding: utf-8 -*-
-
-"""
-Python-nvd3 is a Python wrapper for NVD3 graph library.
-NVD3 is an attempt to build re-usable charts and chart components
-for d3.js without taking away the power that d3.js gives you.
-
-Project location : https://github.com/areski/python-nvd3
-"""
-
-from .NVD3Chart import NVD3Chart, TemplateMixin
-
-
-class lineWithFocusChart(TemplateMixin, NVD3Chart):
-    """
-    A lineWithFocusChart or line graph is a type of chart which displays information
-    as a series of data points connected by straight line segments.
-    The lineWithFocusChart provide a smaller chart that act as a selector,
-    this is very useful if you want to zoom on a specific time period.
-
-    Python example::
-
-        from nvd3 import lineWithFocusChart
-        chart = lineWithFocusChart(name='lineWithFocusChart', x_is_date=True, x_axis_format="%d %b %Y")
-        xdata = [1365026400000000, 1365026500000000, 1365026600000000, 1365026700000000, 1365026800000000, 1365026900000000, 1365027000000000]
-        ydata = [-6, 5, -1, 2, 4, 8, 10]
-
-        extra_serie = {"tooltip": {"y_start": "", "y_end": " ext"},
-                       "date_format": "%d %b %Y"}
-        chart.add_serie(name="Serie 1", y=ydata, x=xdata, extra=extra_serie)
-        chart.buildhtml()
-
-    Javascript generated:
-
-    .. raw:: html
-
-        <div id="lineWithFocusChart"><svg style="height:450px; width:100%"></svg></div>
-        <script>
-            data_lineWithFocusChart=[{"values": [{"y": -6, "x": 1365026400000000}, {"y": 5, "x": 1365026500000000}, {"y": -1, "x": 1365026600000000}], "key": "Serie 1", "yAxis": "1"}];
-            nv.addGraph(function() {
-                var chart = nv.models.lineWithFocusChart();
-                chart.margin({top: 30, right: 60, bottom: 20, left: 60});
-                var datum = data_lineWithFocusChart;
-                        chart.yAxis
-                            .tickFormat(d3.format(',.2f'));
-                        chart.y2Axis
-                            .tickFormat(d3.format(',.2f'));
-                        chart.xAxis
-                            .tickFormat(function(d) { return d3.time.format('%d %b %Y')(new Date(parseInt(d))) });
-                        chart.x2Axis
-                            .tickFormat(function(d) { return d3.time.format('%d %b %Y')(new Date(parseInt(d))) });
-
-                    chart.tooltipContent(function(key, y, e, graph) {
-                        var x = d3.time.format("%d %b %Y")(new Date(parseInt(graph.point.x)));
-                        var y = String(graph.point.y);
-                                            if(key == 'Serie 1'){
-                                var y =  String(graph.point.y)  + ' ext';
-                            }
-
-                        tooltip_str = '<center><b>'+key+'</b></center>' + y + ' on ' + x;
-                        return tooltip_str; });
-
-                    chart.showLegend(true);
-
-                d3.select('#lineWithFocusChart svg')
-                    .datum(datum)
-                    .transition().duration(500)
-                    .attr('height', 450)
-                    .call(chart); });
-        </script>
-
-    """
-
-    CHART_FILENAME = "./linewfocuschart.html"
-    template_chart_nvd3 = NVD3Chart.template_environment.get_template(CHART_FILENAME)
-
-    def __init__(self, **kwargs):
-        super(lineWithFocusChart, self).__init__(**kwargs)
-        self.model = 'lineWithFocusChart'
-
-        height = kwargs.get('height', 450)
-        width = kwargs.get('width', None)
-
-        if kwargs.get('x_is_date', False):
-            self.set_date_flag(True)
-            self.create_x_axis('xAxis', format=kwargs.get('x_axis_format',
-                                                          '%d %b %Y %H %S'),
-                               date=True)
-            self.create_x_axis('x2Axis', format=kwargs.get('x_axis_format',
-                                                           '%d %b %Y %H %S'),
-                               date=True)
-            self.set_custom_tooltip_flag(True)
-        else:
-            self.create_x_axis('xAxis', format=kwargs.get('x_axis_format',
-                                                          '.2f'))
-            self.create_x_axis('x2Axis', format=kwargs.get('x_axis_format',
-                                                           '.2f'))
-
-        self.create_y_axis('yAxis', format=kwargs.get('y_axis_format', '.2f'))
-        self.create_y_axis('y2Axis', format=kwargs.get('y_axis_format', '.2f'))
-
-        self.set_graph_height(height)
-        if width:
-            self.set_graph_width(width)
diff --git a/airflow/_vendor/nvd3/multiBarChart.py b/airflow/_vendor/nvd3/multiBarChart.py
deleted file mode 100644
index cf33591..0000000
--- a/airflow/_vendor/nvd3/multiBarChart.py
+++ /dev/null
@@ -1,95 +0,0 @@
-#!/usr/bin/python
-# -*- coding: utf-8 -*-
-
-"""
-Python-nvd3 is a Python wrapper for NVD3 graph library.
-NVD3 is an attempt to build re-usable charts and chart components
-for d3.js without taking away the power that d3.js gives you.
-
-Project location : https://github.com/areski/python-nvd3
-"""
-
-from .NVD3Chart import NVD3Chart, TemplateMixin
-
-
-class multiBarChart(TemplateMixin, NVD3Chart):
-    """
-    A multiple bar graph contains comparisons of two or more categories or bars.
-    One axis represents a quantity and the other axis identifies a specific feature
-    about the categories. Reading a multiple bar graph includes looking at extremes
-    (tallest/longest vs. shortest) in each grouping.
-
-    Python example::
-
-        from nvd3 import multiBarChart
-        chart = multiBarChart(width=500, height=400, x_axis_format=None)
-        xdata = ['one', 'two', 'three', 'four']
-        ydata1 = [6, 12, 9, 16]
-        ydata2 = [8, 14, 7, 11]
-
-        chart.add_serie(name="Serie 1", y=ydata1, x=xdata)
-        chart.add_serie(name="Serie 2", y=ydata2, x=xdata)
-        chart.buildhtml()
-
-    Javascript generated:
-
-    .. raw:: html
-
-        <div id="multiBarChart"><svg style="height:450px; width:100%"></svg></div>
-        <script>
-
-            data_multiBarChart=[{"values":
-                                [{"y": 6, "x": "one"},
-                                {"y": 12, "x": "two"},
-                                {"y": 9, "x": "three"},
-                                {"y": 16, "x": "four"}],
-                                "key": "Serie 1", "yAxis": "1"},
-                                {"values":
-                                    [{"y": 8, "x": "one"},
-                                    {"y": 14, "x": "two"},
-                                    {"y": 7, "x": "three"},
-                                    {"y": 11, "x": "four"}],
-                                "key": "Serie 2", "yAxis": "1"}];
-
-            nv.addGraph(function() {
-                var chart = nv.models.multiBarChart();
-                chart.margin({top: 30, right: 60, bottom: 20, left: 60});
-                var datum = data_multiBarChart;
-                        chart.yAxis
-                            .tickFormat(d3.format(',.2f'));
-                    chart.showLegend(true);
-                d3.select('#multiBarChart svg')
-                    .datum(datum)
-                    .transition().duration(500)
-                    .attr('width', 500)
-                    .attr('height', 400)
-                    .call(chart);
-            });
-
-
-        </script>
-
-    """
-
-    CHART_FILENAME = "./multibarchart.html"
-    template_chart_nvd3 = NVD3Chart.template_environment.get_template(CHART_FILENAME)
-
-    def __init__(self, **kwargs):
-        super(multiBarChart, self).__init__(**kwargs)
-
-        height = kwargs.get('height', 450)
-        width = kwargs.get('width', None)
-
-        if kwargs.get('x_is_date', False):
-            self.set_date_flag(True)
-            self.create_x_axis('xAxis',
-                               format=kwargs.get('x_axis_format', '%d %b %Y'),
-                               date=True)
-            self.set_custom_tooltip_flag(True)
-        else:
-            self.create_x_axis('xAxis', format=kwargs.get('x_axis_format', '.2f'))
-        self.create_y_axis('yAxis', format=kwargs.get('y_axis_format', '.2f'))
-
-        self.set_graph_height(height)
-        if width:
-            self.set_graph_width(width)
diff --git a/airflow/_vendor/nvd3/multiBarHorizontalChart.py b/airflow/_vendor/nvd3/multiBarHorizontalChart.py
deleted file mode 100644
index ac969c3..0000000
--- a/airflow/_vendor/nvd3/multiBarHorizontalChart.py
+++ /dev/null
@@ -1,100 +0,0 @@
-#!/usr/bin/python
-# -*- coding: utf-8 -*-
-
-"""
-Python-nvd3 is a Python wrapper for NVD3 graph library.
-NVD3 is an attempt to build re-usable charts and chart components
-for d3.js without taking away the power that d3.js gives you.
-
-Project location : https://github.com/areski/python-nvd3
-"""
-
-from .NVD3Chart import NVD3Chart, TemplateMixin
-
-
-class multiBarHorizontalChart(TemplateMixin, NVD3Chart):
-    """
-    A multiple horizontal bar graph contains comparisons of two or more categories or bars.
-
-    Python example::
-
-        from nvd3 import multiBarHorizontalChart
-        chart = multiBarHorizontalChart(name='multiBarHorizontalChart', height=400, width=400)
-        xdata = [-14, -7, 7, 14]
-        ydata = [-6, 5, -1, 9]
-        y2data = [-23, -6, -32, 9]
-
-        extra_serie = {"tooltip": {"y_start": "", "y_end": " balls"}}
-        chart.add_serie(name="Serie 1", y=ydata, x=xdata, extra=extra_serie)
-
-        extra_serie = {"tooltip": {"y_start": "", "y_end": " calls"}}
-        chart.add_serie(name="Serie 2", y=y2data, x=xdata, extra=extra_serie)
-        chart.buildcontent()
-
-    Javascript generated:
-
-    .. raw:: html
-
-        <div id="multiBarHorizontalChart"><svg style="height:450px; width:100%"></svg></div>
-        <script>
-
-            data_multiBarHorizontalChart=[{"values":
-                [{"y": -6, "x": -14}, {"y": 5, "x": -7}, {"y": -1, "x": 7}, {"y": 9, "x": 14}],
-                "key": "Serie 1", "yAxis": "1"},
-                {"values":
-                    [{"y": -23, "x": -14}, {"y": -6, "x": -7}, {"y": -32, "x": 7}, {"y": 9, "x": 14}],
-                "key": "Serie 2", "yAxis": "1"}];
-
-            nv.addGraph(function() {
-                var chart = nv.models.multiBarHorizontalChart();
-
-                chart.margin({top: 30, right: 60, bottom: 20, left: 60});
-
-                var datum = data_multiBarHorizontalChart;
-
-                        chart.xAxis
-                            .tickFormat(d3.format(',.2f'));
-                        chart.yAxis
-                            .tickFormat(d3.format(',.2f'));
-
-                        chart.tooltipContent(function(key, y, e, graph) {
-                            var x = String(graph.point.x);
-                            var y = String(graph.point.y);
-                                                if(key == 'Serie 1'){
-                                var y =  String(graph.point.y)  + ' balls';
-                            }
-                            if(key == 'Serie 2'){
-                                var y =  String(graph.point.y)  + ' calls';
-                            }
-
-                            tooltip_str = '<center><b>'+key+'</b></center>' + y + ' at ' + x;
-                            return tooltip_str;
-                        });
-
-                    chart.showLegend(true);
-
-                d3.select('#multiBarHorizontalChart svg')
-                    .datum(datum)
-                    .transition().duration(500)
-                    .attr('width', 400)
-                    .attr('height', 400)
-                    .call(chart);
-            });
-        </script>
-
-    """
-
-    CHART_FILENAME = "./multibarcharthorizontal.html"
-    template_chart_nvd3 = NVD3Chart.template_environment.get_template(CHART_FILENAME)
-
-    def __init__(self, **kwargs):
-        super(multiBarHorizontalChart, self).__init__(**kwargs)
-        height = kwargs.get('height', 450)
-        width = kwargs.get('width', None)
-
-        self.create_x_axis('xAxis', format=kwargs.get('x_axis_format', '.2f'))
-        self.create_y_axis('yAxis', format=kwargs.get('y_axis_format', '.2f'))
-
-        self.set_graph_height(height)
-        if width:
-            self.set_graph_width(width)
diff --git a/airflow/_vendor/nvd3/pieChart.py b/airflow/_vendor/nvd3/pieChart.py
deleted file mode 100644
index 1db76bd..0000000
--- a/airflow/_vendor/nvd3/pieChart.py
+++ /dev/null
@@ -1,101 +0,0 @@
-#!/usr/bin/python
-# -*- coding: utf-8 -*-
-
-"""
-Python-nvd3 is a Python wrapper for NVD3 graph library.
-NVD3 is an attempt to build re-usable charts and chart components
-for d3.js without taking away the power that d3.js gives you.
-
-Project location : https://github.com/areski/python-nvd3
-"""
-
-from .NVD3Chart import NVD3Chart, TemplateMixin
-
-
-class pieChart(TemplateMixin, NVD3Chart):
-
-    """
-    A pie chart (or a circle graph) is a circular chart divided into sectors,
-    illustrating numerical proportion. In chart, the arc length of each sector
-    is proportional to the quantity it represents.
-
-    Python example::
-
-        from nvd3 import pieChart
-        chart = pieChart(name='pieChart', color_category='category20c',
-                         height=400, width=400)
-
-        xdata = ["Orange", "Banana", "Pear", "Kiwi", "Apple", "Strawbery",
-                 "Pineapple"]
-        ydata = [3, 4, 0, 1, 5, 7, 3]
-
-        extra_serie = {"tooltip": {"y_start": "", "y_end": " cal"}}
-        chart.add_serie(y=ydata, x=xdata, extra=extra_serie)
-        chart.buildhtml()
-
-    Javascript generated:
-
-    .. raw:: html
-
-        <div id="pieChart"><svg style="height:450px; width:100%"></svg></div>
-        <script>
-
-
-            data_pieChart=[{"values": [{"value": 3, "label": "Orange"},
-                           {"value": 4, "label": "Banana"},
-                           {"value": 0, "label": "Pear"},
-                           {"value": 1, "label": "Kiwi"},
-                           {"value": 5, "label": "Apple"},
-                           {"value": 7, "label": "Strawberry"},
-                           {"value": 3, "label": "Pineapple"}],
-                           "key": "Serie 1"}];
-
-            nv.addGraph(function() {
-                var chart = nv.models.pieChart();
-                chart.margin({top: 30, right: 60, bottom: 20, left: 60});
-                var datum = data_pieChart[0].values;
-                        chart.tooltipContent(function(key, y, e, graph) {
-                            var x = String(key);
-                            var y =  String(y)  + ' cal';
-
-                            tooltip_str = '<center><b>'+x+'</b></center>' + y;
-                            return tooltip_str;
-                        });
-                    chart.showLegend(true);
-                    chart.showLabels(true);
-                    chart.donut(false);
-                chart
-                    .x(function(d) { return d.label })
-                    .y(function(d) { return d.value });
-                chart.width(400);
-                chart.height(400);
-
-                d3.select('#pieChart svg')
-                    .datum(datum)
-                    .transition().duration(500)
-                    .attr('width', 400)
-                    .attr('height', 400)
-                    .call(chart);  });
-        </script>
-
-    """
-    CHART_FILENAME = "./piechart.html"
-    template_chart_nvd3 = NVD3Chart.template_environment.get_template(CHART_FILENAME)
-
-    def __init__(self, **kwargs):
-        super(pieChart, self).__init__(**kwargs)
-
-        height = kwargs.get('height', 450)
-        width = kwargs.get('width', None)
-        self.donut = kwargs.get('donut', False)
-        self.donutRatio = kwargs.get('donutRatio', 0.35)
-        self.color_list = []
-        self.create_x_axis('xAxis', format=None)
-        self.create_y_axis('yAxis', format=None)
-        # must have a specified height, otherwise it superimposes both chars
-        if height:
-            self.set_graph_height(height)
-        if width:
-            self.set_graph_width(width)
-        self.donut = kwargs.get('donut', False)
-        self.donutRatio = kwargs.get('donutRatio', 0.35)
diff --git a/airflow/_vendor/nvd3/scatterChart.py b/airflow/_vendor/nvd3/scatterChart.py
deleted file mode 100644
index c3a87d2..0000000
--- a/airflow/_vendor/nvd3/scatterChart.py
+++ /dev/null
@@ -1,121 +0,0 @@
-#!/usr/bin/python
-# -*- coding: utf-8 -*-
-
-"""
-Python-nvd3 is a Python wrapper for NVD3 graph library.
-NVD3 is an attempt to build re-usable charts and chart components
-for d3.js without taking away the power that d3.js gives you.
-
-Project location : https://github.com/areski/python-nvd3
-"""
-
-from .NVD3Chart import NVD3Chart, TemplateMixin
-
-
-class scatterChart(TemplateMixin, NVD3Chart):
-
-    """
-    A scatter plot or scattergraph is a type of mathematical diagram using Cartesian
-    coordinates to display values for two variables for a set of data.
-    The data is displayed as a collection of points, each having the value of one variable
-    determining the position on the horizontal axis and the value of the other variable
-    determining the position on the vertical axis.
-
-    Python example::
-
-        from nvd3 import scatterChart
-        chart = scatterChart(name='scatterChart', height=400, width=400)
-        xdata = [3, 4, 0, -3, 5, 7]
-        ydata = [-1, 2, 3, 3, 15, 2]
-        ydata2 = [1, -2, 4, 7, -5, 3]
-
-        kwargs1 = {'shape': 'circle', 'size': '1'}
-        kwargs2 = {'shape': 'cross', 'size': '10'}
-
-        extra_serie = {"tooltip": {"y_start": "", "y_end": " call"}}
-        chart.add_serie(name="series 1", y=ydata, x=xdata, extra=extra_serie, **kwargs1)
-
-        extra_serie = {"tooltip": {"y_start": "", "y_end": " min"}}
-        chart.add_serie(name="series 2", y=ydata2, x=xdata, extra=extra_serie, **kwargs2)
-        chart.buildhtml()
-
-    Javascript generated:
-
-    .. raw:: html
-
-        <div id="scatterChart"><svg style="height:450px; width:100%"></svg></div>
-        <script>
-
-        data_scatterChart=[{"values": [{"y": -1, "x": 3, "shape": "circle",
-            "size": "1"}, {"y": 2, "x": 4, "shape": "circle", "size": "1"},
-            {"y": 3, "x": 0, "shape": "circle", "size": "1"},
-            {"y": 3, "x": -3, "shape": "circle", "size": "1"},
-            {"y": 15, "x": 5, "shape": "circle", "size": "1"},
-            {"y": 2, "x": 7, "shape": "circle", "size": "1"}],
-            "key": "series 1", "yAxis": "1"},
-            {"values": [{"y": 1, "x": 3, "shape": "cross", "size": "10"},
-            {"y": -2, "x": 4, "shape": "cross", "size": "10"},
-            {"y": 4, "x": 0, "shape": "cross", "size": "10"},
-            {"y": 7, "x": -3, "shape": "cross", "size": "10"},
-            {"y": -5, "x": 5, "shape": "cross", "size": "10"},
-            {"y": 3, "x": 7, "shape": "cross", "size": "10"}],
-            "key": "series 2", "yAxis": "1"}];
-        nv.addGraph(function() {
-        var chart = nv.models.scatterChart();
-
-        chart.margin({top: 30, right: 60, bottom: 20, left: 60});
-
-        var datum = data_scatterChart;
-
-                chart.xAxis
-                    .tickFormat(d3.format(',.02f'));
-                chart.yAxis
-                    .tickFormat(d3.format(',.02f'));
-
-                chart.tooltipContent(function(key, y, e, graph) {
-                    var x = String(graph.point.x);
-                    var y = String(graph.point.y);
-                                        if(key == 'series 1'){
-                        var y =  String(graph.point.y)  + ' call';
-                    }
-                    if(key == 'series 2'){
-                        var y =  String(graph.point.y)  + ' min';
-                    }
-
-                    tooltip_str = '<center><b>'+key+'</b></center>' + y + ' at ' + x;
-                    return tooltip_str;
-                });
-
-        chart.showLegend(true);
-
-        chart
-        .showDistX(true)
-        .showDistY(true)
-        .color(d3.scale.category10().range());
-
-            d3.select('#scatterChart svg')
-                .datum(datum)
-                .transition().duration(500)
-                .attr('width', 400)
-                .attr('height', 400)
-                .call(chart);
-        });
-        </script>
-
-    """
-
-    CHART_FILENAME = "./scatterchart.html"
-    template_chart_nvd3 = NVD3Chart.template_environment.get_template(CHART_FILENAME)
-
-    def __init__(self, **kwargs):
-        super(scatterChart, self).__init__(**kwargs)
-        self.model = 'scatterChart'
-        height = kwargs.get('height', 450)
-        width = kwargs.get('width', None)
-        self.create_x_axis('xAxis', format=kwargs.get('x_axis_format', '.02f'),
-                           label=kwargs.get('x_axis_label', None))
-        self.create_y_axis('yAxis', format=kwargs.get('y_axis_format', '.02f'),
-                           label=kwargs.get('y_axis_label', None))
-        self.set_graph_height(height)
-        if width:
-            self.set_graph_width(width)
diff --git a/airflow/_vendor/nvd3/stackedAreaChart.py b/airflow/_vendor/nvd3/stackedAreaChart.py
deleted file mode 100644
index 8346cd2..0000000
--- a/airflow/_vendor/nvd3/stackedAreaChart.py
+++ /dev/null
@@ -1,99 +0,0 @@
-#!/usr/bin/python
-# -*- coding: utf-8 -*-
-
-"""
-Python-nvd3 is a Python wrapper for NVD3 graph library.
-NVD3 is an attempt to build re-usable charts and chart components
-for d3.js without taking away the power that d3.js gives you.
-
-Project location : https://github.com/areski/python-nvd3
-"""
-
-from .NVD3Chart import NVD3Chart, TemplateMixin
-
-
-class stackedAreaChart(TemplateMixin, NVD3Chart):
-    """
-    The stacked area chart is identical to the area chart, except the areas are stacked
-    on top of each other, rather than overlapping. This can make the chart much easier to read.
-
-    Python example::
-
-        from nvd3 import stackedAreaChart
-        chart = stackedAreaChart(name='stackedAreaChart', height=400, width=400)
-
-        xdata = [100, 101, 102, 103, 104, 105, 106,]
-        ydata = [6, 11, 12, 7, 11, 10, 11]
-        ydata2 = [8, 20, 16, 12, 20, 28, 28]
-
-        extra_serie = {"tooltip": {"y_start": "There is ", "y_end": " min"}}
-        chart.add_serie(name="Serie 1", y=ydata, x=xdata, extra=extra_serie)
-        chart.add_serie(name="Serie 2", y=ydata2, x=xdata, extra=extra_serie)
-        chart.buildhtml()
-
-    Javascript generated:
-
-    .. raw:: html
-
-        <div id="stackedAreaChart"><svg style="height:450px; width:100%"></svg></div>
-        <script>
-
-
-            data_stackedAreaChart=[{"values": [{"y": 6, "x": 100}, {"y": 11, "x": 101}, {"y": 12, "x": 102}, {"y": 7, "x": 103}, {"y": 11, "x": 104}, {"y": 10, "x": 105}, {"y": 11, "x": 106}], "key": "Serie 1", "yAxis": "1"}, {"values": [{"y": 8, "x": 100}, {"y": 20, "x": 101}, {"y": 16, "x": 102}, {"y": 12, "x": 103}, {"y": 20, "x": 104}, {"y": 28, "x": 105}, {"y": 28, "x": 106}], "key": "Serie 2", "yAxis": "1"}];
-            nv.addGraph(function() {
-                var chart = nv.models.stackedAreaChart();
-                chart.margin({top: 30, right: 60, bottom: 20, left: 60});
-                var datum = data_stackedAreaChart;
-                        chart.xAxis
-                            .tickFormat(d3.format(',.2f'));
-                        chart.yAxis
-                            .tickFormat(d3.format(',.2f'));
-
-                        chart.tooltipContent(function(key, y, e, graph) {
-                            var x = String(graph.point.x);
-                            var y = String(graph.point.y);
-                                                if(key == 'Serie 1'){
-                                var y = 'There is ' +  String(graph.point.y)  + ' min';
-                            }
-                            if(key == 'Serie 2'){
-                                var y = 'There is ' +  String(graph.point.y)  + ' min';
-                            }
-
-                            tooltip_str = '<center><b>'+key+'</b></center>' + y + ' at ' + x;
-                            return tooltip_str;
-                        });
-                    chart.showLegend(true);
-                d3.select('#stackedAreaChart svg')
-                    .datum(datum)
-                    .transition().duration(500)
-                    .attr('width', 400)
-                    .attr('height', 400)
-                    .call(chart);
-            });
-        </script>
-
-    """
-
-    CHART_FILENAME = "./stackedareachart.html"
-    template_chart_nvd3 = NVD3Chart.template_environment.get_template(CHART_FILENAME)
-
-    def __init__(self, **kwargs):
-        super(stackedAreaChart, self).__init__(**kwargs)
-        height = kwargs.get('height', 450)
-        width = kwargs.get('width', None)
-        self.model = 'stackedAreaChart'
-
-        if kwargs.get('x_is_date', False):
-            self.set_date_flag(True)
-            self.create_x_axis('xAxis',
-                               format=kwargs.get('x_axis_format', '%d %b %Y'),
-                               date=True)
-            self.set_custom_tooltip_flag(True)
-        else:
-            self.create_x_axis('xAxis', format=kwargs.get('x_axis_format',
-                                                          '.2f'))
-        self.create_y_axis('yAxis', format=kwargs.get('y_axis_format', '.2f'))
-
-        self.set_graph_height(height)
-        if width:
-            self.set_graph_width(width)
diff --git a/airflow/_vendor/nvd3/templates/base.html b/airflow/_vendor/nvd3/templates/base.html
deleted file mode 100644
index 997e633..0000000
--- a/airflow/_vendor/nvd3/templates/base.html
+++ /dev/null
@@ -1,35 +0,0 @@
-{% block container %}
-{% endblock %}
-
-{% block start_script %}
-    {% if chart.tag_script_js %}
-    <script>
-    {% endif %}
-
-    {% if chart.jquery_on_ready %}
-    $(function(){
-    {% endif %}
-{% endblock start_script %}
-
-{% block body %}
-
-    {% block data %}
-    {% endblock data %}
-
-    {% block inject %}
-    {% endblock inject %}
-
-    {% block foot %}
-    {% endblock foot %}
-     <!-- end of body from base-->
-
-     {% endblock body%}
-
-{% block endscript %}
-    {% if chart.jquery_on_ready %}
-        });
-    {% endif %}
-    {% if chart.tag_script_js %}
-    </script>
-    {% endif %}
-{% endblock endscript %}
diff --git a/airflow/_vendor/nvd3/templates/content.html b/airflow/_vendor/nvd3/templates/content.html
deleted file mode 100644
index 787f39b..0000000
--- a/airflow/_vendor/nvd3/templates/content.html
+++ /dev/null
@@ -1,123 +0,0 @@
-{% extends "base.html" %}
-{% block container %}
-{% if chart.display_container %}
-    {{ chart.container }}
-{% endif %}
-{%  endblock  container %}
-
-{% block body %}
-    {% block data %}
-        data_{{ chart.name }}={{ chart.series_js }};
-    {% endblock data %}
-
-    {% block init %}
-    nv.addGraph(function() {
-        var chart = nv.models.{{ chart.model }}(){% if chart.use_interactive_guideline %}.useInteractiveGuideline(true){% endif %};
-
-        chart.margin({top: {{ chart.margin_top }}, right: {{ chart.margin_right }}, bottom: {{ chart.margin_bottom }}, left: {{ chart.margin_left }}});
-
-        var datum = data_{{ chart.name }};
-
-    {% if not chart.color_list and chart.color_category %}
-        chart.color(d3.scale.{{ chart.color_category }}().range());
-    {% endif %}
-    {% endblock init %}
-
-    {% if chart.stacked %}
-        chart.stacked(true);
-    {% endif %}
-
-    {% block focus %}
-    {% endblock focus %}
-
-
-    {% block axes %}
-        {% for axis, a in chart.axislist.items() %}
-            {% if a.items() %}
-            chart.{{ axis }}
-            {% for attr, value in a.items() %}
-                .{{ attr}}({{ value}}){% if loop.last %};
-                {% endif %}
-            {% endfor %}
-            {% endif %}
-        {% endfor %}
-    {% endblock axes %}
-
-    {# generate custom tooltip for the chart #}
-    {% block tooltip %}
-        {% if chart.custom_tooltip_flag %}
-            {% if not chart.date_flag %}
-                {% if chart.model == 'pieChart' %}
-                    {% block pietooltip %}
-                    {% endblock pietooltip %}
-                {% else %}
-                    chart.tooltipContent(function(key, y, e, graph) {
-                        var x = String(graph.point.x);
-                        var y = String(graph.point.y);
-                        {{ chart.tooltip_condition_string }}
-                        tooltip_str = '<center><b>'+key+'</b></center>' + y + ' at ' + x;
-                        return tooltip_str;
-                    });
-                {% endif %}
-            {% else %}
-                chart.tooltipContent(function(key, y, e, graph) {
-                    var x = d3.time.format("{{ chart.charttooltip_dateformat }}")(new Date(parseInt(graph.point.x)));
-                    var y = String(graph.point.y);
-                    {{ chart.tooltip_condition_string }}
-                    tooltip_str = '<center><b>'+key+'</b></center>' + y + ' on ' + x;
-                    return tooltip_str;
-                });
-            {% endif %}
-        {% endif %}
-    {% endblock tooltip %}
-
-    {# the shape attribute in kwargs is not applied when #}
-    {# not allowing other shapes to be rendered #}
-    {% block legend %}
-      chart.showLegend({{chart.show_legend|lower}});
-    {% endblock legend %}
-
-    {% block custoattr %}
-        {# add custom chart attributes #}
-        {% for attr, value in chart.chart_attr.items() %}
-            {% if value is string and value.startswith(".") %}:
-                chart.{{ attr }}{{ value }};
-            {% else %}
-                chart.{{ attr }}({{ value }});
-            {% endif %}
-        {% endfor %}
-
-        {% if chart.resize %}
-            nv.utils.windowResize(chart.update);
-        {% endif %}
-
-        {# include specific subchart #}
-        {{ chart.jschart }}
-
-    {% endblock custoattr %}
-
-    {% block inject %}
-        {# Inject data to D3 #}
-        d3.select('#{{ chart.name }} svg')
-            .datum(datum)
-            .transition().duration(500)
-            {% if chart.width %}
-            .attr('width', {{ chart.width}})
-            {% endif %}
-            {% if chart.height %}
-            .attr('height', {{ chart.height}})
-            {% endif %}
-            .call(chart);
-    {% endblock inject %}
-
-    {# extra chart attributes #}
-    {% if chart.extras %}
-        {{ chart.extras }}
-    {% endif %}
-
-    {# closing nv.addGraph #}
-    {% block close %}
-    });
-    {% endblock close %}
-
-{% endblock body %}
diff --git a/airflow/_vendor/nvd3/templates/cumulativelinechart.html b/airflow/_vendor/nvd3/templates/cumulativelinechart.html
deleted file mode 100644
index 66b6c74..0000000
--- a/airflow/_vendor/nvd3/templates/cumulativelinechart.html
+++ /dev/null
@@ -1,10 +0,0 @@
-{# This template adds attributes unique
-   to cumulativeLineChart #}
-
-{% extends "content.html" %}
-{% block body %}
-
-{# calling super guarantees everying in content is also found here ...#}
-{{super()}}
-
-{% endblock body %}
diff --git a/airflow/_vendor/nvd3/templates/discretebarchart.html b/airflow/_vendor/nvd3/templates/discretebarchart.html
deleted file mode 100644
index 3a52b3a..0000000
--- a/airflow/_vendor/nvd3/templates/discretebarchart.html
+++ /dev/null
@@ -1,31 +0,0 @@
-{# This is a dummy template, we can use that template to add attributes unique
-   to discreteBarChart #}
-
-{% extends "content.html" %}
-{% block body %}
-
-    {% block data %}
-        {{super()}}
-    {% endblock data %}
-
-    {% block init %}
-        {{super()}}
-    {% endblock init %}
-
-    {% block axes %}
-        {{super()}}
-    {% endblock axes %}
-
-    {% block custoattr %}
-    {{super()}}
-    {% endblock custoattr %}
-
-    {% block inject %}
-    {{ super() }}
-    {% endblock inject %}
-
-    {% block close %}
-    {{ super() }}
-    {% endblock close %}
-
-{% endblock body %}
diff --git a/airflow/_vendor/nvd3/templates/linebarwfocuschart.html b/airflow/_vendor/nvd3/templates/linebarwfocuschart.html
deleted file mode 100644
index ad4866c..0000000
--- a/airflow/_vendor/nvd3/templates/linebarwfocuschart.html
+++ /dev/null
@@ -1,60 +0,0 @@
-{# This template adds attributes unique
-   to lineChart #}
-
-{% extends "content.html" %}
-{% block body %}
-    {% block data %}
-        data_{{ chart.name }}={{ chart.series_js }};
-    {% endblock data %}
-
-
-    {% block init %}
-        {{super()}}
-    {% endblock init %}
-    {% block axes %}
-        {{super()}}
-    {% endblock axes %}
-    {% block tooltip %}
-        {{super()}}
-    {% endblock tooltip %}
-
-    chart.showLegend({{chart.show_legend|lower}});
-
-    {# add custom chart attributes #}
-    {% for attr, value in chart.chart_attr.items() %}
-        {% if value is string and value.startswith(".") %}:
-            chart.{{ attr }}{{ value }};
-        {% else %}
-            chart.{{ attr }}({{ value }});
-        {% endif %}
-    {% endfor %}
-
-    {% if chart.x_axis_format == 'AM_PM' %}
-        function get_am_pm(d){
-            if (d > 12) {
-                d = d - 12; return (String(d) + 'PM');
-            }
-            else {
-                return (String(d) + 'AM');
-            }
-        };
-    {% else %}
-        chart.x(function(d,i) { return i });
-    {% endif %}
-
-    {% if chart.resize %}
-        nv.utils.windowResize(chart.update);
-    {% endif %}
-    {% block inject %}
-    {{super()}}
-    {% endblock inject %}
-
-		{% if chart.extras %}
-			{{ chart.extras }}
-		{% endif %}
-
-    {% block close %}
-    });
-    {% endblock close %}
-
-{% endblock body %}
diff --git a/airflow/_vendor/nvd3/templates/linechart.html b/airflow/_vendor/nvd3/templates/linechart.html
deleted file mode 100644
index edd9633..0000000
--- a/airflow/_vendor/nvd3/templates/linechart.html
+++ /dev/null
@@ -1,46 +0,0 @@
-{# This template adds attributes unique
-   to lineChart #}
-
-{% extends "content.html" %}
-{% block body %}
-
-    {% block data %}
-    {{super()}}
-    {% endblock data %}
-
-    {% block init %}
-    {{super()}}
-    {% endblock init %}
-
-    {% block axes %}
-    {{super()}}
-    {% endblock axes %}
-
-    {% if chart.x_axis_format == 'AM_PM' %}
-        function get_am_pm(d){
-            if (d > 12) {
-                d = d - 12; return (String(d) + 'PM');
-            }
-            else {
-                return (String(d) + 'AM');
-            }
-        };
-    {% endif %}
-
-    {% block legend %}
-    {{super()}}
-    {% endblock legend %}
-
-    {% block custoattr %}
-    {{super()}}
-    {% endblock custoattr %}
-
-    {% block inject %}
-    {{ super() }}
-    {% endblock inject %}
-
-    {% block close %}
-    {{ super() }}
-    {% endblock close %}
-
-{% endblock body %}
diff --git a/airflow/_vendor/nvd3/templates/lineplusbarchart.html b/airflow/_vendor/nvd3/templates/lineplusbarchart.html
deleted file mode 100644
index 830a192..0000000
--- a/airflow/_vendor/nvd3/templates/lineplusbarchart.html
+++ /dev/null
@@ -1,43 +0,0 @@
-{# This template adds attributes unique
-   to linePlusBarChart #}
-
-{% extends "content.html" %}
-{% block body %}
-
-    {% block data %}
-    {{super()}}
-    {% endblock data %}
-
-    {% block init %}
-    {{super()}}
-    {% endblock init %}
-
-    {% block focus %}
-    {% if chart.focus_enable %}
-        chart.focusEnable(true);
-    {% else %}
-        chart.focusEnable(false);
-    {% endif %}
-    {% endblock focus %}
-
-    {% block axes %}
-    {{super()}}
-    {% endblock axes %}
-
-    {% block legend %}
-    {{super()}}
-    {% endblock legend %}
-
-    {% block custoattr %}
-    {{super()}}
-    {% endblock custoattr %}
-
-    {% block inject %}
-    {{ super() }}
-    {% endblock inject %}
-
-    {% block close %}
-    {{ super() }}
-    {% endblock close %}
-
-{% endblock body %}
diff --git a/airflow/_vendor/nvd3/templates/linewfocuschart.html b/airflow/_vendor/nvd3/templates/linewfocuschart.html
deleted file mode 100644
index ef02867..0000000
--- a/airflow/_vendor/nvd3/templates/linewfocuschart.html
+++ /dev/null
@@ -1,10 +0,0 @@
-{# This template adds attributes unique
-   to lineWithFocusChart #}
-
-{% extends "content.html" %}
-{% block body %}
-
-{# calling super guarantees everying in content is also found here ...#}
-{{super()}}
-
-{% endblock body %}
diff --git a/airflow/_vendor/nvd3/templates/multibarchart.html b/airflow/_vendor/nvd3/templates/multibarchart.html
deleted file mode 100644
index 17eae7a..0000000
--- a/airflow/_vendor/nvd3/templates/multibarchart.html
+++ /dev/null
@@ -1,10 +0,0 @@
-{# This template adds attributes unique
-   to multiBarChart #}
-
-{% extends "content.html" %}
-{% block body %}
-
-{# calling super guarantees everying in content is also found here ...#}
-{{super()}}
-
-{% endblock body %}
diff --git a/airflow/_vendor/nvd3/templates/multibarcharthorizontal.html b/airflow/_vendor/nvd3/templates/multibarcharthorizontal.html
deleted file mode 100644
index 17eae7a..0000000
--- a/airflow/_vendor/nvd3/templates/multibarcharthorizontal.html
+++ /dev/null
@@ -1,10 +0,0 @@
-{# This template adds attributes unique
-   to multiBarChart #}
-
-{% extends "content.html" %}
-{% block body %}
-
-{# calling super guarantees everying in content is also found here ...#}
-{{super()}}
-
-{% endblock body %}
diff --git a/airflow/_vendor/nvd3/templates/page.html b/airflow/_vendor/nvd3/templates/page.html
deleted file mode 100644
index 2dd0f5d..0000000
--- a/airflow/_vendor/nvd3/templates/page.html
+++ /dev/null
@@ -1,12 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-    <head>
-        <meta charset="utf-8" />
-        {% for header_element in chart.header_css+chart.header_js %}
-        {{ header_element }}
-        {% endfor %}
-    </head>
-    <body>
-        {{ chart.content }}
-    </body>
-</html>
diff --git a/airflow/_vendor/nvd3/templates/piechart.html b/airflow/_vendor/nvd3/templates/piechart.html
deleted file mode 100644
index a200e6d..0000000
--- a/airflow/_vendor/nvd3/templates/piechart.html
+++ /dev/null
@@ -1,80 +0,0 @@
-{# This template adds attributes unique
-   to pieChart #}
-
-{% extends "content.html" %}
-{% block body %}
-
-    data_{{ chart.name }}={{ chart.series_js }};
-
-    nv.addGraph(function() {
-        var chart = nv.models.{{ chart.model }}(){% if chart.use_interactive_guideline %}.useInteractiveGuideline(true){% endif %};
-        chart.margin({top: {{ chart.margin_top }}, right: {{ chart.margin_right }}, bottom: {{ chart.margin_bottom }}, left: {{ chart.margin_left }}});
-        var datum = data_{{ chart.name }}[0].values;
-
-    {% if not chart.color_list and chart.color_category %}
-        chart.color(d3.scale.{{ chart.color_category }}().range());
-    {% endif %}
-
-    chart.tooltipContent(function(key, y, e, graph) {
-          var x = String(key);
-              {{ chart.tooltip_condition_string }}
-              tooltip_str = '<center><b>'+x+'</b></center>' + y;
-              return tooltip_str;
-              });
-    {# showLabels only supported in pieChart #}
-        chart.showLabels({{chart.show_labels|lower}});
-
-        {% if chart.donut %}
-            chart.donut(true);
-            chart.donutRatio({{ chart.donutRatio }});
-        {% else %}
-            chart.donut(false);
-        {% endif %}
-
-    chart.showLegend({{chart.show_legend|lower}});
-
-    {# add custom chart attributes #}
-    {% for attr, value in chart.chart_attr.items() %}
-        {% if value is string and value.startswith(".") %}:
-            chart.{{ attr }}{{ value }};
-        {% else %}
-            chart.{{ attr }}({{ value }});
-        {% endif %}
-    {% endfor %}
-
-    {% if chart.resize %}
-        nv.utils.windowResize(chart.update);
-    {% endif %}
-
-    {% if chart.color_list %}
-        var mycolor = new Array();
-        {% for color in chart.color_list %}
-            mycolor[{{ loop.index - 1}}] = "{{ color }}";
-        {% endfor %}
-    {% endif %}
-
-        chart
-            .x(function(d) { return d.label })
-            .y(function(d) { return d.value });
-
-    {% if chart.width %}
-        chart.width({{ chart.width }});
-    {% endif %}
-
-    {% if chart.height %}
-        chart.height({{ chart.height }});
-    {% endif %}
-
-    {% if chart.color_list %}
-        chart.color(mycolor);
-    {% endif %}
-
-    {% block inject %}
-    {{super()}}
-    {% endblock inject %}
-
-    {% block close %}
-    {{ super() }}
-    {% endblock close %}
-
-{% endblock body %}
diff --git a/airflow/_vendor/nvd3/templates/scatterchart.html b/airflow/_vendor/nvd3/templates/scatterchart.html
deleted file mode 100644
index 8c2adaa..0000000
--- a/airflow/_vendor/nvd3/templates/scatterchart.html
+++ /dev/null
@@ -1,52 +0,0 @@
-{# This template adds attributes unique
-   to  scatterChart #}
-
-{% extends "content.html" %}
-{% block body %}
-
-    {% block data %}
-    {{super()}}
-    {% endblock data %}
-
-    {% block init %}
-    {{super()}}
-    {% endblock init %}
-
-    {% block axes %}
-    {{super()}}
-    {% endblock axes %}
-
-    {% if chart.x_axis_format == 'AM_PM' %}
-        function get_am_pm(d){
-            if (d > 12) {
-                d = d - 12; return (String(d) + 'PM');
-            }
-            else {
-                return (String(d) + 'AM');
-            }
-        };
-    {% endif %}
-
-    {% block legend %}
-    {{super()}}
-    {% endblock legend %}
-
-    {% block custoattr %}
-    {{super()}}
-    {% endblock custoattr %}
-
-    {% block inject %}
-
-    chart
-        .showDistX(true)
-        .showDistY(true)
-        .color(d3.scale.category10().range());
-
-    {{ super() }}
-    {% endblock inject %}
-
-    {% block close %}
-    {{ super() }}
-    {% endblock close %}
-
-{% endblock body %}
diff --git a/airflow/_vendor/nvd3/templates/stackedareachart.html b/airflow/_vendor/nvd3/templates/stackedareachart.html
deleted file mode 100644
index b70833d..0000000
--- a/airflow/_vendor/nvd3/templates/stackedareachart.html
+++ /dev/null
@@ -1,7 +0,0 @@
-{# This is a dummy template, we can use that template to add attributes unique
-   to stackedareachart #}
-
-{% extends "content.html" %}
-{% block body %}
-    {{ super() }}
-{% endblock body %}
diff --git a/airflow/_vendor/nvd3/translator.py b/airflow/_vendor/nvd3/translator.py
deleted file mode 100644
index ffde2c2..0000000
--- a/airflow/_vendor/nvd3/translator.py
+++ /dev/null
@@ -1,71 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: UTF-8 -*-
-
-
-class Tag(object):
-    """Tag class"""
-
-    def __init__(self, content=None):
-        self.content = content
-        self.attrs = ' '.join(['%s="%s"' % (attr, value)
-                               for attr, value in self.attrs])
-
-    def __str__(self):
-        return '<%s%s>\n    %s\n</%s>' % (self.name,
-                                          ' ' + self.attrs if self.attrs else '',
-                                          self.content,
-                                          self.name)
-
-
-class ScriptTag(Tag):
-    name = 'script'
-    attrs = (('type', 'text/javascript'),)
-
-
-class AnonymousFunction(object):
-    def __init__(self, arguments, content):
-        self.arguments = arguments
-        self.content = content
-
-    def __str__(self):
-        return 'function(%s) { %s }' % (self.arguments, self.content)
-
-
-class Function(object):
-
-    def __init__(self, name):
-        self.name = name
-        self._calls = []
-
-    def __str__(self):
-        operations = [self.name]
-        operations.extend(str(call) for call in self._calls)
-        return '%s' % ('.'.join(operations),)
-
-    def __getattr__(self, attr):
-        self._calls.append(attr)
-        return self
-
-    def __call__(self, *args):
-        if not args:
-            self._calls[-1] = self._calls[-1] + '()'
-        else:
-            arguments = ','.join([str(arg) for arg in args])
-            self._calls[-1] = self._calls[-1] + '(%s)' % (arguments,)
-        return self
-
-
-class Assignment(object):
-
-    def __init__(self, key, value, scoped=True):
-        self.key = key
-        self.value = value
-        self.scoped = scoped
-
-    def __str__(self):
-        return '%s%s = %s;' % ('var ' if self.scoped else '', self.key, self.value)
-
-
-def indent(func):
-    # TODO: Add indents to function str
-    return str(func)
diff --git a/airflow/_vendor/slugify/LICENSE b/airflow/_vendor/slugify/LICENSE
deleted file mode 100644
index 82af695..0000000
--- a/airflow/_vendor/slugify/LICENSE
+++ /dev/null
@@ -1,21 +0,0 @@
-The MIT License
-
-Copyright (c) Val Neekman @ Neekware Inc. http://neekware.com
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in
-all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-THE SOFTWARE.
diff --git a/airflow/_vendor/slugify/__init__.py b/airflow/_vendor/slugify/__init__.py
deleted file mode 100644
index 7358b99..0000000
--- a/airflow/_vendor/slugify/__init__.py
+++ /dev/null
@@ -1,6 +0,0 @@
-from .slugify import *
-
-
-__author__ = 'Val Neekman @ Neekware Inc. [@vneekman]'
-__description__ = 'A Python slugify application that also handles Unicode'
-__version__ = '2.0.1'
diff --git a/airflow/_vendor/slugify/slugify.py b/airflow/_vendor/slugify/slugify.py
deleted file mode 100644
index 0e9886d..0000000
--- a/airflow/_vendor/slugify/slugify.py
+++ /dev/null
@@ -1,185 +0,0 @@
-import re
-import unicodedata
-import types
-import sys
-
-try:
-    from htmlentitydefs import name2codepoint
-    _unicode = unicode
-    _unicode_type = types.UnicodeType
-except ImportError:
-    from html.entities import name2codepoint
-    _unicode = str
-    _unicode_type = str
-    unichr = chr
-
-import text_unidecode as unidecode
-
-__all__ = ['slugify', 'smart_truncate']
-
-
-CHAR_ENTITY_PATTERN = re.compile(r'&(%s);' % '|'.join(name2codepoint))
-DECIMAL_PATTERN = re.compile(r'&#(\d+);')
-HEX_PATTERN = re.compile(r'&#x([\da-fA-F]+);')
-QUOTE_PATTERN = re.compile(r'[\']+')
-ALLOWED_CHARS_PATTERN = re.compile(r'[^-a-z0-9]+')
-ALLOWED_CHARS_PATTERN_WITH_UPPERCASE = re.compile(r'[^-a-zA-Z0-9]+')
-DUPLICATE_DASH_PATTERN = re.compile(r'-{2,}')
-NUMBERS_PATTERN = re.compile(r'(?<=\d),(?=\d)')
-DEFAULT_SEPARATOR = '-'
-
-
-def smart_truncate(string, max_length=0, word_boundary=False, separator=' ', save_order=False):
-    """
-    Truncate a string.
-    :param string (str): string for modification
-    :param max_length (int): output string length
-    :param word_boundary (bool):
-    :param save_order (bool): if True then word order of output string is like input string
-    :param separator (str): separator between words
-    :return:
-    """
-
-    string = string.strip(separator)
-
-    if not max_length:
-        return string
-
-    if len(string) < max_length:
-        return string
-
-    if not word_boundary:
-        return string[:max_length].strip(separator)
-
-    if separator not in string:
-        return string[:max_length]
-
-    truncated = ''
-    for word in string.split(separator):
-        if word:
-            next_len = len(truncated) + len(word)
-            if next_len < max_length:
-                truncated += '{0}{1}'.format(word, separator)
-            elif next_len == max_length:
-                truncated += '{0}'.format(word)
-                break
-            else:
-                if save_order:
-                    break
-    if not truncated: # pragma: no cover
-        truncated = string[:max_length]
-    return truncated.strip(separator)
-
-
-def slugify(text, entities=True, decimal=True, hexadecimal=True, max_length=0, word_boundary=False,
-            separator=DEFAULT_SEPARATOR, save_order=False, stopwords=(), regex_pattern=None, lowercase=True,
-            replacements=()):
-    """
-    Make a slug from the given text.
-    :param text (str): initial text
-    :param entities (bool):
-    :param decimal (bool):
-    :param hexadecimal (bool):
-    :param max_length (int): output string length
-    :param word_boundary (bool):
-    :param save_order (bool): if parameter is True and max_length > 0 return whole words in the initial order
-    :param separator (str): separator between words
-    :param stopwords (iterable): words to discount
-    :param regex_pattern (str): regex pattern for allowed characters
-    :param lowercase (bool): activate case sensitivity by setting it to False
-    :param replacements (iterable): list of replacement rules e.g. [['|', 'or'], ['%', 'percent']]
-    :return (str):
-    """
-
-    # user-specific replacements
-    if replacements:
-        for old, new in replacements:
-            text = text.replace(old, new)
-
-    # ensure text is unicode
-    if not isinstance(text, _unicode_type):
-        text = _unicode(text, 'utf-8', 'ignore')
-
-    # replace quotes with dashes - pre-process
-    text = QUOTE_PATTERN.sub(DEFAULT_SEPARATOR, text)
-
-    # decode unicode
-    text = unidecode.unidecode(text)
-
-    # ensure text is still in unicode
-    if not isinstance(text, _unicode_type):
-        text = _unicode(text, 'utf-8', 'ignore')
-
-    # character entity reference
-    if entities:
-        text = CHAR_ENTITY_PATTERN.sub(lambda m: unichr(name2codepoint[m.group(1)]), text)
-
-    # decimal character reference
-    if decimal:
-        try:
-            text = DECIMAL_PATTERN.sub(lambda m: unichr(int(m.group(1))), text)
-        except Exception:
-            pass
-
-    # hexadecimal character reference
-    if hexadecimal:
-        try:
-            text = HEX_PATTERN.sub(lambda m: unichr(int(m.group(1), 16)), text)
-        except Exception:
-            pass
-
-    # translate
-    text = unicodedata.normalize('NFKD', text)
-    if sys.version_info < (3,):
-        text = text.encode('ascii', 'ignore')
-
-    # make the text lowercase (optional)
-    if lowercase:
-        text = text.lower()
-
-    # remove generated quotes -- post-process
-    text = QUOTE_PATTERN.sub('', text)
-
-    # cleanup numbers
-    text = NUMBERS_PATTERN.sub('', text)
-
-    # replace all other unwanted characters
-    if lowercase:
-        pattern = regex_pattern or ALLOWED_CHARS_PATTERN
-    else:
-        pattern = regex_pattern or ALLOWED_CHARS_PATTERN_WITH_UPPERCASE
-    text = re.sub(pattern, DEFAULT_SEPARATOR, text)
-
-    # remove redundant
-    text = DUPLICATE_DASH_PATTERN.sub(DEFAULT_SEPARATOR, text).strip(DEFAULT_SEPARATOR)
-
-    # remove stopwords
-    if stopwords:
-        if lowercase:
-            stopwords_lower = [s.lower() for s in stopwords]
-            words = [w for w in text.split(DEFAULT_SEPARATOR) if w not in stopwords_lower]
-        else:
-            words = [w for w in text.split(DEFAULT_SEPARATOR) if w not in stopwords]
-        text = DEFAULT_SEPARATOR.join(words)
-
-    # finalize user-specific replacements
-    if replacements:
-        for old, new in replacements:
-            text = text.replace(old, new)
-
-    # smart truncate if requested
-    if max_length > 0:
-        text = smart_truncate(text, max_length, word_boundary, DEFAULT_SEPARATOR, save_order)
-
-    if separator != DEFAULT_SEPARATOR:
-        text = text.replace(DEFAULT_SEPARATOR, separator)
-
-    return text
-
-
-def main(): # pragma: no cover
-    if len(sys.argv) < 2:
-        print("Usage %s TEXT TO SLUGIFY" % sys.argv[0])
-    else:
-        text = ' '.join(sys.argv[1:])
-        print(slugify(text))
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 62d2d4b..a069f17 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -35,8 +35,8 @@ from functools import wraps
 from textwrap import dedent
 
 import markdown
-import pendulum
 import sqlalchemy as sqla
+import pendulum
 from flask import (
     abort, jsonify, redirect, url_for, request, Markup, Response,
     current_app, render_template, make_response)
@@ -52,8 +52,8 @@ from jinja2 import escape
 from jinja2.sandbox import ImmutableSandboxedEnvironment
 from past.builtins import basestring
 from pygments import highlight, lexers
-from pygments.formatters import HtmlFormatter
 import six
+from pygments.formatters.html import HtmlFormatter
 from six.moves.urllib.parse import quote, unquote
 
 from sqlalchemy import or_, desc, and_, union_all
@@ -61,6 +61,8 @@ from wtforms import (
     Form, SelectField, TextAreaField, PasswordField,
     StringField, IntegerField, validators)
 
+import nvd3
+
 import airflow
 from airflow import LoggingMixin, configuration
 from airflow.configuration import conf
@@ -83,7 +85,6 @@ from airflow.utils.helpers import alchemy_to_dict, render_log_filename
 from airflow.utils.net import get_hostname
 from airflow.utils.state import State
 from airflow.utils.timezone import datetime
-from airflow._vendor import nvd3
 from airflow.www import utils as wwwutils
 from airflow.www.forms import (DateTimeForm, DateTimeWithNumRunsForm,
                                DateTimeWithNumRunsWithDagRunsForm)
diff --git a/requirements/requirements-python3.6.txt b/requirements/requirements-python3.6.txt
index 979fe59..e020794 100644
--- a/requirements/requirements-python3.6.txt
+++ b/requirements/requirements-python3.6.txt
@@ -251,6 +251,8 @@ python-editor==1.0.4
 python-http-client==3.2.7
 python-jenkins==1.7.0
 python-jose==3.1.0
+python-nvd3==0.15.0
+python-slugify==4.0.0
 python3-openid==3.1.0
 pytz==2020.1
 pytzdata==2019.3
@@ -298,7 +300,7 @@ sshtunnel==0.1.5
 tabulate==0.8.7
 tenacity==4.12.0
 termcolor==1.1.0
-text-unidecode==1.2
+text-unidecode==1.3
 textwrap3==0.9.2
 thrift-sasl==0.4.2
 thrift==0.13.0
diff --git a/requirements/requirements-python3.7.txt b/requirements/requirements-python3.7.txt
index a8e7890..d0dee9e 100644
--- a/requirements/requirements-python3.7.txt
+++ b/requirements/requirements-python3.7.txt
@@ -250,6 +250,8 @@ python-editor==1.0.4
 python-http-client==3.2.7
 python-jenkins==1.7.0
 python-jose==3.1.0
+python-nvd3==0.15.0
+python-slugify==4.0.0
 python3-openid==3.1.0
 pytz==2020.1
 pytzdata==2019.3
@@ -297,7 +299,7 @@ sshtunnel==0.1.5
 tabulate==0.8.7
 tenacity==4.12.0
 termcolor==1.1.0
-text-unidecode==1.2
+text-unidecode==1.3
 textwrap3==0.9.2
 thrift-sasl==0.4.2
 thrift==0.13.0
diff --git a/scripts/ci/in_container/_in_container_utils.sh b/scripts/ci/in_container/_in_container_utils.sh
index 7e1fd4a..9197a04 100644
--- a/scripts/ci/in_container/_in_container_utils.sh
+++ b/scripts/ci/in_container/_in_container_utils.sh
@@ -109,7 +109,7 @@ function in_container_basic_sanity_check() {
     in_container_cleanup_pycache
 }
 
-export DISABLE_CHECKS_FOR_TESTS="missing-docstring,no-self-use,too-many-public-methods,protected-access"
+export DISABLE_CHECKS_FOR_TESTS="missing-docstring,no-self-use,too-many-public-methods,protected-access,do-not-use-asserts"
 
 function start_output_heartbeat() {
     MESSAGE=${1:-"Still working!"}
diff --git a/setup.cfg b/setup.cfg
index ef06e54..6a7acd2 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -23,9 +23,28 @@ author = Apache Airflow PMC
 author-email = dev@airflow.apache.org
 license = Apache License, Version 2.0
 license_files =
-  LICENSE
-  licenses/*
-  NOTICE
+   LICENSE
+   NOTICE
+# Start of licenses generated automatically
+   licenses/LICENSE-bootstrap-toggle.txt
+   licenses/LICENSE-bootstrap.txt
+   licenses/LICENSE-bootstrap3-typeahead.txt
+   licenses/LICENSE-d3-tip.txt
+   licenses/LICENSE-d3js.txt
+   licenses/LICENSE-dagre-d3.txt
+   licenses/LICENSE-datatables.txt
+   licenses/LICENSE-elasticmock.txt
+   licenses/LICENSE-eonasdan-bootstrap-datetimepicker.txt
+   licenses/LICENSE-flask-kerberos.txt
+   licenses/LICENSE-hue.txt
+   licenses/LICENSE-jqclock.txt
+   licenses/LICENSE-jquery.txt
+   licenses/LICENSE-moment-strftime.txt
+   licenses/LICENSE-moment.txt
+   licenses/LICENSE-normalize.txt
+   licenses/LICENSE-python-nvd3.txt
+   licenses/LICENSE-python-slugify.txt
+# End of licences generated automatically
 
 [bdist_wheel]
 universal=1
@@ -41,8 +60,5 @@ ignore_missing_imports = True
 [mypy-airflow.migrations.*]
 ignore_errors = True
 
-[mypy-airflow._vendor.*]
-ignore_errors = True
-
 [mypy-airflow.contrib.auth.*]
 ignore_errors = True
diff --git a/setup.py b/setup.py
index 459fa1d..8178294 100644
--- a/setup.py
+++ b/setup.py
@@ -578,6 +578,8 @@ INSTALL_REQUIREMENTS = [
     'pygments>=2.0.1, <3.0',
     'python-daemon>=2.1.1, <2.2',
     'python-dateutil>=2.3, <3',
+    'python-nvd3~=0.15.0',
+    'python-slugify>=3.0.0,<5.0',
     'requests>=2.20.0, <3',
     'setproctitle>=1.1.8, <2',
     'sqlalchemy~=1.3',
@@ -586,7 +588,6 @@ INSTALL_REQUIREMENTS = [
     'tabulate>=0.7.5, <0.9',
     'tenacity==4.12.0',
     'termcolor==1.1.0',
-    'text-unidecode==1.2',
     'thrift>=0.9.2',
     'typing;python_version<"3.5"',
     'typing-extensions>=3.7.4;python_version<"3.8"',


[airflow] 29/36: Restrict google-cloud-texttospeach to Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 86eeb00f47a93b3ab2abc7e39c29c20d04e9dccb
Author: Ash Berlin-Taylor <as...@firemirror.com>
AuthorDate: Thu Jun 4 14:15:24 2020 +0100

    Restrict google-cloud-texttospeach to <v2 (#9137)
    
    Version 2 of this library causes the following pylint errors:
    
    > airflow/providers/google/cloud/hooks/text_to_speech.py:59:27: E1123: Unexpected keyword argument 'client_info' in constructor call (unexpected-keyword-arg)
    > airflow/providers/google/cloud/hooks/text_to_speech.py:99:15: E1123: Unexpected keyword argument 'input_' in method call (unexpected-keyword-arg)
    
    (cherry picked from commit aa8990031d15b5c5b66fd9fb4b483d938e656bde)
---
 setup.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/setup.py b/setup.py
index 5b279f0..459fa1d 100644
--- a/setup.py
+++ b/setup.py
@@ -260,7 +260,7 @@ gcp = [
     'google-cloud-spanner>=1.10.0',
     'google-cloud-speech>=0.36.3',
     'google-cloud-storage>=1.16',
-    'google-cloud-texttospeech>=0.4.0',
+    'google-cloud-texttospeech>=0.4.0,<2',
     'google-cloud-translate>=1.3.3',
     'google-cloud-videointelligence>=1.7.0',
     'google-cloud-vision>=0.35.2',


[airflow] 18/36: Enable configurable git sync depth (#9094)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 64194eec4efdcbdc151e47979f6ae64ed4820ed8
Author: mgorsk1 <go...@gmail.com>
AuthorDate: Tue Jun 2 06:36:03 2020 +0000

    Enable configurable git sync depth  (#9094)
    
    Enable configurable git sync depth
    
    (cherry picked from commit 93e924d3fab6137fda097716a9078705621eb358)
---
 airflow/config_templates/config.yml                               | 8 ++++++++
 airflow/config_templates/default_airflow.cfg                      | 4 ++++
 airflow/executors/kubernetes_executor.py                          | 2 ++
 airflow/kubernetes/worker_configuration.py                        | 2 +-
 .../kubernetes/app/templates/configmaps.template.yaml             | 1 +
 5 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/airflow/config_templates/config.yml b/airflow/config_templates/config.yml
index 26cfae8..a7fa7a6 100644
--- a/airflow/config_templates/config.yml
+++ b/airflow/config_templates/config.yml
@@ -1893,6 +1893,14 @@
       type: string
       example: ~
       default: ""
+    - name: git_sync_depth
+      description: |
+        Use a shallow clone with a history truncated to the specified number of commits.
+        0 - do not use shallow clone.
+      version_added: 1.10.11
+      type: string
+      example: ~
+      default: "1"
     - name: git_subpath
       description: ~
       version_added: ~
diff --git a/airflow/config_templates/default_airflow.cfg b/airflow/config_templates/default_airflow.cfg
index 5f240de..27fe92a 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -881,6 +881,10 @@ env_from_secret_ref =
 # Git credentials and repository for DAGs mounted via Git (mutually exclusive with volume claim)
 git_repo =
 git_branch =
+
+# Use a shallow clone with a history truncated to the specified number of commits.
+# 0 - do not use shallow clone.
+git_sync_depth = 1
 git_subpath =
 
 # The specific rev or hash the git_sync init container will checkout
diff --git a/airflow/executors/kubernetes_executor.py b/airflow/executors/kubernetes_executor.py
index 6944274..cf40e43 100644
--- a/airflow/executors/kubernetes_executor.py
+++ b/airflow/executors/kubernetes_executor.py
@@ -101,6 +101,8 @@ class KubeConfig:
         self.git_repo = conf.get(self.kubernetes_section, 'git_repo')
         # The branch of the repository to be checked out
         self.git_branch = conf.get(self.kubernetes_section, 'git_branch')
+        # Clone depth for git sync
+        self.git_sync_depth = conf.get(self.kubernetes_section, 'git_sync_depth')
         # Optionally, the directory in the git repository containing the dags
         self.git_subpath = conf.get(self.kubernetes_section, 'git_subpath')
         # Optionally, the root directory for git operations
diff --git a/airflow/kubernetes/worker_configuration.py b/airflow/kubernetes/worker_configuration.py
index 41a65b3..bed1ac2 100644
--- a/airflow/kubernetes/worker_configuration.py
+++ b/airflow/kubernetes/worker_configuration.py
@@ -68,7 +68,7 @@ class WorkerConfiguration(LoggingMixin):
             value=self.kube_config.git_sync_dest
         ), k8s.V1EnvVar(
             name='GIT_SYNC_DEPTH',
-            value='1'
+            value=self.kube_config.git_sync_depth
         ), k8s.V1EnvVar(
             name='GIT_SYNC_ONE_TIME',
             value='true'
diff --git a/scripts/ci/in_container/kubernetes/app/templates/configmaps.template.yaml b/scripts/ci/in_container/kubernetes/app/templates/configmaps.template.yaml
index 868da7c..bb942bc 100644
--- a/scripts/ci/in_container/kubernetes/app/templates/configmaps.template.yaml
+++ b/scripts/ci/in_container/kubernetes/app/templates/configmaps.template.yaml
@@ -185,6 +185,7 @@ data:
     dags_in_image = False
     git_repo = https://github.com/{{CONFIGMAP_GIT_REPO}}.git
     git_branch = {{CONFIGMAP_BRANCH}}
+    git_sync_depth = "1"
     git_subpath = airflow/contrib/example_dags/
     git_user =
     git_password =


[airflow] 26/36: Fix INTEGRATIONS[*]: unbound variable error in breeze (#9135)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 945b0e63d3e7bcb4dfe67644351a648ab2b27d28
Author: Tomek Urbaszek <tu...@gmail.com>
AuthorDate: Thu Jun 4 11:44:26 2020 +0200

    Fix INTEGRATIONS[*]: unbound variable error in breeze (#9135)
    
    
    (cherry picked from commit 1fe804a30ad69619ed24f9a5f32c2e67fadbc717)
---
 breeze | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/breeze b/breeze
index 18a06fd..a8e9295 100755
--- a/breeze
+++ b/breeze
@@ -422,7 +422,7 @@ EOF
 
 EOF
         fi
-
+        set +u
         if [[ ${INTEGRATIONS[*]} == *"kubernetes"* ]]; then
             cat <<EOF
 
@@ -433,6 +433,7 @@ EOF
    Cluster operation:   ${KIND_CLUSTER_OPERATION}
 EOF
         fi
+        set -u
     fi
 }
 


[airflow] 09/36: Re-run all tests when Dockerfile or Github worflow change (#8924)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 431b52ada9502e8c379f58995b2edb1a01df338c
Author: Ash Berlin-Taylor <as...@firemirror.com>
AuthorDate: Wed May 20 18:49:28 2020 +0100

    Re-run all tests when Dockerfile or Github worflow change (#8924)
    
    Fixes #8921
    
    (cherry picked from commit 51d955787b009b9e3a88f3e9b4ca1a3933a061f0)
---
 .github/workflows/ci.yml             | 28 +++++++++++++++++++++++++---
 scripts/ci/ci_count_changed_files.sh |  4 ++--
 2 files changed, 27 insertions(+), 5 deletions(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 37eca52..bcc1505 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -134,11 +134,27 @@ jobs:
         run: ./scripts/ci/ci_build_production_images.sh
       - name: "Tests"
         run: ./scripts/ci/ci_run_airflow_testing.sh
+
+  pyfiles:
+    timeout-minutes: 10
+    name: "Count changed important files"
+    runs-on: ubuntu-latest
+    outputs:
+      count: ${{ steps.pyfiles.outputs.count }}
+    steps:
+      - uses: actions/checkout@master
+      - name: "Get count of changed python files"
+        run: |
+          set +e
+          ./scripts/ci/ci_count_changed_files.sh ${GITHUB_SHA} '\.py$|.github/workflows/|^Dockerfile'
+          echo "::set-output name=count::$?"
+        id: pyfiles
+
   tests-postgres:
     timeout-minutes: 80
     name: "${{matrix.test-type}}:Pg${{matrix.postgres-version}},Py${{matrix.python-version}}"
     runs-on: ubuntu-latest
-    needs: [static-checks]
+    needs: [static-checks, pyfiles]
     strategy:
       matrix:
         python-version: [2.7, 3.5, 3.6, 3.7]
@@ -152,6 +168,8 @@ jobs:
       RUN_TESTS: "true"
       CI_JOB_TYPE: "Tests"
       TEST_TYPE: ${{ matrix.test-type }}
+    # For pull requests only run tests when python files changed
+    if: needs.pyfiles.outputs.count != '0' || github.event_name != 'pull_request'
     steps:
       - uses: actions/checkout@master
       - uses: actions/setup-python@v1
@@ -168,7 +186,7 @@ jobs:
     timeout-minutes: 80
     name: "${{matrix.test-type}}:MySQL${{matrix.mysql-version}}, Py${{matrix.python-version}}"
     runs-on: ubuntu-latest
-    needs: [static-checks]
+    needs: [static-checks, pyfiles]
     strategy:
       matrix:
         python-version: [2.7, 3.5, 3.6, 3.7]
@@ -182,6 +200,8 @@ jobs:
       RUN_TESTS: "true"
       CI_JOB_TYPE: "Tests"
       TEST_TYPE: ${{ matrix.test-type }}
+    # For pull requests only run tests when python files changed
+    if: needs.pyfiles.outputs.count != '0' || github.event_name != 'pull_request'
     steps:
       - uses: actions/checkout@master
       - uses: actions/setup-python@v1
@@ -198,7 +218,7 @@ jobs:
     timeout-minutes: 80
     name: "${{matrix.test-type}}:Sqlite Py${{matrix.python-version}}"
     runs-on: ubuntu-latest
-    needs: [static-checks]
+    needs: [static-checks, pyfiles]
     strategy:
       matrix:
         python-version: [2.7, 3.5, 3.6, 3.7]
@@ -210,6 +230,8 @@ jobs:
       TEST_TYPE: ${{ matrix.test-type }}
       RUN_TESTS: "true"
       CI_JOB_TYPE: "Tests"
+    # For pull requests only run tests when python files changed
+    if: needs.pyfiles.outputs.count != '0' || github.event_name != 'pull_request'
     steps:
       - uses: actions/checkout@master
       - uses: actions/setup-python@v1
diff --git a/scripts/ci/ci_count_changed_files.sh b/scripts/ci/ci_count_changed_files.sh
index 0713dc1..0e72f72 100755
--- a/scripts/ci/ci_count_changed_files.sh
+++ b/scripts/ci/ci_count_changed_files.sh
@@ -42,13 +42,13 @@ echo
 echo
 echo "Changed files matching the ${2} pattern"
 echo
-echo "${CHANGED_FILES}" | grep "${2}" || true
+echo "${CHANGED_FILES}" | grep -E "${2}" || true
 echo
 
 echo
 echo "Count changed files matching the ${2} pattern"
 echo
-COUNT_CHANGED_FILES=$(echo "${CHANGED_FILES}" | grep -c "${2}" || true)
+COUNT_CHANGED_FILES=$(echo "${CHANGED_FILES}" | grep -c -E "${2}" || true)
 echo "${COUNT_CHANGED_FILES}"
 echo
 


[airflow] 11/36: Python base images are stored in cache (#8943)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 40d650b9a9dc31058dc707b96afec94401578362
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Thu May 21 13:39:07 2020 +0200

    Python base images are stored in cache (#8943)
    
    All PRs will used cached "latest good" version of the python
    base images from our GitHub registry. The python versions in
    the Github Registry will only get updated after a master
    build (which pulls latest Python image from DockerHub) builds
    and passes test correctly.
    
    This is to avoid problems that we had recently with Python
    patchlevel releases breaking our Docker builds.
    
    (cherry picked from commit 41481bb40218ce5c394db9b88de2861728279e9e)
---
 .github/workflows/ci.yml |  1 +
 CI.rst                   | 27 +++++++++++++++++++++------
 scripts/ci/_utils.sh     | 28 +++++++++++++++++++++++++---
 3 files changed, 47 insertions(+), 9 deletions(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index bcc1505..e4f16e7 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -344,6 +344,7 @@ jobs:
       matrix:
         python-version: [2.7, 3.5, 3.6, 3.7]
     env:
+      PULL_PYTHON_BASE_IMAGES_FROM_CACHE: "false"
       PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
       CI_JOB_TYPE: "Prod image"
     steps:
diff --git a/CI.rst b/CI.rst
index 76523b3..6985f06 100644
--- a/CI.rst
+++ b/CI.rst
@@ -76,7 +76,9 @@ purpose and context.
   (container registry, code repository). This is necessary as the code in those PRs (including CI job
   definition) might be modified by people who are not committers for the Apache Airflow Code Repository.
   The main purpose of those jobs is to check if PR builds cleanly, if the test run properly and if
-  the PR is ready to review and merge.
+  the PR is ready to review and merge. The runs are using cached images from the Private GitHub registry -
+  CI, Production Images as well as base Python images that are also cached in the Private GitHub registry.
+
 * **Direct Push/Merge Run** - Those runs are results of direct pushes done by the committers or as result
   of merge of a Pull Request by the committers. Those runs execute in the context of the Apache Airflow
   Code Repository and have also write permission for GitHub resources (container registry, code repository).
@@ -85,7 +87,16 @@ purpose and context.
   multiple PRs might cause build and test failures after merge even if they do not fail in isolation. Also
   those runs are already reviewed and confirmed by the committers so they can be used to do some housekeeping
   - for now they are pushing most recent image build in the PR to the Github Private Registry - which is our
-  image cache for all the builds.
+  image cache for all the builds. Another purpose of those runs is to refresh latest Python base images.
+  Python base images are refreshed with varying frequency (once every few months usually but sometimes
+  several times per week) with the latest security and bug fixes. Those patch level images releases can
+  occasionally break Airflow builds (specifically Docker image builds based on those images) therefore
+  in PRs we always use latest "good" python image that we store in the private GitHub cache. The direct
+  push/master builds are not using registry cache to pull the python images - they are directly
+  pulling the images from DockerHub, therefore they will try the latest images after they are released
+  and in case they are fine, CI Docker image is build and tests are passing - those jobs will push the base
+  images to the private GitHub Registry so that they be used by subsequent PR runs.
+
 * **Scheduled Run** - those runs are results of (nightly) triggered jobs - only for well-defined branches:
   ``master`` and ``v1-10-test`` they execute nightly. Their main purpose is to check if there was no impact
   of external dependency changes on the Apache Airflow code (for example transitive dependencies released
@@ -120,11 +131,15 @@ Those jobs often have matrix run strategy which runs several different variation
 +---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
 | Quarantined tests         | Those are tests that are flaky and we need to fix them                                                         | Yes (if pyfiles count >0)          | Yes                             | Yes *                                                                |
 +---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
-| Requirements              | Checks if requirement constraints in the code are up-to-date                                                   | Yes (fails if missing requirement) | YesFails if missing requirement | Yes (Eager dependency upgradeDoes not fail for changed requirements) |
+| Requirements              | Checks if requirement constraints in the code are up-to-date                                                   | Yes (fails if missing requirement) | Yes (fails missing requirement) | Yes (Eager dependency upgrade - does not fail changed requirements)  |
++---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
+| Pull python from cache    | Pulls Python base images from Github Private Image registry to keep the last good python image used in PRs     | Yes                                | No                              | -                                                                    |
++---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
+| Push python to cache      | Pushes Python base images to Github Private Image registry - checks if latest image is fine and pushes if so   | No                                 | Yes                             | -                                                                    |
 +---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
-| Push Prod image           | Pushes production images to GitHub Private Image RegistryThis is to cache the build images for following runs. | -                                  | Yes                             | -                                                                    |
+| Push Prod image           | Pushes production images to GitHub Private Image Registry to cache the build images for following runs         | -                                  | Yes                             | -                                                                    |
 +---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
-| Push CI image             | Pushes CI images to GitHub Private Image RegistryThis is to cache the build images for following runs.         | -                                  | Yes                             | -                                                                    |
+| Push CI image             | Pushes CI images to GitHub Private Image Registry to cache the build images for following runs                 | -                                  | Yes                             | -                                                                    |
 +---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
-| Tag Repo nightly          | Tags the repository with nightly tagIt is a lightweight tag that moves nightly                                 | -                                  | -                               | Yes.Triggers DockerHub build for public registry                     |
+| Tag Repo nightly          | Tags the repository with nightly tagIt is a lightweight tag that moves nightly                                 | -                                  | -                               | Yes. Triggers DockerHub build for public registry                    |
 +---------------------------+----------------------------------------------------------------------------------------------------------------+------------------------------------+---------------------------------+----------------------------------------------------------------------+
diff --git a/scripts/ci/_utils.sh b/scripts/ci/_utils.sh
index 8ac07ab..7bbd2dd 100644
--- a/scripts/ci/_utils.sh
+++ b/scripts/ci/_utils.sh
@@ -96,6 +96,9 @@ function initialize_common_environment {
     # --push-images flag is specified
     export PUSH_IMAGES=${PUSH_IMAGES:="false"}
 
+    # Whether base python images should be pulled from cache
+    export PULL_PYTHON_BASE_IMAGES_FROM_CACHE=${PULL_PYTHON_BASE_IMAGES_FROM_CACHE:="true"}
+
     # Disable writing .pyc files - slightly slower imports but not messing around when switching
     # Python version and avoids problems with root-owned .pyc files in host
     export PYTHONDONTWRITEBYTECODE=${PYTHONDONTWRITEBYTECODE:="true"}
@@ -1187,7 +1190,11 @@ function pull_ci_image_if_needed() {
 Docker pulling ${PYTHON_BASE_IMAGE}.
                     " > "${DETECTED_TERMINAL}"
             fi
-            verbose_docker pull "${PYTHON_BASE_IMAGE}" | tee -a "${OUTPUT_LOG}"
+            if [[ ${PULL_PYTHON_BASE_IMAGES_FROM_CACHE:="true"} == "true" ]]; then
+                pull_image_possibly_from_cache "${PYTHON_BASE_IMAGE}" "${CACHED_PYTHON_BASE_IMAGE}"
+            else
+                verbose_docker pull "${PYTHON_BASE_IMAGE}" | tee -a "${OUTPUT_LOG}"
+            fi
             echo
         fi
         pull_image_possibly_from_cache "${AIRFLOW_CI_IMAGE}" "${CACHED_AIRFLOW_CI_IMAGE}"
@@ -1205,7 +1212,11 @@ function pull_prod_images_if_needed() {
             echo
             echo "Force pull base image ${PYTHON_BASE_IMAGE}"
             echo
-            verbose_docker pull "${PYTHON_BASE_IMAGE}" | tee -a "${OUTPUT_LOG}"
+            if [[ ${PULL_PYTHON_BASE_IMAGES_FROM_CACHE:="true"} == "true" ]]; then
+                pull_image_possibly_from_cache "${PYTHON_BASE_IMAGE}" "${CACHED_PYTHON_BASE_IMAGE}"
+            else
+                verbose_docker pull "${PYTHON_BASE_IMAGE}" | tee -a "${OUTPUT_LOG}"
+            fi
             echo
         fi
         # "Build" segment of production image
@@ -1444,8 +1455,10 @@ function prepare_ci_build() {
             --password-stdin \
             "${CACHE_REGISTRY}"
         export CACHED_AIRFLOW_CI_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_CI_BASE_TAG}"
+        export CACHED_PYTHON_BASE_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
     else
         export CACHED_AIRFLOW_CI_IMAGE=""
+        export CACHED_PYTHON_BASE_IMAGE=""
     fi
     export AIRFLOW_BUILD_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}/${AIRFLOW_CI_BASE_TAG}"
     export AIRFLOW_CI_IMAGE_DEFAULT="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${DEFAULT_BRANCH}-ci"
@@ -1527,9 +1540,11 @@ function prepare_prod_build() {
             "${CACHE_REGISTRY}"
         export CACHED_AIRFLOW_PROD_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_PROD_BASE_TAG}"
         export CACHED_AIRFLOW_PROD_BUILD_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_PROD_BASE_TAG}-build"
+        export CACHED_PYTHON_BASE_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
     else
         export CACHED_AIRFLOW_PROD_IMAGE=""
         export CACHED_AIRFLOW_PROD_BUILD_IMAGE=""
+        export CACHED_PYTHON_BASE_IMAGE=""
     fi
 
     if [[ "${INSTALL_AIRFLOW_REFERENCE:=}" != "" ]]; then
@@ -1556,7 +1571,7 @@ function prepare_prod_build() {
 }
 
 # Pushes Ci image and it's manifest to the registry. In case the image was taken from cache registry
-# it is also pushed to the cache, not to the main registry. Manifest is only pushed to the main registry
+# it is pushed to the cache, not to the main registry. Manifest is only pushed to the main registry
 function push_ci_image() {
     if [[ ${CACHED_AIRFLOW_CI_IMAGE:=} != "" ]]; then
         verbose_docker tag "${AIRFLOW_CI_IMAGE}" "${CACHED_AIRFLOW_CI_IMAGE}"
@@ -1573,6 +1588,11 @@ function push_ci_image() {
             verbose_docker push "${DEFAULT_IMAGE}"
         fi
     fi
+    if [[ ${CACHED_PYTHON_BASE_IMAGE} != "" ]]; then
+        verbose_docker tag "${PYTHON_BASE_IMAGE}" "${CACHED_PYTHON_BASE_IMAGE}"
+        verbose_docker push "${CACHED_PYTHON_BASE_IMAGE}"
+    fi
+
 }
 
 # Pushes PROD image to the registry. In case the image was taken from cache registry
@@ -1595,6 +1615,8 @@ function push_prod_images() {
     if [[ -n ${DEFAULT_IMAGE:=""} && ${CACHED_AIRFLOW_PROD_IMAGE} == "" ]]; then
         verbose_docker push "${DEFAULT_IMAGE}"
     fi
+
+    # we do not need to push PYTHON base image here - they are already pushed in the CI push
 }
 
 # Docker command to generate constraint requirement files.


[airflow] 27/36: Cope with multiple processes get_remote_image_info in parallel (#9105)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 72366166baa763b13f15e37d970f3c67089d8b38
Author: Ash Berlin-Taylor <as...@firemirror.com>
AuthorDate: Thu Jun 4 11:05:51 2020 +0100

    Cope with multiple processes get_remote_image_info in parallel (#9105)
    
    When I'd made a change to a large number of python files, running the
    flake8 pre-commit hook would fail without obvious error (as in no error
    was printed, but exit code was 1).
    
    In debugging this I switch the pre-commit to `require_serial: true` and
    the problem went way - the fix for this is:
    
    - Don't redirect stderr to /dev/null (that silences both our VERBOSE
      trace output, and the errors from docker)
    - Use `--cidfile` option to docker to create a random name and write the
      created container ID to a file
    
    (cherry picked from commit bb67a87fa4a2893976fbdb8f0a95ba50653a15df)
---
 scripts/ci/_utils.sh | 28 ++++++++++++++++++++--------
 1 file changed, 20 insertions(+), 8 deletions(-)

diff --git a/scripts/ci/_utils.sh b/scripts/ci/_utils.sh
index c782ab6..8ac47a0 100644
--- a/scripts/ci/_utils.sh
+++ b/scripts/ci/_utils.sh
@@ -26,6 +26,7 @@ declare -a EXTRA_DOCKER_PROD_BUILD_FLAGS
 export EXTRA_DOCKER_FLAGS
 export EXTRA_DOCKER_PROD_BUILD_FLAGS
 
+
 # In case "VERBOSE_COMMANDS" is set to "true" set -x is used to enable debugging
 function check_verbose_setup {
     if [[ ${VERBOSE_COMMANDS:="false"} == "true" ]]; then
@@ -39,7 +40,8 @@ function check_verbose_setup {
 # In case "VERBOSE" is set to "true" (--verbose flag in Breeze) all docker commands run will be
 # printed before execution
 function verbose_docker {
-    if [[ ${VERBOSE:="false"} == "true" ]]; then
+    if [[ ${VERBOSE:="false"} == "true" && ${VERBOSE_COMMANDS:=} != "true" ]]; then
+       # do not print echo if VERSBOSE_COMMAND is set (set -x does it already)
         echo "docker" "${@}"
     fi
     docker "${@}"
@@ -50,6 +52,8 @@ function initialize_common_environment {
     # default python Major/Minor version
     PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:="3.5"}
 
+    FILES_TO_CLEANUP_ON_EXIT=()
+
     # Sets to where airflow sources are located
     AIRFLOW_SOURCES=${AIRFLOW_SOURCES:=$(cd "${MY_DIR}/../../" && pwd)}
     export AIRFLOW_SOURCES
@@ -712,25 +716,28 @@ function get_local_image_info() {
 function get_remote_image_info() {
     set +e
     # Pull remote manifest image
-    if ! verbose_docker pull "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}" >/dev/null 2>&1 ; then
+    if ! verbose_docker pull "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}" >/dev/null; then
         echo
         echo "Remote docker registry unreachable"
         echo
         REMOTE_DOCKER_REGISTRY_UNREACHABLE="true"
         return
     fi
+    set -e
+
+    # Docker needs the file passed to --cidfile to not exist, so we can't use mktemp
+    TMP_CONTAINER_ID="remote-airflow-manifest-$$.container_id"
+    FILES_TO_CLEANUP_ON_EXIT+=("$TMP_CONTAINER_ID")
+
     TMP_MANIFEST_REMOTE_JSON=$(mktemp)
     TMP_MANIFEST_REMOTE_SHA=$(mktemp)
-    # delete container just in case
-    verbose_docker rm --force "remote-airflow-manifest" >/dev/null 2>&1
-    set -e
     # Create container out of the manifest image without running it
-    verbose_docker create --name "remote-airflow-manifest" "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}" >/dev/null 2>&1
+    verbose_docker create --cidfile "${TMP_CONTAINER_ID}" "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}"
     # Extract manifest and store it in local file
-    verbose_docker cp "remote-airflow-manifest:${AIRFLOW_CI_BASE_TAG}.json" "${TMP_MANIFEST_REMOTE_JSON}" >/dev/null 2>&1
+    verbose_docker cp "$(cat "${TMP_CONTAINER_ID}"):${AIRFLOW_CI_BASE_TAG}.json" "${TMP_MANIFEST_REMOTE_JSON}"
     # Filter everything except SHAs of image layers
     sed 's/ *//g' "${TMP_MANIFEST_REMOTE_JSON}" | grep '^"sha256:' >"${TMP_MANIFEST_REMOTE_SHA}"
-    verbose_docker rm --force "remote-airflow-manifest" >/dev/null 2>&1
+    verbose_docker rm --force "$( cat "${TMP_CONTAINER_ID}")"
 }
 
 # The Number determines the cut-off between local building time and pull + build time.
@@ -896,6 +903,11 @@ function script_end {
     if [[ ${VERBOSE_COMMANDS:="false"} == "true" ]]; then
         set +x
     fi
+
+    if [[ ${#FILES_TO_CLEANUP_ON_EXIT[@]} -gt 0 ]]; then
+      rm -rf -- "${FILES_TO_CLEANUP_ON_EXIT[@]}"
+    fi
+
     END_SCRIPT_TIME=$(date +%s)
     RUN_SCRIPT_TIME=$((END_SCRIPT_TIME-START_SCRIPT_TIME))
     if [[ ${BREEZE:=} != "true" ]]; then


[airflow] 13/36: Add ADDITIONAL_AIRFLOW_EXTRAS (#9032)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit ab5e2ea062e619c8414f5b424d5bc95f0978f50e
Author: Fabian <fa...@t-online.de>
AuthorDate: Wed May 27 12:58:59 2020 +0200

    Add ADDITIONAL_AIRFLOW_EXTRAS (#9032)
    
    * Add build-arg ADDITIONAL_AIRFLOW_EXTRAS
    
    * Add ADDITIONAL_AIRFLOW_EXTRAS example and description
    
    (cherry picked from commit 5a7a3d13eed4b079841e0e9f6a671a3391ef051e)
---
 Dockerfile | 4 +++-
 IMAGES.rst | 6 +++++-
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/Dockerfile b/Dockerfile
index 2ca5046..ef3a988 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -35,6 +35,7 @@
 #
 ARG AIRFLOW_VERSION="2.0.0.dev0"
 ARG AIRFLOW_EXTRAS="async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv"
+ARG ADDITIONAL_AIRFLOW_EXTRAS=""
 ARG ADDITIONAL_PYTHON_DEPS=""
 
 ARG AIRFLOW_HOME=/opt/airflow
@@ -168,7 +169,8 @@ ARG AIRFLOW_VERSION
 ENV AIRFLOW_VERSION=${AIRFLOW_VERSION}
 
 ARG AIRFLOW_EXTRAS
-ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}
+ARG ADDITIONAL_AIRFLOW_EXTRAS
+ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}${ADDITIONAL_AIRFLOW_EXTRAS:+,}${ADDITIONAL_AIRFLOW_EXTRAS}
 
 ARG ADDITIONAL_PYTHON_DEPS
 ENV ADDITIONAL_PYTHON_DEPS=${ADDITIONAL_PYTHON_DEPS}
diff --git a/IMAGES.rst b/IMAGES.rst
index 5119a16..3038bfe 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -257,6 +257,9 @@ The following build arguments (``--build-arg`` in docker build command) can be u
 | ``AIRFLOW_EXTRAS``                       | (see Dockerfile)                         | Default extras with which airflow is     |
 |                                          |                                          | installed                                |
 +------------------------------------------+------------------------------------------+------------------------------------------+
+| ``ADDITIONAL_AIRFLOW_EXTRAS``            | (see Dockerfile)                         | Additional extras with which airflow is  |
+|                                          |                                          | installed                                |
++------------------------------------------+------------------------------------------+------------------------------------------+
 | ``ADDITIONAL_PYTHON_DEPS``               | (see Dockerfile)                         | Optional python packages to extend       |
 |                                          |                                          | the image with some extra dependencies   |
 +------------------------------------------+------------------------------------------+------------------------------------------+
@@ -368,7 +371,7 @@ requirements taken from v1-10-test branch in Github.
     --build-arg AIRFLOW_SOURCES_FROM="entrypoint.sh" \
     --build-arg AIRFLOW_SOURCES_TO="/entrypoint"
 
-This builds the production image in version 3.7 with default airflow extras from 1.10.10 Pypi package and
+This builds the production image in version 3.7 with additional airflow extras from 1.10.10 Pypi package and
 additional python dependencies.
 
 .. code-block::
@@ -382,6 +385,7 @@ additional python dependencies.
     --build-arg ENTRYPOINT_FILE="https://raw.githubusercontent.com/apache/airflow/1.10.10/entrypoint.sh" \
     --build-arg AIRFLOW_SOURCES_FROM="entrypoint.sh" \
     --build-arg AIRFLOW_SOURCES_TO="/entrypoint" \
+    --build-arg ADDITIONAL_AIRFLOW_EXTRAS="mssql,hdfs"
     --build-arg ADDITIONAL_PYTHON_DEPS="sshtunnel oauth2client"
 
 Image manifests


[airflow] 04/36: Fixed test-target command (#8795)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit f8bf546b97d70f359a12053ec772a14d2906e79b
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sat May 9 18:34:52 2020 +0200

    Fixed test-target command (#8795)
    
    Wrong parameter sequence were passed to the run script.
    
    (cherry picked from commit 100f530ed57c5e8755f9e327160c2734ad5632e6)
---
 breeze                                  | 2 +-
 scripts/ci/in_container/run_ci_tests.sh | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/breeze b/breeze
index 09c8fab..96d6f0e 100755
--- a/breeze
+++ b/breeze
@@ -536,7 +536,7 @@ function prepare_command_files() {
 
     # Prepare script for "run test"
     prepare_command_file "${BUILD_CACHE_DIR}/${LAST_DC_TEST_CI_FILE}" \
-        "${DC_RUN_CI_COMMAND}" "${COMPOSE_CI_FILE}" "true" "${AIRFLOW_CI_IMAGE}" '*'
+        "${DC_RUN_CI_COMMAND}" "true" "${COMPOSE_CI_FILE}" "${AIRFLOW_CI_IMAGE}" '*'
 
     # Prepare script for "run docker compose command"
     prepare_command_file "${BUILD_CACHE_DIR}/${LAST_DC_CI_FILE}" \
diff --git a/scripts/ci/in_container/run_ci_tests.sh b/scripts/ci/in_container/run_ci_tests.sh
index 720cde8..b046c08 100755
--- a/scripts/ci/in_container/run_ci_tests.sh
+++ b/scripts/ci/in_container/run_ci_tests.sh
@@ -31,7 +31,7 @@ pytest "${PYTEST_ARGS[@]}"
 RES=$?
 
 set +x
-if [[ "${RES}" == "0" && ${CI} == "true" ]]; then
+if [[ "${RES}" == "0" && ${CI:="false"} == "true" ]]; then
     echo "All tests successful"
     bash <(curl -s https://codecov.io/bash)
 fi


[airflow] 19/36: Don't reuse MY_DIR in breeze to mean different folder from ci/_utils.sh (#9098)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit b66a70600b3636db9abc2caff3460559b56183e2
Author: Ash Berlin-Taylor <as...@firemirror.com>
AuthorDate: Tue Jun 2 07:39:42 2020 +0100

    Don't reuse MY_DIR in breeze to mean different folder from ci/_utils.sh (#9098)
    
    scripts/ci/*.sh uses MY_DIR to mean scripts/ci, but in `breeze` MY_DIR
    is the same as AIRFLOW_SOURCES. When jumping back-and-forth between
    ci/_utils.sh, breeze, and ci/ci_*.sh it can be confusing to keep track
    of what is what.
    
    This changes `breeze` to use `AIRFLOW_SOURCES` to referrer to the top
    level folder instead -- that means I don't have to keep as much context
    in my head
    
    (cherry picked from commit 3dd81b7af107646698f7a69a57a87d657e54b1e4)
---
 breeze               | 31 ++++++++++++++-----------------
 scripts/ci/_utils.sh |  2 +-
 2 files changed, 15 insertions(+), 18 deletions(-)

diff --git a/breeze b/breeze
index b2e3c67..75c00eb 100755
--- a/breeze
+++ b/breeze
@@ -18,7 +18,7 @@
 # under the License.
 set -euo pipefail
 
-MY_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+AIRFLOW_SOURCES="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
 
 # Bash arrays need to be defined outside of functions unfortunately :(
 # Array with extra options for Docker compose
@@ -42,21 +42,18 @@ function setup_default_breeze_variables() {
     # Maximum screen width to print the lines spanning the whole terminal width
     export MAX_SCREEN_WIDTH=100
 
-    # This is where airflow sources reside
-    export AIRFLOW_SOURCES="${MY_DIR}"
-
     # Directory where all CI scripts are located
-    SCRIPTS_CI_DIR="${MY_DIR}/scripts/ci"
+    SCRIPTS_CI_DIR="${AIRFLOW_SOURCES}/scripts/ci"
 
     # Directory where all the build cache is stored - we keep there status of all the docker images
     # As well as hashes of the important files, but also we generate build scripts there that are
     # Used to execute the commands for breeze
-    BUILD_CACHE_DIR="${MY_DIR}/.build"
+    BUILD_CACHE_DIR="${AIRFLOW_SOURCES}/.build"
     # This folder is mounted to inside the container in /files folder. This is the way how
     # We can exchange DAGs, scripts, packages etc with the container environment
-    FILES_DIR="${MY_DIR}/files"
+    FILES_DIR="${AIRFLOW_SOURCES}/files"
     # Temporary dir used well ... temporarily
-    TMP_DIR="${MY_DIR}/tmp"
+    TMP_DIR="${AIRFLOW_SOURCES}/tmp"
 
     # Create those folders above in case they do not exist
     mkdir -pv "${BUILD_CACHE_DIR}"
@@ -95,7 +92,7 @@ function setup_default_breeze_variables() {
     # Update short and long options in the breeze-complete script
     # This way autocomplete will work automatically with all options available
     # shellcheck source=breeze-complete
-    . "${MY_DIR}/breeze-complete"
+    . "${AIRFLOW_SOURCES}/breeze-complete"
 
     # By default we mount local Airflow sources
     MOUNT_LOCAL_SOURCES="true"
@@ -145,8 +142,8 @@ function setup_default_breeze_variables() {
     export FORCE_BUILD_IMAGES=${FORCE_BUILD_IMAGES:="false"}
 
     # If those files are present, the ASCII-art/cheat-sheet are suppressed
-    SUPPRESS_CHEATSHEET_FILE="${MY_DIR}/.suppress_cheatsheet"
-    SUPPRESS_ASCIIART_FILE="${MY_DIR}/.suppress_asciiart"
+    SUPPRESS_CHEATSHEET_FILE="${AIRFLOW_SOURCES}/.suppress_cheatsheet"
+    SUPPRESS_ASCIIART_FILE="${AIRFLOW_SOURCES}/.suppress_asciiart"
 
     # Default values for the flags used
 
@@ -186,9 +183,9 @@ function initialize_virtualenv() {
         echo
         echo "This will wipe out ${AIRFLOW_HOME_DIR} and reset all the databases!"
         echo
-        "${MY_DIR}/confirm" "Proceeding with the initialization"
+        "${AIRFLOW_SOURCES}/confirm" "Proceeding with the initialization"
         echo
-        pushd "${MY_DIR}"
+        pushd "${AIRFLOW_SOURCES}"
         set +e
         pip install -e ".[devel]" --constraint "requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"
         RES=$?
@@ -245,11 +242,11 @@ function setup_autocomplete() {
         echo >&2
         exit 1
     fi
-    "${MY_DIR}/confirm" "This will create ~/.bash_completion.d/ directory and modify ~/.*rc files"
+    "${AIRFLOW_SOURCES}/confirm" "This will create ~/.bash_completion.d/ directory and modify ~/.*rc files"
     echo
     echo
     mkdir -pv ~/.bash_completion.d
-    ln -sf "${MY_DIR}/breeze-complete" "${HOME}/.bash_completion.d/"
+    ln -sf "${AIRFLOW_SOURCES}/breeze-complete" "${HOME}/.bash_completion.d/"
     touch ~/.bashrc
     cat >>~/.bashrc <<"EOF"
 for BCFILE in ~/.bash_completion.d/* ; do
@@ -272,7 +269,7 @@ EOF
         if [[ "${RES}" == "0" ]]; then
             echo "  Seems you already source .bashrc in your .bash_profile so not adding it."
         else
-            "${MY_DIR}/confirm" "This will modify  ~/.bash_profile and source .bashrc from it"
+            "${AIRFLOW_SOURCES}/confirm" "This will modify  ~/.bash_profile and source .bashrc from it"
             echo
             echo
             cat >>~/.bash_profile <<"EOF"
@@ -1730,7 +1727,7 @@ function print_cheatsheet() {
             echo "    When you exit the environment, you can add sources of Airflow to the path - you can"
             echo "    run breeze or the scripts above from any directory by calling 'breeze' commands directly"
             echo
-            echo "     export PATH=\${PATH}:\"${MY_DIR}\""
+            echo "     export PATH=\${PATH}:\"${AIRFLOW_SOURCES}\""
             echo
         fi
         set -e
diff --git a/scripts/ci/_utils.sh b/scripts/ci/_utils.sh
index 85aeb91..6c0204c 100644
--- a/scripts/ci/_utils.sh
+++ b/scripts/ci/_utils.sh
@@ -305,7 +305,7 @@ function sanitize_mounted_files() {
 
     # When KinD cluster is created, the folder keeps authentication information
     # across sessions
-    mkdir -p "${MY_DIR}/.kube" >/dev/null 2>&1
+    mkdir -p "${AIRFLOW_SOURCES}/.kube" >/dev/null 2>&1
 }
 
 #


[airflow] 22/36: Display docs errors summary (#8392)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 060ccedbede5aa9054ddca6dc82e845d3ada039f
Author: Kamil Breguła <mi...@users.noreply.github.com>
AuthorDate: Thu Apr 16 10:56:45 2020 +0200

    Display docs errors summary (#8392)
    
    (cherry picked from commit 670118f04a6908c4b088bfca4f99c88bbe696262)
---
 CONTRIBUTING.rst                          |   7 +-
 docs/Makefile                             | 194 ----------------------
 docs/build                                | 265 ++++++++++++++++++++++++++++++
 docs/build.sh                             | 108 ------------
 scripts/ci/_utils.sh                      |   2 +-
 scripts/ci/ci_docs.sh                     |   4 +-
 scripts/ci/in_container/run_docs_build.sh |   2 +-
 7 files changed, 273 insertions(+), 309 deletions(-)

diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index b9372ed..11199df 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -108,12 +108,13 @@ To generate a local version:
 .. code-block:: bash
 
     cd docs
-    ./build.sh
+    ./build
     ./start_doc_server.sh
 
 .. note::
-    The docs build script ``build.sh`` requires bash 4.0 or greater.
-    If you are building on mac, you can install latest version of bash with homebrew.
+    The docs build script ``build`` requires Python 3.6 or greater.
+
+**Known issues:**
 
 
 If you are creating ``example_dags`` directory, you need to create ``example_dags/__init__.py`` with Apache license or copy another ``__init__.py`` file that contains the necessary license.
diff --git a/docs/Makefile b/docs/Makefile
deleted file mode 100644
index b707924..0000000
--- a/docs/Makefile
+++ /dev/null
@@ -1,194 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#   http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied.  See the License for the
-# specific language governing permissions and limitations
-# under the License.
-# Makefile for Sphinx documentation
-#
-
-# You can set these variables from the command line.
-SPHINXOPTS    =
-SPHINXBUILD   = sphinx-build
-PAPER         =
-BUILDDIR      = _build
-
-# User-friendly check for sphinx-build
-ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1)
-$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/)
-endif
-
-# Internal variables.
-PAPEROPT_a4     = -D latex_paper_size=a4
-PAPEROPT_letter = -D latex_paper_size=letter
-ALLSPHINXOPTS   = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
-# the i18n builder cannot share the environment and doctrees with the others
-I18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
-
-.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext
-
-help:
-	@echo "Please use \`make <target>' where <target> is one of"
-	@echo "  html       to make standalone HTML files"
-	@echo "  dirhtml    to make HTML files named index.html in directories"
-	@echo "  singlehtml to make a single large HTML file"
-	@echo "  pickle     to make pickle files"
-	@echo "  json       to make JSON files"
-	@echo "  htmlhelp   to make HTML files and a HTML help project"
-	@echo "  qthelp     to make HTML files and a qthelp project"
-	@echo "  devhelp    to make HTML files and a Devhelp project"
-	@echo "  epub       to make an epub"
-	@echo "  latex      to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
-	@echo "  latexpdf   to make LaTeX files and run them through pdflatex"
-	@echo "  latexpdfja to make LaTeX files and run them through platex/dvipdfmx"
-	@echo "  text       to make text files"
-	@echo "  man        to make manual pages"
-	@echo "  texinfo    to make Texinfo files"
-	@echo "  info       to make Texinfo files and run them through makeinfo"
-	@echo "  gettext    to make PO message catalogs"
-	@echo "  changes    to make an overview of all changed/added/deprecated items"
-	@echo "  xml        to make Docutils-native XML files"
-	@echo "  pseudoxml  to make pseudoxml-XML files for display purposes"
-	@echo "  linkcheck  to check all external links for integrity"
-	@echo "  doctest    to run all doctests embedded in the documentation (if enabled)"
-
-clean:
-	rm -rf $(BUILDDIR)/*
-
-html:
-	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
-	@echo
-	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
-
-dirhtml:
-	$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
-	@echo
-	@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
-
-singlehtml:
-	$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
-	@echo
-	@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
-
-pickle:
-	$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
-	@echo
-	@echo "Build finished; now you can process the pickle files."
-
-json:
-	$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
-	@echo
-	@echo "Build finished; now you can process the JSON files."
-
-htmlhelp:
-	$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
-	@echo
-	@echo "Build finished; now you can run HTML Help Workshop with the" \
-	      ".hhp project file in $(BUILDDIR)/htmlhelp."
-
-qthelp:
-	$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
-	@echo
-	@echo "Build finished; now you can run "qcollectiongenerator" with the" \
-	      ".qhcp project file in $(BUILDDIR)/qthelp, like this:"
-	@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/Flux.qhcp"
-	@echo "To view the help file:"
-	@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/Flux.qhc"
-
-devhelp:
-	$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
-	@echo
-	@echo "Build finished."
-	@echo "To view the help file:"
-	@echo "# mkdir -p $$HOME/.local/share/devhelp/Flux"
-	@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/Flux"
-	@echo "# devhelp"
-
-epub:
-	$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
-	@echo
-	@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
-
-latex:
-	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
-	@echo
-	@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
-	@echo "Run \`make' in that directory to run these through (pdf)latex" \
-	      "(use \`make latexpdf' here to do that automatically)."
-
-latexpdf:
-	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
-	@echo "Running LaTeX files through pdflatex..."
-	$(MAKE) -C $(BUILDDIR)/latex all-pdf
-	@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
-
-latexpdfja:
-	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
-	@echo "Running LaTeX files through platex and dvipdfmx..."
-	$(MAKE) -C $(BUILDDIR)/latex all-pdf-ja
-	@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
-
-text:
-	$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
-	@echo
-	@echo "Build finished. The text files are in $(BUILDDIR)/text."
-
-man:
-	$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
-	@echo
-	@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
-
-texinfo:
-	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
-	@echo
-	@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
-	@echo "Run \`make' in that directory to run these through makeinfo" \
-	      "(use \`make info' here to do that automatically)."
-
-info:
-	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
-	@echo "Running Texinfo files through makeinfo..."
-	make -C $(BUILDDIR)/texinfo info
-	@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
-
-gettext:
-	$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
-	@echo
-	@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
-
-changes:
-	$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
-	@echo
-	@echo "The overview file is in $(BUILDDIR)/changes."
-
-linkcheck:
-	$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
-	@echo
-	@echo "Link check complete; look for any errors in the above output " \
-	      "or in $(BUILDDIR)/linkcheck/output.txt."
-
-doctest:
-	$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
-	@echo "Testing of doctests in the sources finished, look at the " \
-	      "results in $(BUILDDIR)/doctest/output.txt."
-
-xml:
-	$(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml
-	@echo
-	@echo "Build finished. The XML files are in $(BUILDDIR)/xml."
-
-pseudoxml:
-	$(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml
-	@echo
-	@echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml."
diff --git a/docs/build b/docs/build
new file mode 100755
index 0000000..3b51212
--- /dev/null
+++ b/docs/build
@@ -0,0 +1,265 @@
+#!/usr/bin/env python
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import atexit
+import os
+import re
+import shlex
+import shutil
+import sys
+from subprocess import run
+from contextlib import suppress
+from glob import glob
+from tempfile import NamedTemporaryFile
+from typing import NamedTuple, List, Iterable, Set, Optional
+
+if __name__ != "__main__":
+    raise Exception(
+        "This file is intended to be executed as an executable program. You cannot use it as a module."
+        "To run this script, run the ./build command"
+    )
+
+
+class DocBuildError(NamedTuple):
+    file_path: Optional[str]
+    line_no: Optional[int]
+    message: str
+
+
+build_errors: List[DocBuildError] = []
+
+os.chdir(os.path.dirname(os.path.abspath(__file__)))
+
+ROOT_PROJECT_DIR = os.path.abspath(os.path.join(os.getcwd(), os.pardir))
+ROOT_PACKAGE_DIR = os.path.join(ROOT_PROJECT_DIR, "airflow")
+
+
+def clean_files() -> None:
+    print("Removing content of the _build and _api folders")
+    with suppress(FileNotFoundError):
+        for filename in glob("_build/*"):
+            shutil.rmtree(f"_build/{filename}")
+    with suppress(FileNotFoundError):
+        for filename in glob("_api/*"):
+            shutil.rmtree(f"_api/{filename}")
+    print("Removed content of the _build and _api folders")
+
+
+def prepare_directories() -> None:
+    if os.path.exists("/.dockerenv"):
+        # This script can be run both - in container and outside of it.
+        # Here we are inside the container which means that we should (when the host is Linux)
+        # fix permissions of the _build and _api folders via sudo.
+        # Those files are mounted from the host via docs folder and we might not have permissions to
+        # write to those directories (and remove the _api folder).
+        # We know we have sudo capabilities inside the container.
+        print("Creating the _build and _api folders in case they do not exist")
+        run(["sudo", "mkdir", "-pv", "_build"], check=True)
+        run(["sudo", "mkdir", "-pv", "_api"], check=True)
+        print("Created the _build and _api folders in case they do not exist")
+
+        def restore_ownership() -> None:
+            # We are inside the container which means that we should fix back the permissions of the
+            # _build and _api folder files, so that they can be accessed by the host user
+            # The _api folder should be deleted by then but just in case we should change the ownership
+            host_user_id = os.environ["HOST_USER_ID"]
+            host_group_id = os.environ["HOST_GROUP_ID"]
+            print(f"Changing ownership of docs/_build folder back to ${host_user_id}:${host_group_id}")
+            run(["sudo", "chown", f'"${host_user_id}":"${host_group_id}"', "_build"], check=True)
+            if os.path.exists("_api"):
+                run(["sudo", "chown", f'"${host_user_id}":"${host_group_id}"', "_api"], check=True)
+
+            print(f"Changed ownership of docs/_build folder back to ${host_user_id}:${host_group_id}")
+
+        atexit.register(restore_ownership)
+
+    else:
+        # We are outside the container so we simply make sure that the directories exist
+        print("Creating the _build and _api folders in case they do not exist")
+        run(["mkdir", "-pv", "_build"], check=True)
+        run(["mkdir", "-pv", "_api"], check=True)
+        print("Creating the _build and _api folders in case they do not exist")
+
+
+def display_errors_summary() -> None:
+    for warning_no, error in enumerate(sorted(build_errors), 1):
+        print("=" * 20, f"Error {warning_no:3}", "=" * 20)
+        print(error.message)
+        print()
+        if error.file_path and error.line_no:
+            print(f"File path: {error.file_path} ({error.line_no})")
+            print()
+            print(prepare_code_snippet(error.file_path, error.line_no))
+        elif error.file_path:
+            print(f"File path: {error.file_path}")
+
+    print("=" * 50)
+
+
+def check_file_not_contains(file_path: str, pattern: str, message: str) -> None:
+    with open(file_path, "rb", 0) as doc_file:
+        pattern_compiled = re.compile(pattern)
+
+        for num, line in enumerate(doc_file, 1):
+            line_decode = line.decode()
+            if re.search(pattern_compiled, line_decode):
+                build_errors.append(DocBuildError(file_path=file_path, line_no=num, message=message))
+
+
+def filter_file_list_by_pattern(file_paths: Iterable[str], pattern: str) -> List[str]:
+    output_paths = []
+    pattern_compiled = re.compile(pattern)
+    for file_path in file_paths:
+        with open(file_path, "rb", 0) as text_file:
+            text_file_content = text_file.read().decode()
+            if re.findall(pattern_compiled, text_file_content):
+                output_paths.append(file_path)
+    return output_paths
+
+
+def find_modules(deprecated_only: bool = False) -> Set[str]:
+    file_paths = glob(f"{ROOT_PACKAGE_DIR}/**/*.py", recursive=True)
+    # Exclude __init__.py
+    file_paths = (f for f in file_paths if not f.endswith("__init__.py"))
+    if deprecated_only:
+        file_paths = filter_file_list_by_pattern(file_paths, r"This module is deprecated.")
+    # Make path relative
+    file_paths = (os.path.relpath(f, ROOT_PROJECT_DIR) for f in file_paths)
+    # Convert filename to module
+    modules_names = {file_path.rpartition(".")[0].replace("/", ".") for file_path in file_paths}
+    return modules_names
+
+
+def check_exampleinclude_for_example_dags():
+    all_docs_files = glob("**/*rst", recursive=True)
+
+    for doc_file in all_docs_files:
+        check_file_not_contains(
+            file_path=doc_file,
+            pattern=r"literalinclude::.+example_dags",
+            message=(
+                "literalinclude directive is is prohibited for example DAGs. \n"
+                "You should use a exampleinclude directive to include example DAGs."
+            )
+        )
+
+
+def prepare_code_snippet(file_path: str, line_no: int, context_lines_count: int=5) -> str:
+    def guess_lexer_for_filename(filename):
+        from pygments.util import ClassNotFound
+        from pygments.lexers import get_lexer_for_filename
+
+        try:
+            lexer = get_lexer_for_filename(filename)
+        except ClassNotFound:
+            from pygments.lexers.special import TextLexer
+            lexer = TextLexer()
+        return lexer
+
+    with open(file_path) as text_file:
+        # Highlight code
+        code = text_file.read()
+        with suppress(ImportError):
+            import pygments
+
+            from pygments.formatters.terminal import TerminalFormatter
+            code = pygments.highlight(
+                code=code, formatter=TerminalFormatter(), lexer=guess_lexer_for_filename(file_path)
+            )
+
+        code_lines = code.split("\n")
+        # Prepend line number
+        code_lines = [f"{line_no:4} | {line}" for line_no, line in enumerate(code_lines, 1)]
+        # # Cut out the snippet
+        start_line_no = max(0, line_no - context_lines_count)
+        end_line_no = line_no + context_lines_count
+        code_lines = code_lines[start_line_no:end_line_no]
+        # Join lines
+        code = "\n".join(code_lines)
+    return code
+
+
+def parse_sphinx_warnings(warning_text: str) -> List[DocBuildError]:
+    sphinx_build_errors = []
+    for sphinx_warning in warning_text.split("\n"):
+        if not sphinx_warning:
+            continue
+        warining_parts = sphinx_warning.split(":", 2)
+        if len(warining_parts) == 3:
+            sphinx_build_errors.append(
+                DocBuildError(
+                    file_path=warining_parts[0], line_no=int(warining_parts[1]), message=warining_parts[2]
+                )
+            )
+        else:
+            sphinx_build_errors.append(DocBuildError(file_path=None, line_no=None, message=sphinx_warning))
+    return sphinx_build_errors
+
+
+def build_sphinx_docs() -> None:
+    with NamedTemporaryFile() as tmp_file:
+        build_cmd = [
+            "sphinx-build",
+            "-b",
+            "html",
+            "-d",
+            "_build/doctrees",
+            "--color",
+            "-w",
+            tmp_file.name,
+            ".",
+            "_build/html",
+        ]
+        print("Executing cmd: ", " ".join([shlex.quote(c) for c in build_cmd]))
+
+        completed_proc = run(build_cmd)
+        if completed_proc.returncode != 0:
+            build_errors.append(
+                DocBuildError(
+                    file_path=None,
+                    line_no=None,
+                    message=f"Sphinx returned non-zero exit status: {completed_proc.returncode}.",
+                )
+            )
+        tmp_file.seek(0)
+        warning_text = tmp_file.read().decode()
+        # Remove 7-bit C1 ANSI escape sequences
+        warning_text = re.sub(r"\x1B[@-_][0-?]*[ -/]*[@-~]", "", warning_text)
+        sphinx_build_errrors = parse_sphinx_warnings(warning_text)
+        build_errors.extend(sphinx_build_errrors)
+
+
+print("Current working directory: ", os.getcwd())
+
+prepare_directories()
+clean_files()
+
+check_exampleinclude_for_example_dags()
+
+if build_errors:
+    display_errors_summary()
+    print()
+    print("The documentation has errors. Fix them to build documentation.")
+    sys.exit(1)
+
+build_sphinx_docs()
+
+if build_errors:
+    display_errors_summary()
+    print()
+    print("The documentation has errors.")
+    sys.exit(1)
diff --git a/docs/build.sh b/docs/build.sh
deleted file mode 100755
index 2118830..0000000
--- a/docs/build.sh
+++ /dev/null
@@ -1,108 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#   http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied.  See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-set -euo pipefail
-
-MY_DIR="$(cd "$(dirname "$0")" && pwd)"
-pushd "${MY_DIR}" &>/dev/null || exit 1
-
-echo
-echo "Working in ${MY_DIR} folder"
-echo
-
-
-if [[ -f /.dockerenv ]]; then
-    # This script can be run both - in container and outside of it.
-    # Here we are inside the container which means that we should (when the host is Linux)
-    # fix permissions of the _build and _api folders via sudo.
-    # Those files are mounted from the host via docs folder and we might not have permissions to
-    # write to those directories (and remove the _api folder).
-    # We know we have sudo capabilities inside the container.
-    echo "Creating the _build and _api folders in case they do not exist"
-    sudo mkdir -pv _build
-    sudo mkdir -pv _api
-    echo "Created the _build and _api folders in case they do not exist"
-else
-    # We are outside the container so we simply make sure that the directories exist
-    echo "Creating the _build and _api folders in case they do not exist"
-    mkdir -pv _build
-    mkdir -pv _api
-    echo "Creating the _build and _api folders in case they do not exist"
-fi
-
-echo "Removing content of the _build and _api folders"
-rm -rf _build/*
-rm -rf _api/*
-echo "Removed content of the _build and _api folders"
-
-
-set +e
-# shellcheck disable=SC2063
-NUM_INCORRECT_USE_LITERALINCLUDE=$(grep -inR --include \*.rst 'literalinclude::.\+example_dags' . | \
-    tee /dev/tty |
-    wc -l |\
-    tr -d '[:space:]')
-set -e
-
-echo
-echo "Checking for presence of literalinclude in example DAGs"
-echo
-
-if [[ "${NUM_INCORRECT_USE_LITERALINCLUDE}" -ne "0" ]]; then
-    echo
-    echo "Unexpected problems found in the documentation. "
-    echo "You should use a exampleinclude directive to include example DAGs."
-    echo "Currently, ${NUM_INCORRECT_USE_LITERALINCLUDE} problem found."
-    echo
-    exit 1
-else
-    echo
-    echo "No literalincludes in example DAGs found"
-    echo
-fi
-
-SUCCEED_LINE=$(make html |\
-    tee /dev/tty |\
-    grep 'build succeeded' |\
-    head -1)
-
-NUM_CURRENT_WARNINGS=$(echo "${SUCCEED_LINE}" |\
-    sed -E 's/build succeeded, ([0-9]+) warnings?\./\1/g')
-
-if [[  -f /.dockerenv ]]; then
-    # We are inside the container which means that we should fix back the permissions of the
-    # _build and _api folder files, so that they can be accessed by the host user
-    # The _api folder should be deleted by then but just in case we should change the ownership
-    echo "Changing ownership of docs/_build folder back to ${HOST_USER_ID}:${HOST_GROUP_ID}"
-    sudo chown "${HOST_USER_ID}":"${HOST_GROUP_ID}" _build
-    if [[ -d _api ]]; then
-        sudo chown "${HOST_USER_ID}":"${HOST_GROUP_ID}" _api
-    fi
-    echo "Changed ownership of docs/_build folder back to ${HOST_USER_ID}:${HOST_GROUP_ID}"
-fi
-
-
-if echo "${SUCCEED_LINE}" | grep -q "warning"; then
-    echo
-    echo "Unexpected problems found in the documentation. "
-    echo "Currently, ${NUM_CURRENT_WARNINGS} warnings found. "
-    echo
-    exit 1
-fi
-
-popd &>/dev/null || exit 1
diff --git a/scripts/ci/_utils.sh b/scripts/ci/_utils.sh
index 6766a3a..68778d8 100644
--- a/scripts/ci/_utils.sh
+++ b/scripts/ci/_utils.sh
@@ -1117,7 +1117,7 @@ function run_docs() {
             --env HOST_GROUP_ID="$(id -gr)" \
             --rm \
             "${AIRFLOW_CI_IMAGE}" \
-            "--" "/opt/airflow/docs/build.sh" \
+            "--" "/opt/airflow/docs/build" \
             | tee -a "${OUTPUT_LOG}"
 }
 
diff --git a/scripts/ci/ci_docs.sh b/scripts/ci/ci_docs.sh
index 7766b9e..e84de9b 100755
--- a/scripts/ci/ci_docs.sh
+++ b/scripts/ci/ci_docs.sh
@@ -15,7 +15,7 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
-export PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:-3.5}
+export PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:-3.6}
 
 # shellcheck source=scripts/ci/_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
@@ -30,7 +30,7 @@ function run_docs() {
             --env HOST_GROUP_ID="$(id -gr)" \
             --rm \
             "${AIRFLOW_CI_IMAGE}" \
-            "--" "/opt/airflow/docs/build.sh" \
+            "--" "/opt/airflow/docs/build" \
             | tee -a "${OUTPUT_LOG}"
 }
 
diff --git a/scripts/ci/in_container/run_docs_build.sh b/scripts/ci/in_container/run_docs_build.sh
index c30c1a0..8c632f8 100755
--- a/scripts/ci/in_container/run_docs_build.sh
+++ b/scripts/ci/in_container/run_docs_build.sh
@@ -25,4 +25,4 @@ trap "${HANDLERS}${HANDLERS:+;}in_container_fix_ownership" EXIT
 
 sudo rm -rf "$(pwd)/docs/_build/*"
 sudo rm -rf "$(pwd)/docs/_api/*"
-sudo -E "$(pwd)/docs/build.sh"
+sudo -E "$(pwd)/docs/build"


[airflow] 32/36: Replaces cloud-provider CLIs in CI image with scripts running containers (#9129)

Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit c0534d83ebbb5550f82842cca4c890432983fe49
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Thu Jun 4 19:12:09 2020 +0200

    Replaces cloud-provider CLIs in CI image with scripts running containers (#9129)
    
    The clis are replaced with scripts that will pull and run
    docker images when they are needed.
    
    Added Azure CLI as well.
    
    Closes: #8946 #8947 #8785
    (cherry picked from commit a39e9a352050f96dedadc39ab3d985065971c98c)
---
 BREEZE.rst                                        | 43 ++++++++++++++++--
 Dockerfile.ci                                     | 55 ++++++++++-------------
 TESTING.rst                                       | 10 ++---
 scripts/ci/_utils.sh                              |  4 +-
 scripts/ci/docker-compose/forward-credentials.yml |  6 +--
 scripts/ci/docker-compose/local-prod.yml          |  1 +
 scripts/ci/docker-compose/local.yml               |  2 +-
 7 files changed, 72 insertions(+), 49 deletions(-)

diff --git a/BREEZE.rst b/BREEZE.rst
index 7648b9d..aec975c 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -105,8 +105,9 @@ Docker in WSL 2
     and git pull the Airflow repo there.
 
 - **WSL 2 Memory Usage** :
-    WSL 2 can consume a lot of memory under the process name "Vmmem". To reclaim
-    the memory after development you can:
+    WSL 2 can consume a lot of memory under the process name "Vmmem". To reclaim the memory after
+    development you can:
+
       * On the Linux distro clear cached memory: ``sudo sysctl -w vm.drop_caches=3``
       * If no longer using Docker you can quit Docker Desktop
         (right click system try icon and select "Quit Docker Desktop")
@@ -195,7 +196,7 @@ On macOS, 2GB of RAM are available for your Docker containers by default, but mo
 On Windows WSL 2 expect the Linux Disto and Docker containers to use 7 - 8 GB of RAM.
 
 Airflow Directory Structure inside Docker
------------------------------------------
+=========================================
 
 When you are in the CI container, the following directories are used:
 
@@ -231,6 +232,42 @@ from your ``logs`` directory in the Airflow sources, so all logs created in the
 visible in the host as well. Every time you enter the container, the ``logs`` directory is
 cleaned so that logs do not accumulate.
 
+CLIs for cloud providers
+========================
+
+For development convenience we installed simple wrappers for the most common cloud providers CLIs. Those
+CLIs are not installed when you build or pull the image - they will be downloaded as docker images
+the first time you attempt to use them. It is downloaded and executed in your host's docker engine so once
+it is downloaded, it will stay until you remove the downloaded images from your host container.
+
+For each of those CLI credentials are taken (automatically) from the credentials you have defined in
+your ${HOME} directory on host.
+
+Those are currently installed CLIs (they are available as aliases to the docker commands):
+
++-----------------------+----------+-------------------------------------------------+-------------------+
+| Cloud Provider        | CLI tool | Docker image                                    | Configuration dir |
++=======================+==========+=================================================+===================+
+| Amazon Web Services   | aws      | amazon/aws-cli:latest                           | .aws              |
++-----------------------+----------+-------------------------------------------------+-------------------+
+| Microsoft Azure       | az       | mcr.microsoft.com/azure-cli:latest              | .azure            |
++-----------------------+----------+-------------------------------------------------+-------------------+
+| Google Cloud Platform | bq       | gcr.io/google.com/cloudsdktool/cloud-sdk:latest | .config/gcloud    |
+|                       +----------+-------------------------------------------------+-------------------+
+|                       | gcloud   | gcr.io/google.com/cloudsdktool/cloud-sdk:latest | .config/gcloud    |
+|                       +----------+-------------------------------------------------+-------------------+
+|                       | gsutil   | gcr.io/google.com/cloudsdktool/cloud-sdk:latest | .config/gcloud    |
++-----------------------+----------+-------------------------------------------------+-------------------+
+
+For each of the CLIs we have also an accompanying ``*-update`` alias (for example ``aws-update``) which
+will pull the latest image for the tool. Note that all Google Cloud Platform tools are served by one
+image and they are updated together.
+
+Also - in case you run several different Breeze containers in parallel (from different directories,
+with different versions) - they docker images for CLI Cloud Providers tools are shared so if you update it
+for one Breeze container, they will also get updated for all the other containers.
+
+
 Using the Airflow Breeze Environment
 =====================================
 
diff --git a/Dockerfile.ci b/Dockerfile.ci
index d15cff0..8035fa2 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -165,36 +165,6 @@ RUN echo "Pip version: ${PIP_VERSION}"
 
 RUN pip install --upgrade pip==${PIP_VERSION}
 
-# Install Google SDK
-ENV GCLOUD_HOME="/opt/gcloud" CLOUDSDK_PYTHON=python${PYTHON_MAJOR_MINOR_VERSION}
-
-RUN GCLOUD_VERSION="274.0.1" \
-    && GCOUD_URL="https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-${GCLOUD_VERSION}-linux-x86_64.tar.gz" \
-    && GCLOUD_TMP_FILE="/tmp/gcloud.tar.gz" \
-    && export CLOUDSDK_CORE_DISABLE_PROMPTS=1 \
-    && mkdir -p /opt/gcloud \
-    && curl "${GCOUD_URL}" -o "${GCLOUD_TMP_FILE}"\
-    && tar xzf "${GCLOUD_TMP_FILE}" --strip-components 1 -C "${GCLOUD_HOME}" \
-    && rm -rf "${GCLOUD_TMP_FILE}" \
-    && ${GCLOUD_HOME}/bin/gcloud components install beta \
-    && echo '. /opt/gcloud/completion.bash.inc' >> /etc/bash.bashrc
-
-ENV PATH="$PATH:${GCLOUD_HOME}/bin"
-
-# Install AWS CLI
-# Unfortunately, AWS does not provide a versioned bundle
-ENV AWS_HOME="/opt/aws"
-
-RUN AWS_TMP_DIR="/tmp/awscli/" \
-    && AWS_TMP_BUNDLE="${AWS_TMP_DIR}/awscli-bundle.zip" \
-    && AWS_URL="https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" \
-    && mkdir -pv "${AWS_TMP_DIR}" \
-    && curl "${AWS_URL}" -o "${AWS_TMP_BUNDLE}" \
-    && unzip "${AWS_TMP_BUNDLE}" -d "${AWS_TMP_DIR}" \
-    && "${AWS_TMP_DIR}/awscli-bundle/install" -i "${AWS_HOME}" -b /usr/local/bin/aws \
-    && echo "complete -C '${AWS_HOME}/bin/aws_completer' aws" >> /etc/bash.bashrc \
-    && rm -rf "${AWS_TMP_DIR}"
-
 ARG HOME=/root
 ENV HOME=${HOME}
 
@@ -206,8 +176,8 @@ ENV AIRFLOW_SOURCES=${AIRFLOW_SOURCES}
 
 WORKDIR ${AIRFLOW_SOURCES}
 
-RUN mkdir -pv ${AIRFLOW_HOME} \
-    mkdir -pv ${AIRFLOW_HOME}/dags \
+RUN mkdir -pv ${AIRFLOW_HOME} && \
+    mkdir -pv ${AIRFLOW_HOME}/dags && \
     mkdir -pv ${AIRFLOW_HOME}/logs
 
 # Increase the value here to force reinstalling Apache Airflow pip dependencies
@@ -337,6 +307,27 @@ RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
         pip install ${ADDITIONAL_PYTHON_DEPS}; \
     fi
 
+RUN \
+    AWSCLI_IMAGE="amazon/aws-cli:latest" && \
+    AZURECLI_IMAGE="mcr.microsoft.com/azure-cli:latest" && \
+    GCLOUD_IMAGE="gcr.io/google.com/cloudsdktool/cloud-sdk:latest" && \
+    echo "docker run --rm -it -v \${HOST_HOME}/.aws:/root/.aws ${AWSCLI_IMAGE} \"\$@\"" \
+        > /usr/bin/aws && \
+    echo "docker pull ${AWSCLI_IMAGE}" > /usr/bin/aws-update && \
+    echo "docker run --rm -it -v \${HOST_HOME}/.azure:/root/.azure ${AZURECLI_IMAGE} \"\$@\"" \
+        > /usr/bin/az && \
+    echo "docker pull ${AZURECLI_IMAGE}" > /usr/bin/az-update && \
+    echo "docker run --rm -it -v \${HOST_HOME}/.config:/root/.config ${GCLOUD_IMAGE} bq \"\$@\"" \
+        > /usr/bin/bq && \
+    echo "docker pull ${GCLOUD_IMAGE}" > /usr/bin/bq-update && \
+    echo "docker run --rm -it -v \${HOST_HOME}/.config:/root/.config ${GCLOUD_IMAGE} gcloud \"\$@\"" \
+        > /usr/bin/gcloud && \
+    echo "docker pull ${GCLOUD_IMAGE}" > /usr/bin/gcloud-update && \
+    echo "docker run --rm -it -v \${HOST_HOME}/.config:/root/.config ${GCLOUD_IMAGE} gsutil \"\$@\"" \
+        > /usr/bin/gsutil && \
+    echo "docker pull ${GCLOUD_IMAGE}" > /usr/bin/gsutil-update && \
+    chmod a+x /usr/bin/aws /usr/bin/az /usr/bin/bq /usr/bin/gcloud /usr/bin/gsutil
+
 WORKDIR ${AIRFLOW_SOURCES}
 
 ENV PATH="${HOME}:${PATH}"
diff --git a/TESTING.rst b/TESTING.rst
index fc8a046..ea6884e 100644
--- a/TESTING.rst
+++ b/TESTING.rst
@@ -621,12 +621,10 @@ credentials stored in your ``home`` directory. Use this feature with care as it
 visible to anything that you have installed inside the Docker container.
 
 Currently forwarded credentials are:
-  * all credentials stored in ``${HOME}/.config`` (for example, GCP credentials)
-  * credentials stored in ``${HOME}/.gsutil`` for ``gsutil`` tool from GCS
-  * credentials stored in ``${HOME}/.aws``, ``${HOME}/.boto``, and ``${HOME}/.s3`` (for AWS authentication)
-  * credentials stored in ``${HOME}/.docker`` for docker
-  * credentials stored in ``${HOME}/.kube`` for kubectl
-
+  * credentials stored in ``${HOME}/.aws`` for the aws Amazon Web Services client
+  * credentials stored in ``${HOME}/.azure`` for the az Microsoft Azure client
+  * credentials stored in ``${HOME}/.config`` for gcloud Google Cloud Platform client (among others)
+  * credentials stored in ``${HOME}/.docker`` for docker client
 
 Adding a New System Test
 --------------------------
diff --git a/scripts/ci/_utils.sh b/scripts/ci/_utils.sh
index d0b8251..3bb6ee2 100644
--- a/scripts/ci/_utils.sh
+++ b/scripts/ci/_utils.sh
@@ -257,7 +257,6 @@ function generate_local_mounts_list {
         "$prefix".flake8:/opt/airflow/.flake8:cached
         "$prefix".github:/opt/airflow/.github:cached
         "$prefix".inputrc:/root/.inputrc:cached
-        "$prefix".kube:/root/.kube:cached
         "$prefix".rat-excludes:/opt/airflow/.rat-excludes:cached
         "$prefix"CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached
         "$prefix"LICENSE:/opt/airflow/LICENSE:cached
@@ -754,7 +753,7 @@ function get_remote_image_info() {
 # Note that this only matters if you have any of the important files changed since the last build
 # of your image such as Dockerfile.ci, setup.py etc.
 #
-MAGIC_CUT_OFF_NUMBER_OF_LAYERS=34
+MAGIC_CUT_OFF_NUMBER_OF_LAYERS=41
 
 # Compares layers from both remote and local image and set FORCE_PULL_IMAGES to true in case
 # More than the last NN layers are different.
@@ -1805,7 +1804,6 @@ function delete_cluster() {
     echo
     echo "Deleted cluster ${KIND_CLUSTER_NAME}"
     echo
-    rm -rf "${HOME}/.kube/*"
 }
 
 function perform_kind_cluster_operation() {
diff --git a/scripts/ci/docker-compose/forward-credentials.yml b/scripts/ci/docker-compose/forward-credentials.yml
index e2d5f75..875b1ce 100644
--- a/scripts/ci/docker-compose/forward-credentials.yml
+++ b/scripts/ci/docker-compose/forward-credentials.yml
@@ -23,9 +23,7 @@ services:
     # To inside docker. Use with care - your credentials will be available to
     # Everything you install in Docker
     volumes:
+      - ${HOME}/.aws:/root/.aws:cached
+      - ${HOME}/.azure:/root/.azure:cached
       - ${HOME}/.config:/root/.config:cached
-      - ${HOME}/.boto:/root/.boto:cached
       - ${HOME}/.docker:/root/.docker:cached
-      - ${HOME}/.gsutil:/root/.gsutil:cached
-      - ${HOME}/.kube:/root/.kube:cached
-      - ${HOME}/.s3:/root/.s3:cached
diff --git a/scripts/ci/docker-compose/local-prod.yml b/scripts/ci/docker-compose/local-prod.yml
index 6092323..ae8317d 100644
--- a/scripts/ci/docker-compose/local-prod.yml
+++ b/scripts/ci/docker-compose/local-prod.yml
@@ -39,4 +39,5 @@ services:
     environment:
       - HOST_USER_ID
       - HOST_GROUP_ID
+      - HOST_HOME=${HOME}
       - PYTHONDONTWRITEBYTECODE
diff --git a/scripts/ci/docker-compose/local.yml b/scripts/ci/docker-compose/local.yml
index 42609e6..793a322 100644
--- a/scripts/ci/docker-compose/local.yml
+++ b/scripts/ci/docker-compose/local.yml
@@ -32,7 +32,6 @@ services:
       - ../../../.flake8:/opt/airflow/.flake8:cached
       - ../../../.github:/opt/airflow/.github:cached
       - ../../../.inputrc:/root/.inputrc:cached
-      - ../../../.kube:/root/.kube:cached
       - ../../../.rat-excludes:/opt/airflow/.rat-excludes:cached
       - ../../../CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached
       - ../../../LICENSE:/opt/airflow/LICENSE:cached
@@ -60,6 +59,7 @@ services:
     environment:
       - HOST_USER_ID
       - HOST_GROUP_ID
+      - HOST_HOME=${HOME}
       - PYTHONDONTWRITEBYTECODE
     ports:
       - "${WEBSERVER_HOST_PORT}:8080"