You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2021/08/10 20:36:11 UTC
[airflow] branch v2-1-test-jarek-cherry-picks created (now 3833fb7)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git.
at 3833fb7 Better diagnostics and self-healing of docker-compose (#17484)
This branch includes the following new commits:
new ab3634a Switch back http provider after requests removes LGPL dependency (#16974)
new 68977b9 Remove SQLAlchemy <1.4 constraint (#16630)
new fe5920e Bump Jinja2 upper-bound from 2.12.0 to 4.0.0 (#16595)
new dfeb73d Add type annotations to setup.py (#16658)
new 867adda bump dnspython (#16698)
new 42334b0 AIRFLOW-5529 Add Apache Drill provider. (#16884)
new 37c935d Update alias for field_mask in Google Memmcache (#16975)
new 3c0f4d9 Updates to FlaskAppBuilder 3.3.2+ (#17208)
new 7a43960 Enhancement to bash scripts (#17098)
new 70794b0 Do not pull CI image for ownership fixing on first, fresh breeze run (#17419)
new 38c7115 Increases timeout for helm chart builds (#17417)
new 8609c82 Improve image building documentation for new users (#17409)
new 106d9f0 Optimizes structure of the Dockerfiles and use latest tools (#17418)
new 7371f98 Add timeout when asking whether to rebuild image (#17412)
new 2df7e6e Improve diagnostics message when users have secret_key misconfigured (#17410)
new f196690 Fix failing static checks in main (#17424)
new 8fc7697 Switches to "/" convention in ghcr.io images (#17356)
new 2ba71c9 Fixes some static check errors (#17432)
new 562be8c Disable Helm tests when branch is not main
new 8bd1a7d Attempt to reduce flakiness of PythonVirtualeEnv test_airflow_context (#17486)
new 0948366 Install providers from sources in prod image only on main (#17458)
new 3833fb7 Better diagnostics and self-healing of docker-compose (#17484)
The 22 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[airflow] 20/22: Attempt to reduce flakiness of PythonVirtualeEnv
test_airflow_context (#17486)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 8bd1a7d3148104b9fa920ff59c70896cd0a7c5ef
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sat Aug 7 22:47:15 2021 +0200
Attempt to reduce flakiness of PythonVirtualeEnv test_airflow_context (#17486)
We have a global limit (60 seconds) for individual test execution,
however 'test_airflow_context' of the Python Virtualenv test might take longer in
case they are run in parallel - because they are using dill serialization
including a lot of serializable data from the context of the task.
We give the test 120 seconds to complete now.
(cherry picked from commit 793afaade115dccda7b544cfe8af756b88350abb)
---
scripts/in_container/entrypoint_ci.sh | 2 ++
tests/operators/test_python.py | 3 +++
2 files changed, 5 insertions(+)
diff --git a/scripts/in_container/entrypoint_ci.sh b/scripts/in_container/entrypoint_ci.sh
index 6ba7f6e..0c00163 100755
--- a/scripts/in_container/entrypoint_ci.sh
+++ b/scripts/in_container/entrypoint_ci.sh
@@ -216,6 +216,8 @@ EXTRA_PYTEST_ARGS=(
"--pythonwarnings=ignore::PendingDeprecationWarning"
"--junitxml=${RESULT_LOG_FILE}"
# timeouts in seconds for individual tests
+ "--timeouts-order"
+ "moi"
"--setup-timeout=20"
"--execution-timeout=60"
"--teardown-timeout=20"
diff --git a/tests/operators/test_python.py b/tests/operators/test_python.py
index 3b1ad92..fd7f259 100644
--- a/tests/operators/test_python.py
+++ b/tests/operators/test_python.py
@@ -921,6 +921,9 @@ class TestPythonVirtualenvOperator(unittest.TestCase):
self._run_as_operator(f, templates_dict={'ds': '{{ ds }}'})
+ # This tests might take longer than default 60 seconds as it is serializing a lot of
+ # context using dill (which is slow apparently).
+ @pytest.mark.execution_timeout(120)
def test_airflow_context(self):
def f(
# basic
[airflow] 01/22: Switch back http provider after requests removes
LGPL dependency (#16974)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit ab3634a28cc5601c6bb78a81aaa7f349afa27377
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Tue Jul 13 22:13:30 2021 +0200
Switch back http provider after requests removes LGPL dependency (#16974)
Following merging the https://github.com/psf/requests/pull/5797
and requests 2.26.0 release without LGPL chardet dependency,
we can now bring back http as pre-installed provider as it does
not bring chardet automatically any more.
(cherry picked from commit c46e841519ef2df7dc40ff2596dd49c010514d87)
---
docs/apache-airflow/extra-packages-ref.rst | 2 +-
setup.py | 13 ++++++-------
2 files changed, 7 insertions(+), 8 deletions(-)
diff --git a/docs/apache-airflow/extra-packages-ref.rst b/docs/apache-airflow/extra-packages-ref.rst
index b4b4bb4..b1dff07 100644
--- a/docs/apache-airflow/extra-packages-ref.rst
+++ b/docs/apache-airflow/extra-packages-ref.rst
@@ -258,7 +258,7 @@ Those are extras that provide support for integration with external systems via
+---------------------+-----------------------------------------------------+--------------------------------------+--------------+
| grpc | ``pip install 'apache-airflow[grpc]'`` | Grpc hooks and operators | |
+---------------------+-----------------------------------------------------+--------------------------------------+--------------+
-| http | ``pip install 'apache-airflow[http]'`` | HTTP hooks, operators and sensors | |
+| http | ``pip install 'apache-airflow[http]'`` | HTTP hooks, operators and sensors | * |
+---------------------+-----------------------------------------------------+--------------------------------------+--------------+
| imap | ``pip install 'apache-airflow[imap]'`` | IMAP hooks and sensors | * |
+---------------------+-----------------------------------------------------+--------------------------------------+--------------+
diff --git a/setup.py b/setup.py
index 5d6f752..21097e6 100644
--- a/setup.py
+++ b/setup.py
@@ -231,13 +231,13 @@ dask = [
'distributed>=2.11.1, <2.20',
]
databricks = [
- 'requests>=2.20.0, <3',
+ 'requests>=2.26.0, <3',
]
datadog = [
'datadog>=0.14.0',
]
deprecated_api = [
- 'requests>=2.20.0',
+ 'requests>=2.26.0',
]
doc = [
# Sphinx is limited to < 3.5.0 because of https://github.com/sphinx-doc/sphinx/issues/8880
@@ -330,7 +330,9 @@ hive = [
'thrift>=0.9.2',
]
http = [
- 'requests>=2.20.0',
+ # The 2.26.0 release of requests got rid of the chardet LGPL mandatory dependency, allowing us to
+ # release it as a requirement for airflow
+ 'requests>=2.26.0',
]
http_provider = [
# NOTE ! The HTTP provider is NOT preinstalled by default when Airflow is installed - because it
@@ -810,12 +812,9 @@ EXTRAS_REQUIREMENTS = sort_extras_requirements()
# Those providers are pre-installed always when airflow is installed.
# Those providers do not have dependency on airflow2.0 because that would lead to circular dependencies.
# This is not a problem for PIP but some tools (pipdeptree) show those as a warning.
-# NOTE ! The HTTP provider is NOT preinstalled by default when Airflow is installed - because it
-# depends on `requests` library and until `chardet` is mandatory dependency of `requests`
-# we cannot make it mandatory dependency. See https://github.com/psf/requests/pull/5797
PREINSTALLED_PROVIDERS = [
'ftp',
- # 'http',
+ 'http',
'imap',
'sqlite',
]
[airflow] 10/22: Do not pull CI image for ownership fixing on first,
fresh breeze run (#17419)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 70794b00175e01c18d72fca3ce8eb58ef306e76b
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Wed Aug 4 22:00:33 2021 +0200
Do not pull CI image for ownership fixing on first, fresh breeze run (#17419)
When you run Breeze on fresh machine, this script pulled the CI
image before any operation. It is not harmful in most cases but
it unnecessarily delays the first real image check and rebuild,
where fixing ownership is not really needed (as we've never run
Breeze before).
(cherry picked from commit 537c25417814d86bcee195ee03027840ce5837b6)
---
scripts/ci/tools/fix_ownership.sh | 14 +++++++++-----
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/scripts/ci/tools/fix_ownership.sh b/scripts/ci/tools/fix_ownership.sh
index 6ed1161..de15621 100755
--- a/scripts/ci/tools/fix_ownership.sh
+++ b/scripts/ci/tools/fix_ownership.sh
@@ -33,8 +33,12 @@ sanity_checks::sanitize_mounted_files
read -r -a EXTRA_DOCKER_FLAGS <<<"$(local_mounts::convert_local_mounts_to_docker_params)"
-docker_v run --entrypoint /bin/bash "${EXTRA_DOCKER_FLAGS[@]}" \
- --rm \
- --env-file "${AIRFLOW_SOURCES}/scripts/ci/docker-compose/_docker.env" \
- "${AIRFLOW_CI_IMAGE}" \
- -c /opt/airflow/scripts/in_container/run_fix_ownership.sh || true
+if docker image inspect "${AIRFLOW_CI_IMAGE}" >/dev/null 2>&1; then
+ docker_v run --entrypoint /bin/bash "${EXTRA_DOCKER_FLAGS[@]}" \
+ --rm \
+ --env-file "${AIRFLOW_SOURCES}/scripts/ci/docker-compose/_docker.env" \
+ "${AIRFLOW_CI_IMAGE}" \
+ -c /opt/airflow/scripts/in_container/run_fix_ownership.sh || true
+else
+ echo "Skip fixing ownership as seems that you do not have the ${AIRFLOW_CI_IMAGE} image yet"
+fi
[airflow] 03/22: Bump Jinja2 upper-bound from 2.12.0 to 4.0.0
(#16595)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit fe5920e5ed3e1158bffbc3a15d24d57c10d79843
Author: Ashwin Madavan <as...@gmail.com>
AuthorDate: Thu Jun 24 15:07:23 2021 -0400
Bump Jinja2 upper-bound from 2.12.0 to 4.0.0 (#16595)
(cherry picked from commit 5d5268f5e553a7031ebfb08754c31fca5c13bda7)
---
setup.cfg | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/setup.cfg b/setup.cfg
index 0e4868f..fbe58cb 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -114,7 +114,7 @@ install_requires =
iso8601>=0.1.12
# Logging is broken with itsdangerous > 2
itsdangerous>=1.1.0, <2.0
- jinja2>=2.10.1, <2.12.0
+ jinja2>=2.10.1,<4
jsonschema~=3.0
lazy-object-proxy
lockfile>=0.12.2
[airflow] 04/22: Add type annotations to setup.py (#16658)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit dfeb73da4f1d3f7a18ad6ab0e320b780588abe13
Author: Ali Muhammad <al...@gmail.com>
AuthorDate: Fri Jun 25 22:25:52 2021 +0500
Add type annotations to setup.py (#16658)
(cherry picked from commit 402274641168f412f44c545c34f3e7edf5cf1476)
---
setup.py | 40 ++++++++++++++++++++++++----------------
1 file changed, 24 insertions(+), 16 deletions(-)
diff --git a/setup.py b/setup.py
index 21097e6..e6c1d46 100644
--- a/setup.py
+++ b/setup.py
@@ -62,14 +62,14 @@ class CleanCommand(Command):
description = "Tidy up the project root"
user_options: List[str] = []
- def initialize_options(self):
+ def initialize_options(self) -> None:
"""Set default values for options."""
- def finalize_options(self):
+ def finalize_options(self) -> None:
"""Set final values for options."""
@staticmethod
- def rm_all_files(files: List[str]):
+ def rm_all_files(files: List[str]) -> None:
"""Remove all files from the list"""
for file in files:
try:
@@ -77,7 +77,7 @@ class CleanCommand(Command):
except Exception as e:
logger.warning("Error when removing %s: %s", file, e)
- def run(self):
+ def run(self) -> None:
"""Remove temporary files and directories."""
os.chdir(my_dir)
self.rm_all_files(glob.glob('./build/*'))
@@ -98,10 +98,10 @@ class CompileAssets(Command):
description = "Compile and build the frontend assets"
user_options: List[str] = []
- def initialize_options(self):
+ def initialize_options(self) -> None:
"""Set default values for options."""
- def finalize_options(self):
+ def finalize_options(self) -> None:
"""Set final values for options."""
def run(self) -> None:
@@ -118,10 +118,10 @@ class ListExtras(Command):
description = "List available extras"
user_options: List[str] = []
- def initialize_options(self):
+ def initialize_options(self) -> None:
"""Set default values for options."""
- def finalize_options(self):
+ def finalize_options(self) -> None:
"""Set final values for options."""
def run(self) -> None:
@@ -165,7 +165,7 @@ def git_version(version_: str) -> str:
return 'no_git_version'
-def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version"])):
+def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version"])) -> None:
"""
Write the Semver version + git hash to file, e.g. ".dev0+2f635dc265e78db6708f59f68e8009abb92c1e65".
@@ -766,7 +766,7 @@ PACKAGES_EXCLUDED_FOR_ALL.extend(
)
-def is_package_excluded(package: str, exclusion_list: List[str]):
+def is_package_excluded(package: str, exclusion_list: List[str]) -> bool:
"""
Checks if package should be excluded.
@@ -820,7 +820,7 @@ PREINSTALLED_PROVIDERS = [
]
-def get_provider_package_from_package_id(package_id: str):
+def get_provider_package_from_package_id(package_id: str) -> str:
"""
Builds the name of provider package out of the package id provided/
@@ -831,16 +831,18 @@ def get_provider_package_from_package_id(package_id: str):
return f"apache-airflow-providers-{package_suffix}"
-def get_excluded_providers():
+def get_excluded_providers() -> List[str]:
"""
Returns packages excluded for the current python version.
+
Currently the only excluded provider is apache hive for Python 3.9.
Until https://github.com/dropbox/PyHive/issues/380 is fixed.
+
"""
return ['apache.hive'] if PY39 else []
-def get_all_provider_packages():
+def get_all_provider_packages() -> str:
"""Returns all provider packages configured in setup.py"""
excluded_providers = get_excluded_providers()
return " ".join(
@@ -851,7 +853,13 @@ def get_all_provider_packages():
class AirflowDistribution(Distribution):
- """The setuptools.Distribution subclass with Airflow specific behaviour"""
+ """
+ The setuptools.Distribution subclass with Airflow specific behaviour
+
+ The reason for pylint: disable=signature-differs of parse_config_files is explained here:
+ https://github.com/PyCQA/pylint/issues/3737
+
+ """
def parse_config_files(self, *args, **kwargs) -> None:
"""
@@ -949,7 +957,7 @@ def add_all_provider_packages() -> None:
class Develop(develop_orig):
"""Forces removal of providers in editable mode."""
- def run(self):
+ def run(self) -> None:
self.announce('Installing in editable mode. Uninstalling provider packages!', level=log.INFO)
# We need to run "python3 -m pip" because it might be that older PIP binary is in the path
# And it results with an error when running pip directly (cannot import pip module)
@@ -973,7 +981,7 @@ class Develop(develop_orig):
class Install(install_orig):
"""Forces installation of providers from sources in editable mode."""
- def run(self):
+ def run(self) -> None:
self.announce('Standard installation. Providers are installed from packages', level=log.INFO)
super().run()
[airflow] 05/22: bump dnspython (#16698)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 867adda5e28b5d3c366e3cd3e1f8f60f5f412c4b
Author: kurtqq <47...@users.noreply.github.com>
AuthorDate: Tue Jun 29 00:21:24 2021 +0300
bump dnspython (#16698)
(cherry picked from commit 57dcac22137bc958c1ed9f12fa54484e13411a6f)
---
setup.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/setup.py b/setup.py
index e6c1d46..9e8c28d 100644
--- a/setup.py
+++ b/setup.py
@@ -370,7 +370,7 @@ ldap = [
]
leveldb = ['plyvel']
mongo = [
- 'dnspython>=1.13.0,<2.0.0',
+ 'dnspython>=1.13.0,<3.0.0',
'pymongo>=3.6.0',
]
mssql = [
[airflow] 13/22: Optimizes structure of the Dockerfiles and use
latest tools (#17418)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 106d9f09852090da21f3dfb744c69e4be5fd90e5
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Wed Aug 4 23:32:12 2021 +0200
Optimizes structure of the Dockerfiles and use latest tools (#17418)
* Remove CONTINUE_ON_PIP_CHECK_FAILURE parameter
This parameter was useful when upgrading new dependencies,
however it is going to be replaced with better approach in the
upcoming image convention change.
* Optimizes structure of the Dockerfiles and use latest tools
This PR optimizes the structure of Dockerfile by moving some
expensive operations before the COPY sources so that
rebuilding image when only few sources change is much faster.
At the same time, we upgrade PIP and HELM chart used to latest
versions and clean-up some parameter inconsistencies.
(cherry picked from commit 94b03f6f43277e7332c25fdc63aedfde605f9773)
---
.github/workflows/build-images.yml | 1 -
BREEZE.rst | 10 +----
Dockerfile | 27 +++++------
Dockerfile.ci | 52 +++++++++++-----------
IMAGES.rst | 6 ---
breeze | 9 ----
breeze-complete | 2 +-
docs/docker-stack/build-arg-ref.rst | 6 ---
scripts/ci/libraries/_build_images.sh | 20 ---------
scripts/ci/libraries/_initialization.sh | 10 ++---
scripts/docker/install_additional_dependencies.sh | 5 +--
scripts/docker/install_airflow.sh | 4 +-
...nstall_airflow_dependencies_from_branch_tip.sh} | 11 ++---
.../docker/install_from_docker_context_files.sh | 2 +-
14 files changed, 55 insertions(+), 110 deletions(-)
diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml
index f29e199..ec8f435 100644
--- a/.github/workflows/build-images.yml
+++ b/.github/workflows/build-images.yml
@@ -148,7 +148,6 @@ jobs:
BACKEND: postgres
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgradeToNewerDependencies }}
- CONTINUE_ON_PIP_CHECK_FAILURE: "true"
DOCKER_CACHE: ${{ needs.build-info.outputs.cacheDirective }}
CHECK_IF_BASE_PYTHON_IMAGE_UPDATED: >
${{ github.event_name == 'pull_request_target' && 'false' || 'true' }}
diff --git a/BREEZE.rst b/BREEZE.rst
index 90d3f0b..468709b 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -1279,9 +1279,6 @@ This is the current syntax for `./breeze <./breeze>`_:
--upgrade-to-newer-dependencies
Upgrades PIP packages to latest versions available without looking at the constraints.
- --continue-on-pip-check-failure
- Continue even if 'pip check' fails.
-
-I, --production-image
Use production image for entering the environment and builds (not for tests).
@@ -2382,9 +2379,9 @@ This is the current syntax for `./breeze <./breeze>`_:
Helm version - only used in case one of kind-cluster commands is used.
One of:
- v3.2.4
+ v3.6.3
- Default: v3.2.4
+ Default: v3.6.3
--executor EXECUTOR
Executor to use in a kubernetes cluster.
@@ -2435,9 +2432,6 @@ This is the current syntax for `./breeze <./breeze>`_:
--upgrade-to-newer-dependencies
Upgrades PIP packages to latest versions available without looking at the constraints.
- --continue-on-pip-check-failure
- Continue even if 'pip check' fails.
-
****************************************************************************************************
Use different Airflow version at runtime in CI image
diff --git a/Dockerfile b/Dockerfile
index 66c9649..210918c 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -44,7 +44,8 @@ ARG AIRFLOW_GID="50000"
ARG PYTHON_BASE_IMAGE="python:3.6-slim-buster"
-ARG AIRFLOW_PIP_VERSION=21.1.2
+ARG AIRFLOW_PIP_VERSION=21.2.2
+ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow"
# By default PIP has progress bar but you can disable it.
ARG PIP_PROGRESS_BAR="on"
@@ -108,12 +109,13 @@ ARG DEV_APT_COMMAND="\
&& curl https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - > /dev/null \
&& echo 'deb https://dl.yarnpkg.com/debian/ stable main' > /etc/apt/sources.list.d/yarn.list"
ARG ADDITIONAL_DEV_APT_COMMAND="echo"
+ARG ADDITIONAL_DEV_APT_ENV=""
ENV DEV_APT_DEPS=${DEV_APT_DEPS} \
ADDITIONAL_DEV_APT_DEPS=${ADDITIONAL_DEV_APT_DEPS} \
DEV_APT_COMMAND=${DEV_APT_COMMAND} \
ADDITIONAL_DEV_APT_COMMAND=${ADDITIONAL_DEV_APT_COMMAND} \
- ADDITIONAL_DEV_APT_ENV=""
+ ADDITIONAL_DEV_APT_ENV=${ADDITIONAL_DEV_APT_ENV}
# Note missing man directories on debian-buster
# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199
@@ -216,7 +218,7 @@ ENV AIRFLOW_PRE_CACHED_PIP_PACKAGES=${AIRFLOW_PRE_CACHED_PIP_PACKAGES} \
RUN bash /scripts/docker/install_pip_version.sh; \
if [[ ${AIRFLOW_PRE_CACHED_PIP_PACKAGES} == "true" && \
${UPGRADE_TO_NEWER_DEPENDENCIES} == "false" ]]; then \
- bash /scripts/docker/install_airflow_from_branch_tip.sh; \
+ bash /scripts/docker/install_airflow_dependencies_from_branch_tip.sh; \
fi
COPY ${AIRFLOW_SOURCES_FROM} ${AIRFLOW_SOURCES_TO}
@@ -236,14 +238,11 @@ ARG INSTALL_FROM_PYPI="true"
# * pyjwt<2.0.0: flask-jwt-extended requires it
# * dill<0.3.3 required by apache-beam
ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="pyjwt<2.0.0 dill<0.3.3 certifi<2021.0.0"
-ARG CONTINUE_ON_PIP_CHECK_FAILURE="false"
-
ENV ADDITIONAL_PYTHON_DEPS=${ADDITIONAL_PYTHON_DEPS} \
INSTALL_FROM_DOCKER_CONTEXT_FILES=${INSTALL_FROM_DOCKER_CONTEXT_FILES} \
INSTALL_FROM_PYPI=${INSTALL_FROM_PYPI} \
- EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS=${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS} \
- CONTINUE_ON_PIP_CHECK_FAILURE=${CONTINUE_ON_PIP_CHECK_FAILURE}
+ EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS=${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS}
WORKDIR /opt/airflow
@@ -276,7 +275,7 @@ RUN if [[ -f /docker-context-files/requirements.txt ]]; then \
ARG BUILD_ID
ARG COMMIT_SHA
-ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow"
+ARG AIRFLOW_IMAGE_REPOSITORY
ARG AIRFLOW_IMAGE_DATE_CREATED
ENV BUILD_ID=${BUILD_ID} COMMIT_SHA=${COMMIT_SHA}
@@ -293,15 +292,14 @@ LABEL org.apache.airflow.distro="debian" \
org.opencontainers.image.created=${AIRFLOW_IMAGE_DATE_CREATED} \
org.opencontainers.image.authors="dev@airflow.apache.org" \
org.opencontainers.image.url="https://airflow.apache.org" \
- org.opencontainers.image.documentation="https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html" \
- org.opencontainers.image.source="https://github.com/apache/airflow" \
+ org.opencontainers.image.documentation="https://airflow.apache.org/docs/docker-stack/index.html" \
org.opencontainers.image.version="${AIRFLOW_VERSION}" \
org.opencontainers.image.revision="${COMMIT_SHA}" \
org.opencontainers.image.vendor="Apache Software Foundation" \
org.opencontainers.image.licenses="Apache-2.0" \
org.opencontainers.image.ref.name="airflow-build-image" \
org.opencontainers.image.title="Build Image Segment for Production Airflow Image" \
- org.opencontainers.image.description="Installed Apache Airflow with build-time dependencies"
+ org.opencontainers.image.description="Reference build-time dependencies image for production-ready Apache Airflow image"
##############################################################################################
# This is the actual Airflow image - much smaller than the build one. We copy
@@ -379,7 +377,7 @@ ARG AIRFLOW_HOME
ARG AIRFLOW_INSTALLATION_METHOD="apache-airflow"
ARG BUILD_ID
ARG COMMIT_SHA
-ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow"
+ARG AIRFLOW_IMAGE_REPOSITORY
ARG AIRFLOW_IMAGE_DATE_CREATED
# By default PIP will install everything in ~/.local
ARG PIP_USER="true"
@@ -468,15 +466,14 @@ LABEL org.apache.airflow.distro="debian" \
org.opencontainers.image.created=${AIRFLOW_IMAGE_DATE_CREATED} \
org.opencontainers.image.authors="dev@airflow.apache.org" \
org.opencontainers.image.url="https://airflow.apache.org" \
- org.opencontainers.image.documentation="https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html" \
- org.opencontainers.image.source="https://github.com/apache/airflow" \
+ org.opencontainers.image.documentation="https://airflow.apache.org/docs/docker-stack/index.html" \
org.opencontainers.image.version="${AIRFLOW_VERSION}" \
org.opencontainers.image.revision="${COMMIT_SHA}" \
org.opencontainers.image.vendor="Apache Software Foundation" \
org.opencontainers.image.licenses="Apache-2.0" \
org.opencontainers.image.ref.name="airflow" \
org.opencontainers.image.title="Production Airflow Image" \
- org.opencontainers.image.description="Installed Apache Airflow"
+ org.opencontainers.image.description="Reference, production-ready Apache Airflow image"
ENTRYPOINT ["/usr/bin/dumb-init", "--", "/entrypoint"]
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 7b1dbf1..8d599db 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -22,6 +22,8 @@ SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"]
ARG PYTHON_BASE_IMAGE="python:3.6-slim-buster"
ARG AIRFLOW_VERSION="2.1.0.dev0"
+ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow"
+
# By increasing this number we can do force build of all dependencies
ARG DEPENDENCIES_EPOCH_NUMBER="6"
@@ -125,6 +127,15 @@ ARG RUNTIME_APT_DEPS="\
unzip \
vim \
xxd"
+
+# Install Helm
+ARG HELM_VERSION="v3.6.3"
+
+RUN SYSTEM=$(uname -s | tr '[:upper:]' '[:lower:]') \
+ && HELM_URL="https://get.helm.sh/helm-${HELM_VERSION}-${SYSTEM}-amd64.tar.gz" \
+ && curl --location "${HELM_URL}" | tar -xvz -O "${SYSTEM}"-amd64/helm > /usr/local/bin/helm \
+ && chmod +x /usr/local/bin/helm
+
ARG ADDITIONAL_RUNTIME_APT_DEPS=""
ARG RUNTIME_APT_COMMAND=""
ARG ADDITIONAL_RUNTIME_APT_COMMAND=""
@@ -207,7 +218,7 @@ ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="true"
# By default in the image, we are installing all providers when installing from sources
ARG INSTALL_PROVIDERS_FROM_SOURCES="true"
ARG INSTALL_FROM_PYPI="true"
-ARG AIRFLOW_PIP_VERSION=21.1.2
+ARG AIRFLOW_PIP_VERSION=21.2.2
# Setup PIP
# By default PIP install run without cache to make image smaller
ARG PIP_NO_CACHE_DIR="true"
@@ -270,7 +281,7 @@ ENV EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS=${EAGER_UPGRADE_ADDITIONAL_REQUIREMENT
RUN bash /scripts/docker/install_pip_version.sh; \
if [[ ${AIRFLOW_PRE_CACHED_PIP_PACKAGES} == "true" && \
${UPGRADE_TO_NEWER_DEPENDENCIES} == "false" ]]; then \
- bash /scripts/docker/install_airflow_from_branch_tip.sh; \
+ bash /scripts/docker/install_airflow_dependencies_from_branch_tip.sh; \
fi
# Generate random hex dump file so that we can determine whether it's faster to rebuild the image
@@ -299,8 +310,6 @@ COPY setup.cfg ${AIRFLOW_SOURCES}/setup.cfg
COPY airflow/__init__.py ${AIRFLOW_SOURCES}/airflow/__init__.py
-ARG CONTINUE_ON_PIP_CHECK_FAILURE="false"
-
# The goal of this line is to install the dependencies from the most current setup.py from sources
# This will be usually incremental small set of packages in CI optimized build, so it will be very fast
# In non-CI optimized build this will install all dependencies before installing sources.
@@ -325,11 +334,13 @@ RUN chmod a+x /entrypoint
COPY scripts/docker/load.bash /opt/bats/lib/
-# We can copy everything here. The Context is filtered by dockerignore. This makes sure we are not
-# copying over stuff that is accidentally generated or that we do not need (such as egg-info)
-# if you want to add something that is missing and you expect to see it in the image you can
-# add it with ! in .dockerignore next to the airflow, test etc. directories there
-COPY . ${AIRFLOW_SOURCES}/
+# Additional python deps to install
+ARG ADDITIONAL_PYTHON_DEPS=""
+
+RUN bash /scripts/docker/install_pip_version.sh; \
+ if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
+ bash /scripts/docker/install_additional_dependencies.sh; \
+ fi
# Install autocomplete for airflow
RUN if command -v airflow; then \
@@ -339,27 +350,16 @@ RUN if command -v airflow; then \
# Install autocomplete for Kubectl
RUN echo "source /etc/bash_completion" >> ~/.bashrc
-WORKDIR ${AIRFLOW_SOURCES}
-
-# Install Helm
-ARG HELM_VERSION="v3.2.4"
-
-RUN SYSTEM=$(uname -s | tr '[:upper:]' '[:lower:]') \
- && HELM_URL="https://get.helm.sh/helm-${HELM_VERSION}-${SYSTEM}-amd64.tar.gz" \
- && curl --location "${HELM_URL}" | tar -xvz -O "${SYSTEM}"-amd64/helm > /usr/local/bin/helm \
- && chmod +x /usr/local/bin/helm
-
-# Additional python deps to install
-ARG ADDITIONAL_PYTHON_DEPS=""
+# We can copy everything here. The Context is filtered by dockerignore. This makes sure we are not
+# copying over stuff that is accidentally generated or that we do not need (such as egg-info)
+# if you want to add something that is missing and you expect to see it in the image you can
+# add it with ! in .dockerignore next to the airflow, test etc. directories there
+COPY . ${AIRFLOW_SOURCES}/
-RUN bash /scripts/docker/install_pip_version.sh; \
- if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
- bash /scripts/docker/install_additional_dependencies.sh; \
- fi
+WORKDIR ${AIRFLOW_SOURCES}
ARG BUILD_ID
ARG COMMIT_SHA
-ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow"
ARG AIRFLOW_IMAGE_DATE_CREATED
ENV PATH="/files/bin/:/opt/airflow/scripts/in_container/bin/:${HOME}:${PATH}" \
diff --git a/IMAGES.rst b/IMAGES.rst
index 82e6989..bc34e6c 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -445,12 +445,6 @@ The following build arguments (``--build-arg`` in docker build command) can be u
| | | upgraded to newer versions matching |
| | | setup.py before installation. |
+------------------------------------------+------------------------------------------+------------------------------------------+
-| ``CONTINUE_ON_PIP_CHECK_FAILURE`` | ``false`` | By default the image will fail if pip |
-| | | check fails for it. This is good for |
-| | | interactive building but on CI the |
-| | | image should be built regardless - we |
-| | | have a separate step to verify image. |
-+------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_PRE_CACHED_PIP_PACKAGES`` | ``true`` | Allows to pre-cache airflow PIP packages |
| | | from the GitHub of Apache Airflow |
| | | This allows to optimize iterations for |
diff --git a/breeze b/breeze
index 7decdf6..1f45ed1 100755
--- a/breeze
+++ b/breeze
@@ -1177,12 +1177,6 @@ function breeze::parse_arguments() {
echo
shift
;;
- --continue-on-pip-check-failure)
- export CONTINUE_ON_PIP_CHECK_FAILURE="true"
- echo "Skip PIP check failure."
- echo
- shift
- ;;
--package-format)
export PACKAGE_FORMAT="${2}"
echo "Selected package type: ${PACKAGE_FORMAT}"
@@ -2400,9 +2394,6 @@ ${FORMATTED_INSTALLATION_METHOD}
--upgrade-to-newer-dependencies
Upgrades PIP packages to latest versions available without looking at the constraints.
---continue-on-pip-check-failure
- Continue even if 'pip check' fails.
-
"
}
diff --git a/breeze-complete b/breeze-complete
index db9e42a..045bb65 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -29,7 +29,7 @@ _breeze_allowed_integrations="cassandra kerberos mongo openldap pinot rabbitmq r
_breeze_allowed_generate_constraints_modes="source-providers pypi-providers no-providers"
_breeze_allowed_kubernetes_modes="image"
_breeze_allowed_kubernetes_versions="v1.20.2 v1.19.7 v1.18.15"
-_breeze_allowed_helm_versions="v3.2.4"
+_breeze_allowed_helm_versions="v3.6.3"
_breeze_allowed_kind_versions="v0.11.1"
_breeze_allowed_mysql_versions="5.7 8"
_breeze_allowed_postgres_versions="9.6 10 11 12 13"
diff --git a/docs/docker-stack/build-arg-ref.rst b/docs/docker-stack/build-arg-ref.rst
index 8780970..f2507e0 100644
--- a/docs/docker-stack/build-arg-ref.rst
+++ b/docs/docker-stack/build-arg-ref.rst
@@ -79,12 +79,6 @@ for examples of using those arguments.
+------------------------------------------+------------------------------------------+------------------------------------------+
| Build argument | Default value | Description |
+==========================================+==========================================+==========================================+
-| ``CONTINUE_ON_PIP_CHECK_FAILURE`` | ``false`` | By default the image build fails if pip |
-| | | check fails for it. This is good for |
-| | | interactive building but on CI the |
-| | | image should be built regardless - we |
-| | | have a separate step to verify image. |
-+------------------------------------------+------------------------------------------+------------------------------------------+
| ``UPGRADE_TO_NEWER_DEPENDENCIES`` | ``false`` | If set to true, the dependencies are |
| | | upgraded to newer versions matching |
| | | setup.py before installation. |
diff --git a/scripts/ci/libraries/_build_images.sh b/scripts/ci/libraries/_build_images.sh
index ca94e4c..4ead442 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -669,7 +669,6 @@ Docker building ${AIRFLOW_CI_IMAGE}.
--build-arg ADDITIONAL_RUNTIME_APT_DEPS="${ADDITIONAL_RUNTIME_APT_DEPS}" \
--build-arg ADDITIONAL_RUNTIME_APT_ENV="${ADDITIONAL_RUNTIME_APT_ENV}" \
--build-arg UPGRADE_TO_NEWER_DEPENDENCIES="${UPGRADE_TO_NEWER_DEPENDENCIES}" \
- --build-arg CONTINUE_ON_PIP_CHECK_FAILURE="${CONTINUE_ON_PIP_CHECK_FAILURE}" \
--build-arg CONSTRAINTS_GITHUB_REPOSITORY="${CONSTRAINTS_GITHUB_REPOSITORY}" \
--build-arg AIRFLOW_CONSTRAINTS_REFERENCE="${DEFAULT_CONSTRAINTS_BRANCH}" \
--build-arg AIRFLOW_CONSTRAINTS="${AIRFLOW_CONSTRAINTS}" \
@@ -810,7 +809,6 @@ function build_images::build_prod_images() {
--build-arg INSTALL_FROM_PYPI="${INSTALL_FROM_PYPI}" \
--build-arg INSTALL_FROM_DOCKER_CONTEXT_FILES="${INSTALL_FROM_DOCKER_CONTEXT_FILES}" \
--build-arg UPGRADE_TO_NEWER_DEPENDENCIES="${UPGRADE_TO_NEWER_DEPENDENCIES}" \
- --build-arg CONTINUE_ON_PIP_CHECK_FAILURE="${CONTINUE_ON_PIP_CHECK_FAILURE}" \
--build-arg BUILD_ID="${CI_BUILD_ID}" \
--build-arg COMMIT_SHA="${COMMIT_SHA}" \
--build-arg CONSTRAINTS_GITHUB_REPOSITORY="${CONSTRAINTS_GITHUB_REPOSITORY}" \
@@ -845,7 +843,6 @@ function build_images::build_prod_images() {
--build-arg INSTALL_FROM_PYPI="${INSTALL_FROM_PYPI}" \
--build-arg INSTALL_FROM_DOCKER_CONTEXT_FILES="${INSTALL_FROM_DOCKER_CONTEXT_FILES}" \
--build-arg UPGRADE_TO_NEWER_DEPENDENCIES="${UPGRADE_TO_NEWER_DEPENDENCIES}" \
- --build-arg CONTINUE_ON_PIP_CHECK_FAILURE="${CONTINUE_ON_PIP_CHECK_FAILURE}" \
--build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
--build-arg AIRFLOW_BRANCH="${AIRFLOW_BRANCH_FOR_PYPI_PRELOADING}" \
--build-arg AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS}" \
@@ -1014,23 +1011,6 @@ ${COLOR_BLUE}
./breeze build-image --production-image --upgrade-to-newer-dependencies --python 3.6
${COLOR_RESET}
-* If you want to build the image regardless if 'pip check' fails for it, you can add
- --continue-on-pip-check-failure flag and enter the image and inspect dependencies.
-
-CI image:
-
-${COLOR_BLUE}
- ./breeze build-image --upgrade-to-newer-dependencies --python 3.6 --continue-on-pip-check-failure
- docker run -it apache/airflow:main-3.6-ci bash
-${COLOR_RESET}
-
-Production image:
-
-${COLOR_BLUE}
- ./breeze build-image --production-image --upgrade-to-newer-dependencies --python 3.6 --continue-on-pip-check-failure
- docker run -it apache/airflow:main-3.6 bash
-${COLOR_RESET}
-
* You will see error messages there telling which requirements are conflicting and which packages caused the
conflict. Add the limitation that caused the conflict to EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS
variable in Dockerfile.ci. Note that the limitations might be different for Dockerfile.ci and Dockerfile
diff --git a/scripts/ci/libraries/_initialization.sh b/scripts/ci/libraries/_initialization.sh
index cbeb932..2cebe7d 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -200,7 +200,7 @@ function initialization::initialize_files_for_rebuild_check() {
"scripts/docker/common.sh"
"scripts/docker/install_additional_dependencies.sh"
"scripts/docker/install_airflow.sh"
- "scripts/docker/install_airflow_from_branch_tip.sh"
+ "scripts/docker/install_airflow_dependencies_from_branch_tip.sh"
"scripts/docker/install_from_docker_context_files.sh"
"scripts/docker/install_mysql.sh"
"airflow/www/package.json"
@@ -401,7 +401,7 @@ function initialization::initialize_image_build_variables() {
export INSTALLED_PROVIDERS
export INSTALLED_EXTRAS="async,amazon,celery,cncf.kubernetes,docker,dask,elasticsearch,ftp,grpc,hashicorp,http,imap,ldap,google,microsoft.azure,mysql,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv"
- AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION:="21.1"}
+ AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION:="21.2.2"}
export AIRFLOW_PIP_VERSION
# We also pin version of wheel used to get consistent builds
@@ -425,9 +425,6 @@ function initialization::initialize_image_build_variables() {
# Installs different airflow version than current from the sources
export INSTALL_AIRFLOW_VERSION=${INSTALL_AIRFLOW_VERSION:=""}
- # Continue on PIP CHECK failure
- export CONTINUE_ON_PIP_CHECK_FAILURE=${CONTINUE_ON_PIP_CHECK_FAILURE:="false"}
-
# Determines if airflow should be installed from a specified reference in GitHub
export INSTALL_AIRFLOW_REFERENCE=${INSTALL_AIRFLOW_REFERENCE:=""}
@@ -482,7 +479,7 @@ function initialization::initialize_kubernetes_variables() {
CURRENT_KIND_VERSIONS+=("v0.11.1")
export CURRENT_KIND_VERSIONS
# Currently supported versions of Helm
- CURRENT_HELM_VERSIONS+=("v3.2.4")
+ CURRENT_HELM_VERSIONS+=("v3.6.3")
export CURRENT_HELM_VERSIONS
# Current executor in chart
CURRENT_EXECUTOR+=("KubernetesExecutor")
@@ -663,7 +660,6 @@ Common image build variables:
INSTALL_FROM_PYPI: '${INSTALL_FROM_PYPI}'
AIRFLOW_PRE_CACHED_PIP_PACKAGES: '${AIRFLOW_PRE_CACHED_PIP_PACKAGES}'
UPGRADE_TO_NEWER_DEPENDENCIES: '${UPGRADE_TO_NEWER_DEPENDENCIES}'
- CONTINUE_ON_PIP_CHECK_FAILURE: '${CONTINUE_ON_PIP_CHECK_FAILURE}'
CHECK_IMAGE_FOR_REBUILD: '${CHECK_IMAGE_FOR_REBUILD}'
AIRFLOW_CONSTRAINTS_LOCATION: '${AIRFLOW_CONSTRAINTS_LOCATION}'
AIRFLOW_CONSTRAINTS_REFERENCE: '${AIRFLOW_CONSTRAINTS_REFERENCE}'
diff --git a/scripts/docker/install_additional_dependencies.sh b/scripts/docker/install_additional_dependencies.sh
index 6c035ae..4f9c05f 100755
--- a/scripts/docker/install_additional_dependencies.sh
+++ b/scripts/docker/install_additional_dependencies.sh
@@ -23,7 +23,6 @@ test -v ADDITIONAL_PYTHON_DEPS
test -v EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS
test -v AIRFLOW_INSTALL_USER_FLAG
test -v AIRFLOW_PIP_VERSION
-test -v CONTINUE_ON_PIP_CHECK_FAILURE
# shellcheck source=scripts/docker/common.sh
. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
@@ -41,7 +40,7 @@ function install_additional_dependencies() {
${ADDITIONAL_PYTHON_DEPS} ${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS}
# make sure correct PIP version is used
pip install ${AIRFLOW_INSTALL_USER_FLAG} --upgrade "pip==${AIRFLOW_PIP_VERSION}"
- pip check || ${CONTINUE_ON_PIP_CHECK_FAILURE}
+ pip check
else
echo
echo Installing additional dependencies upgrading only if needed
@@ -51,7 +50,7 @@ function install_additional_dependencies() {
${ADDITIONAL_PYTHON_DEPS}
# make sure correct PIP version is used
pip install ${AIRFLOW_INSTALL_USER_FLAG} --upgrade "pip==${AIRFLOW_PIP_VERSION}"
- pip check || ${CONTINUE_ON_PIP_CHECK_FAILURE}
+ pip check
fi
}
diff --git a/scripts/docker/install_airflow.sh b/scripts/docker/install_airflow.sh
index 4904027..e2bca4f 100755
--- a/scripts/docker/install_airflow.sh
+++ b/scripts/docker/install_airflow.sh
@@ -60,7 +60,7 @@ function install_airflow() {
# make sure correct PIP version is used
pip install ${AIRFLOW_INSTALL_USER_FLAG} --upgrade "pip==${AIRFLOW_PIP_VERSION}"
- pip check || ${CONTINUE_ON_PIP_CHECK_FAILURE}
+ pip check
else \
echo
echo Installing all packages with constraints and upgrade if needed
@@ -76,7 +76,7 @@ function install_airflow() {
"${AIRFLOW_INSTALLATION_METHOD}[${AIRFLOW_EXTRAS}]${AIRFLOW_VERSION_SPECIFICATION}" \
# make sure correct PIP version is used
pip install ${AIRFLOW_INSTALL_USER_FLAG} --upgrade "pip==${AIRFLOW_PIP_VERSION}"
- pip check || ${CONTINUE_ON_PIP_CHECK_FAILURE}
+ pip check
fi
}
diff --git a/scripts/docker/install_airflow_from_branch_tip.sh b/scripts/docker/install_airflow_dependencies_from_branch_tip.sh
similarity index 85%
rename from scripts/docker/install_airflow_from_branch_tip.sh
rename to scripts/docker/install_airflow_dependencies_from_branch_tip.sh
index 925a872..61aaa13 100755
--- a/scripts/docker/install_airflow_from_branch_tip.sh
+++ b/scripts/docker/install_airflow_dependencies_from_branch_tip.sh
@@ -30,28 +30,29 @@
. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
-function install_airflow_from_branch_tip() {
+function install_airflow_dependencies_from_branch_tip() {
echo
echo "Installing airflow from ${AIRFLOW_BRANCH}. It is used to cache dependencies"
echo
if [[ ${INSTALL_MYSQL_CLIENT} != "true" ]]; then
AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS/mysql,}
fi
- # Install latest set of dependencies using constraints
+ # Install latest set of dependencies using constraints. In case constraints were upgraded and there
+ # are conflicts, this might fail, but it should be fixed in the following installation steps
pip install ${AIRFLOW_INSTALL_USER_FLAG} \
"https://github.com/${AIRFLOW_REPO}/archive/${AIRFLOW_BRANCH}.tar.gz#egg=apache-airflow[${AIRFLOW_EXTRAS}]" \
- --constraint "${AIRFLOW_CONSTRAINTS_LOCATION}"
+ --constraint "${AIRFLOW_CONSTRAINTS_LOCATION}" || true
# make sure correct PIP version is used
pip install ${AIRFLOW_INSTALL_USER_FLAG} --upgrade "pip==${AIRFLOW_PIP_VERSION}"
pip freeze | grep apache-airflow-providers | xargs pip uninstall --yes || true
echo
echo Uninstalling just airflow. Dependencies remain.
echo
- pip uninstall --yes apache-airflow
+ pip uninstall --yes apache-airflow || true
}
common::get_airflow_version_specification
common::override_pip_version_if_needed
common::get_constraints_location
-install_airflow_from_branch_tip
+install_airflow_dependencies_from_branch_tip
diff --git a/scripts/docker/install_from_docker_context_files.sh b/scripts/docker/install_from_docker_context_files.sh
index 813d1b0..d8ed6bc 100755
--- a/scripts/docker/install_from_docker_context_files.sh
+++ b/scripts/docker/install_from_docker_context_files.sh
@@ -96,7 +96,7 @@ function install_airflow_and_providers_from_docker_context_files(){
# make sure correct PIP version is left installed
pip install ${AIRFLOW_INSTALL_USER_FLAG} --upgrade "pip==${AIRFLOW_PIP_VERSION}"
- pip check || ${CONTINUE_ON_PIP_CHECK_FAILURE}
+ pip check
}
[airflow] 08/22: Updates to FlaskAppBuilder 3.3.2+ (#17208)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 3c0f4d9908a81b662296e747230dba5ddb96f213
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Wed Jul 28 21:37:48 2021 +0200
Updates to FlaskAppBuilder 3.3.2+ (#17208)
There are some clarifications about using the authentication
via FlaskAppBuilder - the change implements minimum version of the
FAB to 3.3.2 and clarifies the dependencies used in FAB 3 series
to be only authlib rather than flask-oauth.
Fixes: #16944 (this is the second, proper fix this time).
(cherry picked from commit 6d7fa874ff201af7f602be9c58a827998814bdd1)
---
setup.cfg | 2 +-
setup.py | 10 ++++------
2 files changed, 5 insertions(+), 7 deletions(-)
diff --git a/setup.cfg b/setup.cfg
index fbe58cb..d3c5f57 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -100,7 +100,7 @@ install_requires =
# https://github.com/readthedocs/sphinx_rtd_theme/issues/1115
docutils<0.17
flask>=1.1.0, <2.0
- flask-appbuilder~=3.3
+ flask-appbuilder>=3.3.2, <4.0.0
flask-caching>=1.5.0, <2.0.0
flask-login>=0.3, <0.5
flask-wtf>=0.14.3, <0.15
diff --git a/setup.py b/setup.py
index 46ff73d..c74808a 100644
--- a/setup.py
+++ b/setup.py
@@ -270,10 +270,8 @@ exasol = [
facebook = [
'facebook-business>=6.0.2',
]
-flask_oauth = [
- 'Flask-OAuthlib>=0.9.1,<0.9.6', # Flask OAuthLib 0.9.6 requires Flask-Login 0.5.0 - breaks FAB
- 'oauthlib!=2.0.3,!=2.0.4,!=2.0.5,<3.0.0,>=1.1.2',
- 'requests-oauthlib<1.2.0',
+flask_appbuilder_authlib = [
+ 'authlib',
]
google = [
'PyOpenSSL',
@@ -622,8 +620,8 @@ CORE_EXTRAS_REQUIREMENTS: Dict[str, List[str]] = {
'cncf.kubernetes': kubernetes, # also has provider, but it extends the core with the KubernetesExecutor
'dask': dask,
'deprecated_api': deprecated_api,
- 'github_enterprise': flask_oauth,
- 'google_auth': flask_oauth,
+ 'github_enterprise': flask_appbuilder_authlib,
+ 'google_auth': flask_appbuilder_authlib,
'kerberos': kerberos,
'ldap': ldap,
'leveldb': leveldb,
[airflow] 12/22: Improve image building documentation for new users
(#17409)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 8609c82c7cdd3047efee0c55bb214541e3ea293d
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Wed Aug 4 23:07:22 2021 +0200
Improve image building documentation for new users (#17409)
* Improve image building documentation for new users
This PR improves documentation for building images of airflow,
specifically targetting users who do not have big experience with
building the images. It shows examples on how custom image building
can be easily used to upgrade provider packages as well as how
image building can be easily integrated in quick-start using
docker-compose.
(cherry picked from commit 4ee4199f6f0107f39726fc3551d1bf20b8b1283c)
---
docs/apache-airflow-providers/index.rst | 5 +++++
docs/apache-airflow/start/docker-compose.yaml | 6 +++++-
docs/apache-airflow/start/docker.rst | 12 ++++++++++++
docs/docker-stack/build.rst | 13 +++++++++++++
.../docker-examples/extending/add-apt-packages/Dockerfile | 2 +-
.../extending/add-build-essential-extend/Dockerfile | 2 +-
.../extending/{embedding-dags => add-providers}/Dockerfile | 7 +++----
.../docker-examples/extending/add-pypi-packages/Dockerfile | 2 +-
.../docker-examples/extending/embedding-dags/Dockerfile | 2 +-
.../docker-examples/extending/writable-directory/Dockerfile | 2 +-
10 files changed, 43 insertions(+), 10 deletions(-)
diff --git a/docs/apache-airflow-providers/index.rst b/docs/apache-airflow-providers/index.rst
index 7329b7f..71c5132 100644
--- a/docs/apache-airflow-providers/index.rst
+++ b/docs/apache-airflow-providers/index.rst
@@ -21,6 +21,8 @@ Provider packages
.. contents:: :local:
+.. _providers:community-maintained-providers:
+
Community maintained providers
''''''''''''''''''''''''''''''
@@ -31,6 +33,9 @@ Those provider packages are separated per-provider (for example ``amazon``, ``go
etc.). Those packages are available as ``apache-airflow-providers`` packages - separately per each provider
(for example there is an ``apache-airflow-providers-amazon`` or ``apache-airflow-providers-google`` package).
+The full list of community managed providers is available at
+`Providers Index <https://airflow.apache.org/docs/#providers-packages-docs-apache-airflow-providers-index-html>`_.
+
You can install those provider packages separately in order to interface with a given service. For those
providers that have corresponding extras, the provider packages (latest version from PyPI) are installed
automatically when Airflow is installed with the extra.
diff --git a/docs/apache-airflow/start/docker-compose.yaml b/docs/apache-airflow/start/docker-compose.yaml
index 5a301cf..06991e7 100644
--- a/docs/apache-airflow/start/docker-compose.yaml
+++ b/docs/apache-airflow/start/docker-compose.yaml
@@ -44,7 +44,11 @@
version: '3'
x-airflow-common:
&airflow-common
+ # In order to add custom dependencies or upgrade provider packages you can use your extended image.
+ # Comment the image line, place your Dockerfile in the directory where you placed the docker-compose.yaml
+ # and uncomment the "build" line below, Then run `docker-compose build` to build the images.
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:|version|}
+ # build: .
environment:
&airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
@@ -60,7 +64,7 @@ x-airflow-common:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./plugins:/opt/airflow/plugins
- user: "${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}"
+ user: "${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-0}"
depends_on:
redis:
condition: service_healthy
diff --git a/docs/apache-airflow/start/docker.rst b/docs/apache-airflow/start/docker.rst
index 77c9333..747f87c 100644
--- a/docs/apache-airflow/start/docker.rst
+++ b/docs/apache-airflow/start/docker.rst
@@ -68,6 +68,18 @@ If you need install a new Python library or system library, you can :doc:`build
.. _initializing_docker_compose_environment:
+Using custom images
+===================
+
+When you want to run Airflow locally, you might want to use an extended image, containing some additional dependencies - for
+example you might add new python packages, or upgrade airflow providers to a later version. This can be done very easily
+by placing a custom Dockerfile alongside your `docker-compose.yaml`. Then you can use `docker-compose build` command to build your image (you need to
+do it only once). You can also add the `--build` flag to your `docker-compose` commands to rebuild the images
+on-the-fly when you run other `docker-compose` commands.
+
+Examples of how you can extend the image with custom providers, python packages,
+apt packages and more can be found in :doc:`Building the image <docker-stack:build>`.
+
Initializing Environment
========================
diff --git a/docs/docker-stack/build.rst b/docs/docker-stack/build.rst
index 7d89f2f..c469a35 100644
--- a/docs/docker-stack/build.rst
+++ b/docs/docker-stack/build.rst
@@ -250,6 +250,19 @@ You should be aware, about a few things:
Examples of image extending
---------------------------
+Example of upgrading Airflow Provider packages
+..............................................
+
+The :ref:`Airflow Providers <providers:community-maintained-providers>` are released independently of core
+Airflow and sometimes you might want to upgrade specific providers only to fix some problems or
+use features available in that provider version. Here is an example of how you can do it
+
+.. exampleinclude:: docker-examples/extending/add-providers/Dockerfile
+ :language: Dockerfile
+ :start-after: [START Dockerfile]
+ :end-before: [END Dockerfile]
+
+
Example of adding ``apt`` package
.................................
diff --git a/docs/docker-stack/docker-examples/extending/add-apt-packages/Dockerfile b/docs/docker-stack/docker-examples/extending/add-apt-packages/Dockerfile
index 62de197..f11e87a 100644
--- a/docs/docker-stack/docker-examples/extending/add-apt-packages/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/add-apt-packages/Dockerfile
@@ -15,7 +15,7 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow
+FROM apache/airflow:2.1.2
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
diff --git a/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile b/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile
index b34fdc9..47ac51f 100644
--- a/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/add-build-essential-extend/Dockerfile
@@ -15,7 +15,7 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow
+FROM apache/airflow:2.1.2
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
diff --git a/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile b/docs/docker-stack/docker-examples/extending/add-providers/Dockerfile
similarity index 90%
copy from docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile
copy to docs/docker-stack/docker-examples/extending/add-providers/Dockerfile
index c849697..cdf7a42 100644
--- a/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/add-providers/Dockerfile
@@ -15,8 +15,7 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow
-
-COPY --chown=airflow:root test_dag.py /opt/airflow/dags
-
+FROM apache/airflow:2.1.2
+RUN pip install --no-cache-dir apache-airflow-providers-docker==2.1.0
# [END Dockerfile]
+
diff --git a/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile b/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile
index cc2559f..310f84c 100644
--- a/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/add-pypi-packages/Dockerfile
@@ -15,6 +15,6 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow
+FROM apache/airflow:2.1.2
RUN pip install --no-cache-dir lxml
# [END Dockerfile]
diff --git a/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile b/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile
index c849697..48701aa 100644
--- a/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/embedding-dags/Dockerfile
@@ -15,7 +15,7 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow
+FROM apache/airflow:2.1.2
COPY --chown=airflow:root test_dag.py /opt/airflow/dags
diff --git a/docs/docker-stack/docker-examples/extending/writable-directory/Dockerfile b/docs/docker-stack/docker-examples/extending/writable-directory/Dockerfile
index ba07f68..8fbb98d 100644
--- a/docs/docker-stack/docker-examples/extending/writable-directory/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/writable-directory/Dockerfile
@@ -15,7 +15,7 @@
# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
-FROM apache/airflow
+FROM apache/airflow:2.1.2
RUN umask 0002; \
mkdir -p ~/writeable-directory
# [END Dockerfile]
[airflow] 06/22: AIRFLOW-5529 Add Apache Drill provider. (#16884)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 42334b0f9145e70eb23938cad78589f6a97731dc
Author: dzamo <91...@users.noreply.github.com>
AuthorDate: Mon Jul 12 19:59:35 2021 +0200
AIRFLOW-5529 Add Apache Drill provider. (#16884)
(cherry picked from commit 8808b641942e1b81c21db054fd6d36e2031cfab8)
---
CONTRIBUTING.rst | 22 ++---
INSTALL | 24 +++---
airflow/providers/apache/drill/CHANGELOG.rst | 25 ++++++
airflow/providers/apache/drill/__init__.py | 17 ++++
.../apache/drill/example_dags/example_drill_dag.py | 46 +++++++++++
airflow/providers/apache/drill/hooks/__init__.py | 17 ++++
airflow/providers/apache/drill/hooks/drill.py | 89 +++++++++++++++++++++
.../providers/apache/drill/operators/__init__.py | 17 ++++
airflow/providers/apache/drill/operators/drill.py | 71 ++++++++++++++++
airflow/providers/apache/drill/provider.yaml | 49 ++++++++++++
airflow/ui/src/views/Docs.tsx | 1 +
airflow/utils/db.py | 10 +++
.../commits.rst | 23 ++++++
.../connections/drill.rst | 44 ++++++++++
.../index.rst | 50 ++++++++++++
.../operators.rst | 51 ++++++++++++
docs/apache-airflow/extra-packages-ref.rst | 2 +
docs/conf.py | 1 +
docs/integration-logos/apache/drill.png | Bin 0 -> 40173 bytes
docs/spelling_wordlist.txt | 1 +
setup.py | 3 +
tests/providers/apache/drill/__init__.py | 17 ++++
tests/providers/apache/drill/hooks/__init__.py | 17 ++++
tests/providers/apache/drill/hooks/test_drill.py | 84 +++++++++++++++++++
tests/providers/apache/drill/operators/__init__.py | 17 ++++
.../providers/apache/drill/operators/test_drill.py | 63 +++++++++++++++
26 files changed, 738 insertions(+), 23 deletions(-)
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index 98cdf93..be807f4 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -573,17 +573,17 @@ This is the full list of those extras:
.. START EXTRAS HERE
-airbyte, all, all_dbs, amazon, apache.atlas, apache.beam, apache.cassandra, apache.druid,
-apache.hdfs, apache.hive, apache.kylin, apache.livy, apache.pig, apache.pinot, apache.spark,
-apache.sqoop, apache.webhdfs, asana, async, atlas, aws, azure, cassandra, celery, cgroups, cloudant,
-cncf.kubernetes, crypto, dask, databricks, datadog, deprecated_api, devel, devel_all, devel_ci,
-devel_hadoop, dingding, discord, doc, docker, druid, elasticsearch, exasol, facebook, ftp, gcp,
-gcp_api, github_enterprise, google, google_auth, grpc, hashicorp, hdfs, hive, http, imap, jdbc,
-jenkins, jira, kerberos, kubernetes, ldap, leveldb, microsoft.azure, microsoft.mssql,
-microsoft.winrm, mongo, mssql, mysql, neo4j, odbc, openfaas, opsgenie, oracle, pagerduty, papermill,
-password, pinot, plexus, postgres, presto, qds, qubole, rabbitmq, redis, s3, salesforce, samba,
-segment, sendgrid, sentry, sftp, singularity, slack, snowflake, spark, sqlite, ssh, statsd, tableau,
-telegram, trino, vertica, virtualenv, webhdfs, winrm, yandex, zendesk
+airbyte, all, all_dbs, amazon, apache.atlas, apache.beam, apache.cassandra, apache.drill,
+apache.druid, apache.hdfs, apache.hive, apache.kylin, apache.livy, apache.pig, apache.pinot,
+apache.spark, apache.sqoop, apache.webhdfs, asana, async, atlas, aws, azure, cassandra, celery,
+cgroups, cloudant, cncf.kubernetes, crypto, dask, databricks, datadog, deprecated_api, devel,
+devel_all, devel_ci, devel_hadoop, dingding, discord, doc, docker, druid, elasticsearch, exasol,
+facebook, ftp, gcp, gcp_api, github_enterprise, google, google_auth, grpc, hashicorp, hdfs, hive,
+http, imap, jdbc, jenkins, jira, kerberos, kubernetes, ldap, leveldb, microsoft.azure,
+microsoft.mssql, microsoft.winrm, mongo, mssql, mysql, neo4j, odbc, openfaas, opsgenie, oracle,
+pagerduty, papermill, password, pinot, plexus, postgres, presto, qds, qubole, rabbitmq, redis, s3,
+salesforce, samba, segment, sendgrid, sentry, sftp, singularity, slack, snowflake, spark, sqlite,
+ssh, statsd, tableau, telegram, trino, vertica, virtualenv, webhdfs, winrm, yandex, zendesk
.. END EXTRAS HERE
diff --git a/INSTALL b/INSTALL
index 111b51f..554af5c 100644
--- a/INSTALL
+++ b/INSTALL
@@ -1,6 +1,6 @@
# INSTALL / BUILD instructions for Apache Airflow
-This ia a generic installation method that requires a number of dependencies to be installed.
+This is a generic installation method that requires a number of dependencies to be installed.
Depending on your system you might need different prerequisites, but the following
systems/prerequisites are known to work:
@@ -89,17 +89,17 @@ The list of available extras:
# START EXTRAS HERE
-airbyte, all, all_dbs, amazon, apache.atlas, apache.beam, apache.cassandra, apache.druid,
-apache.hdfs, apache.hive, apache.kylin, apache.livy, apache.pig, apache.pinot, apache.spark,
-apache.sqoop, apache.webhdfs, asana, async, atlas, aws, azure, cassandra, celery, cgroups, cloudant,
-cncf.kubernetes, crypto, dask, databricks, datadog, deprecated_api, devel, devel_all, devel_ci,
-devel_hadoop, dingding, discord, doc, docker, druid, elasticsearch, exasol, facebook, ftp, gcp,
-gcp_api, github_enterprise, google, google_auth, grpc, hashicorp, hdfs, hive, http, imap, jdbc,
-jenkins, jira, kerberos, kubernetes, ldap, leveldb, microsoft.azure, microsoft.mssql,
-microsoft.winrm, mongo, mssql, mysql, neo4j, odbc, openfaas, opsgenie, oracle, pagerduty, papermill,
-password, pinot, plexus, postgres, presto, qds, qubole, rabbitmq, redis, s3, salesforce, samba,
-segment, sendgrid, sentry, sftp, singularity, slack, snowflake, spark, sqlite, ssh, statsd, tableau,
-telegram, trino, vertica, virtualenv, webhdfs, winrm, yandex, zendesk
+airbyte, all, all_dbs, amazon, apache.atlas, apache.beam, apache.cassandra, apache.drill,
+apache.druid, apache.hdfs, apache.hive, apache.kylin, apache.livy, apache.pig, apache.pinot,
+apache.spark, apache.sqoop, apache.webhdfs, asana, async, atlas, aws, azure, cassandra, celery,
+cgroups, cloudant, cncf.kubernetes, crypto, dask, databricks, datadog, deprecated_api, devel,
+devel_all, devel_ci, devel_hadoop, dingding, discord, doc, docker, druid, elasticsearch, exasol,
+facebook, ftp, gcp, gcp_api, github_enterprise, google, google_auth, grpc, hashicorp, hdfs, hive,
+http, imap, jdbc, jenkins, jira, kerberos, kubernetes, ldap, leveldb, microsoft.azure,
+microsoft.mssql, microsoft.winrm, mongo, mssql, mysql, neo4j, odbc, openfaas, opsgenie, oracle,
+pagerduty, papermill, password, pinot, plexus, postgres, presto, qds, qubole, rabbitmq, redis, s3,
+salesforce, samba, segment, sendgrid, sentry, sftp, singularity, slack, snowflake, spark, sqlite,
+ssh, statsd, tableau, telegram, trino, vertica, virtualenv, webhdfs, winrm, yandex, zendesk
# END EXTRAS HERE
diff --git a/airflow/providers/apache/drill/CHANGELOG.rst b/airflow/providers/apache/drill/CHANGELOG.rst
new file mode 100644
index 0000000..cef7dda
--- /dev/null
+++ b/airflow/providers/apache/drill/CHANGELOG.rst
@@ -0,0 +1,25 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+Changelog
+---------
+
+1.0.0
+.....
+
+Initial version of the provider.
diff --git a/airflow/providers/apache/drill/__init__.py b/airflow/providers/apache/drill/__init__.py
new file mode 100644
index 0000000..217e5db
--- /dev/null
+++ b/airflow/providers/apache/drill/__init__.py
@@ -0,0 +1,17 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/airflow/providers/apache/drill/example_dags/example_drill_dag.py b/airflow/providers/apache/drill/example_dags/example_drill_dag.py
new file mode 100644
index 0000000..60a35ee
--- /dev/null
+++ b/airflow/providers/apache/drill/example_dags/example_drill_dag.py
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG to submit Apache Spark applications using
+`SparkSubmitOperator`, `SparkJDBCOperator` and `SparkSqlOperator`.
+"""
+from airflow.models import DAG
+from airflow.providers.apache.drill.operators.drill import DrillOperator
+from airflow.utils.dates import days_ago
+
+args = {
+ 'owner': 'Airflow',
+}
+
+with DAG(
+ dag_id='example_drill_dag',
+ default_args=args,
+ schedule_interval=None,
+ start_date=days_ago(2),
+ tags=['example'],
+) as dag:
+ # [START howto_operator_drill]
+ sql_task = DrillOperator(
+ task_id='json_to_parquet_table',
+ sql='''
+ drop table if exists dfs.tmp.employee;
+ create table dfs.tmp.employee as select * from cp.`employee.json`;
+ ''',
+ )
+ # [END howto_operator_drill]
diff --git a/airflow/providers/apache/drill/hooks/__init__.py b/airflow/providers/apache/drill/hooks/__init__.py
new file mode 100644
index 0000000..217e5db
--- /dev/null
+++ b/airflow/providers/apache/drill/hooks/__init__.py
@@ -0,0 +1,17 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/airflow/providers/apache/drill/hooks/drill.py b/airflow/providers/apache/drill/hooks/drill.py
new file mode 100644
index 0000000..470be8c
--- /dev/null
+++ b/airflow/providers/apache/drill/hooks/drill.py
@@ -0,0 +1,89 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Iterable, Optional, Tuple
+
+from sqlalchemy import create_engine
+from sqlalchemy.engine import Connection
+
+from airflow.hooks.dbapi import DbApiHook
+
+
+class DrillHook(DbApiHook):
+ """
+ Interact with Apache Drill via sqlalchemy-drill.
+
+ You can specify the SQLAlchemy dialect and driver that sqlalchemy-drill
+ will employ to communicate with Drill in the extras field of your
+ connection, e.g. ``{"dialect_driver": "drill+sadrill"}`` for communication
+ over Drill's REST API. See the sqlalchemy-drill documentation for
+ descriptions of the supported dialects and drivers.
+
+ You can specify the default storage_plugin for the sqlalchemy-drill
+ connection using the extras field e.g. ``{"storage_plugin": "dfs"}``.
+ """
+
+ conn_name_attr = 'drill_conn_id'
+ default_conn_name = 'drill_default'
+ conn_type = 'drill'
+ hook_name = 'Drill'
+ supports_autocommit = False
+
+ def get_conn(self) -> Connection:
+ """Establish a connection to Drillbit."""
+ conn_md = self.get_connection(getattr(self, self.conn_name_attr))
+ creds = f'{conn_md.login}:{conn_md.password}@' if conn_md.login else ''
+ engine = create_engine(
+ f'{conn_md.extra_dejson.get("dialect_driver", "drill+sadrill")}://{creds}'
+ f'{conn_md.host}:{conn_md.port}/'
+ f'{conn_md.extra_dejson.get("storage_plugin", "dfs")}'
+ )
+
+ self.log.info(
+ 'Connected to the Drillbit at %s:%s as user %s', conn_md.host, conn_md.port, conn_md.login
+ )
+ return engine.raw_connection()
+
+ def get_uri(self) -> str:
+ """
+ Returns the connection URI
+
+ e.g: ``drill://localhost:8047/dfs``
+ """
+ conn_md = self.get_connection(getattr(self, self.conn_name_attr))
+ host = conn_md.host
+ if conn_md.port is not None:
+ host += f':{conn_md.port}'
+ conn_type = 'drill' if not conn_md.conn_type else conn_md.conn_type
+ dialect_driver = conn_md.extra_dejson.get('dialect_driver', 'drill+sadrill')
+ storage_plugin = conn_md.extra_dejson.get('storage_plugin', 'dfs')
+ return f'{conn_type}://{host}/{storage_plugin}' f'?dialect_driver={dialect_driver}'
+
+ def set_autocommit(self, conn: Connection, autocommit: bool) -> NotImplemented:
+ raise NotImplementedError("There are no transactions in Drill.")
+
+ def insert_rows(
+ self,
+ table: str,
+ rows: Iterable[Tuple[str]],
+ target_fields: Optional[Iterable[str]] = None,
+ commit_every: int = 1000,
+ replace: bool = False,
+ **kwargs: Any,
+ ) -> NotImplemented:
+ raise NotImplementedError("There is no INSERT statement in Drill.")
diff --git a/airflow/providers/apache/drill/operators/__init__.py b/airflow/providers/apache/drill/operators/__init__.py
new file mode 100644
index 0000000..217e5db
--- /dev/null
+++ b/airflow/providers/apache/drill/operators/__init__.py
@@ -0,0 +1,17 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/airflow/providers/apache/drill/operators/drill.py b/airflow/providers/apache/drill/operators/drill.py
new file mode 100644
index 0000000..459c623
--- /dev/null
+++ b/airflow/providers/apache/drill/operators/drill.py
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Iterable, Mapping, Optional, Union
+
+import sqlparse
+
+from airflow.models import BaseOperator
+from airflow.providers.apache.drill.hooks.drill import DrillHook
+from airflow.utils.decorators import apply_defaults
+
+
+class DrillOperator(BaseOperator):
+ """
+ Executes the provided SQL in the identified Drill environment.
+
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:DrillOperator`
+
+ :param sql: the SQL code to be executed. (templated)
+ :type sql: Can receive a str representing a sql statement,
+ a list of str (sql statements), or a reference to a template file.
+ Template references are recognized by str ending in '.sql'
+ :param drill_conn_id: id of the connection config for the target Drill
+ environment
+ :type drill_conn_id: str
+ :param parameters: (optional) the parameters to render the SQL query with.
+ :type parameters: dict or iterable
+ """
+
+ template_fields = ('sql',)
+ template_fields_renderers = {'sql': 'sql'}
+ template_ext = ('.sql',)
+ ui_color = '#ededed'
+
+ @apply_defaults
+ def __init__(
+ self,
+ *,
+ sql: str,
+ drill_conn_id: str = 'drill_default',
+ parameters: Optional[Union[Mapping, Iterable]] = None,
+ **kwargs,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.sql = sql
+ self.drill_conn_id = drill_conn_id
+ self.parameters = parameters
+ self.hook = None
+
+ def execute(self, context):
+ self.log.info('Executing: %s on %s', self.sql, self.drill_conn_id)
+ self.hook = DrillHook(drill_conn_id=self.drill_conn_id)
+ sql = sqlparse.split(sqlparse.format(self.sql, strip_comments=True))
+ no_term_sql = [s[:-1] for s in sql if s[-1] == ';']
+ self.hook.run(no_term_sql, parameters=self.parameters)
diff --git a/airflow/providers/apache/drill/provider.yaml b/airflow/providers/apache/drill/provider.yaml
new file mode 100644
index 0000000..88021e6
--- /dev/null
+++ b/airflow/providers/apache/drill/provider.yaml
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+---
+package-name: apache-airflow-providers-apache-drill
+name: Apache Drill
+description: |
+ `Apache Drill <https://drill.apache.org/>`__.
+
+versions:
+ - 1.0.0
+
+additional-dependencies:
+ - apache-airflow>=2.1.0
+
+integrations:
+ - integration-name: Apache Drill
+ external-doc-url: https://drill.apache.org/
+ how-to-guide:
+ - /docs/apache-airflow-providers-apache-drill/operators.rst
+ logo: /integration-logos/apache/drill.png
+ tags: [apache]
+
+operators:
+ - integration-name: Apache Drill
+ python-modules:
+ - airflow.providers.apache.drill.operators.drill
+
+hooks:
+ - integration-name: Apache Drill
+ python-modules:
+ - airflow.providers.apache.drill.hooks.drill
+
+hook-class-names:
+ - airflow.providers.apache.drill.hooks.drill.DrillHook
diff --git a/airflow/ui/src/views/Docs.tsx b/airflow/ui/src/views/Docs.tsx
index 754b803..ea42ca6 100644
--- a/airflow/ui/src/views/Docs.tsx
+++ b/airflow/ui/src/views/Docs.tsx
@@ -41,6 +41,7 @@ const Docs: React.FC = () => {
{ path: 'amazon', name: 'Amazon' },
{ path: 'apache-beam', name: 'Apache Beam' },
{ path: 'apache-cassandra', name: 'Apache Cassandra' },
+ { path: 'apache-drill', name: 'Apache Drill' },
{ path: 'apache-druid', name: 'Apache Druid' },
{ path: 'apache-hdfs', name: 'Apache HDFS' },
{ path: 'apache-hive', name: 'Apache Hive' },
diff --git a/airflow/utils/db.py b/airflow/utils/db.py
index ae8dc0e..69e2e00 100644
--- a/airflow/utils/db.py
+++ b/airflow/utils/db.py
@@ -168,6 +168,16 @@ def create_default_connections(session=None):
)
merge_conn(
Connection(
+ conn_id="drill_default",
+ conn_type="drill",
+ host="localhost",
+ port=8047,
+ extra='{"dialect_driver": "drill+sadrill", "storage_plugin": "dfs"}',
+ ),
+ session,
+ )
+ merge_conn(
+ Connection(
conn_id="druid_broker_default",
conn_type="druid",
host="druid-broker",
diff --git a/docs/apache-airflow-providers-apache-drill/commits.rst b/docs/apache-airflow-providers-apache-drill/commits.rst
new file mode 100644
index 0000000..deb31b3
--- /dev/null
+++ b/docs/apache-airflow-providers-apache-drill/commits.rst
@@ -0,0 +1,23 @@
+
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+Package apache-airflow-providers-apache-drill
+------------------------------------------------------
+
+`Apache Drill <https://drill.apache.org/>`__.
diff --git a/docs/apache-airflow-providers-apache-drill/connections/drill.rst b/docs/apache-airflow-providers-apache-drill/connections/drill.rst
new file mode 100644
index 0000000..05e00a2
--- /dev/null
+++ b/docs/apache-airflow-providers-apache-drill/connections/drill.rst
@@ -0,0 +1,44 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+.. _howto/connection:drill:
+
+Apache Drill Connection
+=======================
+
+The Apache Drill connection type configures a connection to Apache Drill via the sqlalchemy-drill Python package.
+
+Default Connection IDs
+----------------------
+
+Drill hooks and operators use ``drill_default`` by default.
+
+Configuring the Connection
+--------------------------
+Host (required)
+ The host of the Drillbit to connect to (HTTP, JDBC) or the DSN of the Drill ODBC connection.
+
+Port (optional)
+ The port of the Drillbit to connect to.
+
+Extra (optional)
+ A JSON dictionary specifying the extra parameters that can be used in sqlalchemy-drill connection.
+
+ * ``dialect_driver`` - The dialect and driver as understood by sqlalchemy-drill. Defaults to ``drill_sadrill`` (HTTP).
+ * ``storage_plugin`` - The default Drill storage plugin for this connection. Defaults to ``dfs``.
diff --git a/docs/apache-airflow-providers-apache-drill/index.rst b/docs/apache-airflow-providers-apache-drill/index.rst
new file mode 100644
index 0000000..9ef0f2e
--- /dev/null
+++ b/docs/apache-airflow-providers-apache-drill/index.rst
@@ -0,0 +1,50 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+``apache-airflow-providers-apache-drill``
+=========================================
+
+Content
+-------
+
+.. toctree::
+ :maxdepth: 1
+ :caption: Guides
+
+ Connection types <connections/drill>
+ Operators <operators>
+
+.. toctree::
+ :maxdepth: 1
+ :caption: References
+
+ Python API <_api/airflow/providers/apache/drill/index>
+
+.. toctree::
+ :maxdepth: 1
+ :caption: Resources
+
+ Example DAGs <https://github.com/apache/airflow/tree/main/airflow/providers/apache/drill/example_dags>
+ PyPI Repository <https://pypi.org/project/apache-airflow-providers-apache-drill/>
+
+.. THE REMINDER OF THE FILE IS AUTOMATICALLY GENERATED. IT WILL BE OVERWRITTEN AT RELEASE TIME!
+
+.. toctree::
+ :maxdepth: 1
+ :caption: Commits
+
+ Detailed list of commits <commits>
diff --git a/docs/apache-airflow-providers-apache-drill/operators.rst b/docs/apache-airflow-providers-apache-drill/operators.rst
new file mode 100644
index 0000000..f2d5cf5
--- /dev/null
+++ b/docs/apache-airflow-providers-apache-drill/operators.rst
@@ -0,0 +1,51 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+Apache Drill Operators
+======================
+
+.. contents::
+ :depth: 1
+ :local:
+
+Prerequisite
+------------
+
+To use ``DrillOperator``, you must configure a :doc:`Drill Connection <connections/drill>`.
+
+
+.. _howto/operator:DrillOperator:
+
+DrillOperator
+-------------
+
+Executes one or more SQL queries on an Apache Drill server. The ``sql`` parameter can be templated and be an external ``.sql`` file.
+
+Using the operator
+""""""""""""""""""
+
+.. exampleinclude:: /../../airflow/providers/apache/drill/example_dags/example_drill_dag.py
+ :language: python
+ :dedent: 4
+ :start-after: [START howto_operator_drill]
+ :end-before: [END howto_operator_drill]
+
+Reference
+"""""""""
+
+For further information, see `the Drill documentation on querying data <http://apache.github.io/drill/docs/query-data/>`_.
diff --git a/docs/apache-airflow/extra-packages-ref.rst b/docs/apache-airflow/extra-packages-ref.rst
index b1dff07..2d4769e 100644
--- a/docs/apache-airflow/extra-packages-ref.rst
+++ b/docs/apache-airflow/extra-packages-ref.rst
@@ -116,6 +116,8 @@ custom bash/python providers).
+---------------------+-----------------------------------------------------+------------------------------------------------+
| apache.cassandra | ``pip install 'apache-airflow[apache.cassandra]'`` | Cassandra related operators & hooks |
+---------------------+-----------------------------------------------------+------------------------------------------------+
+| apache.drill | ``pip install 'apache-airflow[apache.drill]'`` | Drill related operators & hooks |
++---------------------+-----------------------------------------------------+------------------------------------------------+
| apache.druid | ``pip install 'apache-airflow[apache.druid]'`` | Druid related operators & hooks |
+---------------------+-----------------------------------------------------+------------------------------------------------+
| apache.hdfs | ``pip install 'apache-airflow[apache.hdfs]'`` | HDFS hooks and operators |
diff --git a/docs/conf.py b/docs/conf.py
index 3046303..da3b8e2 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -512,6 +512,7 @@ autodoc_mock_imports = [
'slack_sdk',
'smbclient',
'snowflake',
+ 'sqlalchemy-drill',
'sshtunnel',
'telegram',
'tenacity',
diff --git a/docs/integration-logos/apache/drill.png b/docs/integration-logos/apache/drill.png
new file mode 100644
index 0000000..9f76b61
Binary files /dev/null and b/docs/integration-logos/apache/drill.png differ
diff --git a/docs/spelling_wordlist.txt b/docs/spelling_wordlist.txt
index c476512..7631c6c 100644
--- a/docs/spelling_wordlist.txt
+++ b/docs/spelling_wordlist.txt
@@ -120,6 +120,7 @@ Docstring
Docstrings
Dont
Driesprong
+Drillbit
Drivy
Dsn
Dynamodb
diff --git a/setup.py b/setup.py
index 9e8c28d..9dde824 100644
--- a/setup.py
+++ b/setup.py
@@ -255,6 +255,7 @@ doc = [
docker = [
'docker',
]
+drill = ['sqlalchemy-drill>=1.1.0', 'sqlparse>=0.4.1']
druid = [
'pydruid>=0.4.1',
]
@@ -534,6 +535,7 @@ PROVIDERS_REQUIREMENTS: Dict[str, List[str]] = {
'amazon': amazon,
'apache.beam': apache_beam,
'apache.cassandra': cassandra,
+ 'apache.drill': drill,
'apache.druid': druid,
'apache.hdfs': hdfs,
'apache.hive': hive,
@@ -724,6 +726,7 @@ ALL_PROVIDERS = list(PROVIDERS_REQUIREMENTS.keys())
ALL_DB_PROVIDERS = [
'apache.cassandra',
+ 'apache.drill',
'apache.druid',
'apache.hdfs',
'apache.hive',
diff --git a/tests/providers/apache/drill/__init__.py b/tests/providers/apache/drill/__init__.py
new file mode 100644
index 0000000..217e5db
--- /dev/null
+++ b/tests/providers/apache/drill/__init__.py
@@ -0,0 +1,17 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/tests/providers/apache/drill/hooks/__init__.py b/tests/providers/apache/drill/hooks/__init__.py
new file mode 100644
index 0000000..217e5db
--- /dev/null
+++ b/tests/providers/apache/drill/hooks/__init__.py
@@ -0,0 +1,17 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/tests/providers/apache/drill/hooks/test_drill.py b/tests/providers/apache/drill/hooks/test_drill.py
new file mode 100644
index 0000000..97ed71f
--- /dev/null
+++ b/tests/providers/apache/drill/hooks/test_drill.py
@@ -0,0 +1,84 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+import unittest
+from unittest.mock import MagicMock
+
+from airflow.providers.apache.drill.hooks.drill import DrillHook
+
+
+class TestDrillHook(unittest.TestCase):
+ def setUp(self):
+ self.cur = MagicMock(rowcount=0)
+ self.conn = conn = MagicMock()
+ self.conn.login = 'drill_user'
+ self.conn.password = 'secret'
+ self.conn.host = 'host'
+ self.conn.port = '8047'
+ self.conn.conn_type = 'drill'
+ self.conn.extra_dejson = {'dialect_driver': 'drill+sadrill', 'storage_plugin': 'dfs'}
+ self.conn.cursor.return_value = self.cur
+
+ class TestDrillHook(DrillHook):
+ def get_conn(self):
+ return conn
+
+ def get_connection(self, conn_id):
+ return conn
+
+ self.db_hook = TestDrillHook
+
+ def test_get_uri(self):
+ db_hook = self.db_hook()
+ assert 'drill://host:8047/dfs?dialect_driver=drill+sadrill' == db_hook.get_uri()
+
+ def test_get_first_record(self):
+ statement = 'SQL'
+ result_sets = [('row1',), ('row2',)]
+ self.cur.fetchone.return_value = result_sets[0]
+
+ assert result_sets[0] == self.db_hook().get_first(statement)
+ assert self.conn.close.call_count == 1
+ assert self.cur.close.call_count == 1
+ self.cur.execute.assert_called_once_with(statement)
+
+ def test_get_records(self):
+ statement = 'SQL'
+ result_sets = [('row1',), ('row2',)]
+ self.cur.fetchall.return_value = result_sets
+
+ assert result_sets == self.db_hook().get_records(statement)
+ assert self.conn.close.call_count == 1
+ assert self.cur.close.call_count == 1
+ self.cur.execute.assert_called_once_with(statement)
+
+ def test_get_pandas_df(self):
+ statement = 'SQL'
+ column = 'col'
+ result_sets = [('row1',), ('row2',)]
+ self.cur.description = [(column,)]
+ self.cur.fetchall.return_value = result_sets
+ df = self.db_hook().get_pandas_df(statement)
+
+ assert column == df.columns[0]
+ for i in range(len(result_sets)): # pylint: disable=consider-using-enumerate
+ assert result_sets[i][0] == df.values.tolist()[i][0]
+ assert self.conn.close.call_count == 1
+ assert self.cur.close.call_count == 1
+ self.cur.execute.assert_called_once_with(statement)
diff --git a/tests/providers/apache/drill/operators/__init__.py b/tests/providers/apache/drill/operators/__init__.py
new file mode 100644
index 0000000..217e5db
--- /dev/null
+++ b/tests/providers/apache/drill/operators/__init__.py
@@ -0,0 +1,17 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/tests/providers/apache/drill/operators/test_drill.py b/tests/providers/apache/drill/operators/test_drill.py
new file mode 100644
index 0000000..3572d85
--- /dev/null
+++ b/tests/providers/apache/drill/operators/test_drill.py
@@ -0,0 +1,63 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+
+import pytest
+
+from airflow.models.dag import DAG
+from airflow.providers.apache.drill.operators.drill import DrillOperator
+from airflow.utils import timezone
+
+DEFAULT_DATE = timezone.datetime(2015, 1, 1)
+DEFAULT_DATE_ISO = DEFAULT_DATE.isoformat()
+DEFAULT_DATE_DS = DEFAULT_DATE_ISO[:10]
+TEST_DAG_ID = 'unit_test_dag'
+
+
+@pytest.mark.backend("drill")
+class TestDrillOperator(unittest.TestCase):
+ def setUp(self):
+ args = {'owner': 'airflow', 'start_date': DEFAULT_DATE}
+ dag = DAG(TEST_DAG_ID, default_args=args)
+ self.dag = dag
+
+ def tearDown(self):
+ tables_to_drop = ['dfs.tmp.test_airflow']
+ from airflow.providers.apache.drill.hooks.drill import DrillHook
+
+ with DrillHook().get_conn() as conn:
+ with conn.cursor() as cur:
+ for table in tables_to_drop:
+ cur.execute(f"DROP TABLE IF EXISTS {table}")
+
+ def test_drill_operator_single(self):
+ sql = """
+ create table dfs.tmp.test_airflow as
+ select * from cp.`employee.json` limit 10
+ """
+ op = DrillOperator(task_id='drill_operator_test_single', sql=sql, dag=self.dag)
+ op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
+
+ def test_drill_operator_multi(self):
+ sql = [
+ "create table dfs.tmp.test_airflow as" "select * from cp.`employee.json` limit 10",
+ "select sum(employee_id), any_value(full_name)" "from dfs.tmp.test_airflow",
+ ]
+ op = DrillOperator(task_id='drill_operator_test_multi', sql=sql, dag=self.dag)
+ op.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
[airflow] 02/22: Remove SQLAlchemy <1.4 constraint (#16630)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 68977b9ed93e3522221f55ea93e1848be13ccef0
Author: Tzu-ping Chung <tp...@astronomer.io>
AuthorDate: Thu Jun 24 21:10:15 2021 +0800
Remove SQLAlchemy <1.4 constraint (#16630)
This was added due to flask-sqlalchemy and sqlalchemy-utils not declaring
the upper bounds. They have since released sqlalchemy 1.4-compatible
versions, so we can remove that hack.
Note that this does *not* actually make us run on sqlalchemy 1.4 since
flask-appbuilder still has a <1.4 pin. But that's for flask-appbuilder
to worry about -- code in Airflow is compatible, so we can remove the
constraint now, and get sqlalchemy 1.4 as soon as flask-appbuilder
allows us to.
(cherry picked from commit d181604739c048c6969d8997dbaf8b159607904b)
---
setup.cfg | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/setup.cfg b/setup.cfg
index 8b03296..0e4868f 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -150,8 +150,7 @@ install_requires =
pyyaml>=5.1
rich>=9.2.0
setproctitle>=1.1.8, <2
- # SQLAlchemy 1.4 breaks sqlalchemy-utils https://github.com/kvesteri/sqlalchemy-utils/issues/505
- sqlalchemy>=1.3.18, <1.4
+ sqlalchemy>=1.3.18
sqlalchemy_jsonfield~=1.0
# Required by vendored-in connexion
swagger-ui-bundle>=0.0.2
[airflow] 07/22: Update alias for field_mask in Google Memmcache
(#16975)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 37c935d601ff6c3f501fcfab3bf1b57665fddbc1
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Tue Jul 13 20:54:38 2021 +0200
Update alias for field_mask in Google Memmcache (#16975)
The July 12 2021 release of google-memcache library removed
field_mask alias from the library which broke our typechecking
and made google provider unimportable. This PR fixes the import
to use the actual import.
(cherry picked from commit a3f5c93806258b5ad396a638ba0169eca7f9d065)
---
.../providers/google/cloud/hooks/cloud_memorystore.py | 16 ++++++++--------
setup.py | 4 +++-
2 files changed, 11 insertions(+), 9 deletions(-)
diff --git a/airflow/providers/google/cloud/hooks/cloud_memorystore.py b/airflow/providers/google/cloud/hooks/cloud_memorystore.py
index caf1cd6..8f4165b 100644
--- a/airflow/providers/google/cloud/hooks/cloud_memorystore.py
+++ b/airflow/providers/google/cloud/hooks/cloud_memorystore.py
@@ -487,8 +487,8 @@ class CloudMemorystoreHook(GoogleBaseHook):
- ``redisConfig``
If a dict is provided, it must be of the same form as the protobuf message
- :class:`~google.cloud.redis_v1.types.FieldMask`
- :type update_mask: Union[Dict, google.cloud.redis_v1.types.FieldMask]
+ :class:`~google.protobuf.field_mask_pb2.FieldMask`
+ :type update_mask: Union[Dict, google.protobuf.field_mask_pb2.FieldMask]
:param instance: Required. Update description. Only fields specified in ``update_mask`` are updated.
If a dict is provided, it must be of the same form as the protobuf message
@@ -871,7 +871,7 @@ class CloudMemorystoreMemcachedHook(GoogleBaseHook):
@GoogleBaseHook.fallback_to_default_project_id
def update_instance(
self,
- update_mask: Union[Dict, cloud_memcache.field_mask.FieldMask],
+ update_mask: Union[Dict, FieldMask],
instance: Union[Dict, cloud_memcache.Instance],
project_id: str,
location: Optional[str] = None,
@@ -889,9 +889,9 @@ class CloudMemorystoreMemcachedHook(GoogleBaseHook):
- ``displayName``
If a dict is provided, it must be of the same form as the protobuf message
- :class:`~google.cloud.memcache_v1beta2.types.cloud_memcache.field_mask.FieldMask`
+ :class:`~google.protobuf.field_mask_pb2.FieldMask`)
:type update_mask:
- Union[Dict, google.cloud.memcache_v1beta2.types.cloud_memcache.field_mask.FieldMask]
+ Union[Dict, google.protobuf.field_mask_pb2.FieldMask]
:param instance: Required. Update description. Only fields specified in ``update_mask`` are updated.
If a dict is provided, it must be of the same form as the protobuf message
@@ -935,7 +935,7 @@ class CloudMemorystoreMemcachedHook(GoogleBaseHook):
@GoogleBaseHook.fallback_to_default_project_id
def update_parameters(
self,
- update_mask: Union[Dict, cloud_memcache.field_mask.FieldMask],
+ update_mask: Union[Dict, FieldMask],
parameters: Union[Dict, cloud_memcache.MemcacheParameters],
project_id: str,
location: str,
@@ -951,9 +951,9 @@ class CloudMemorystoreMemcachedHook(GoogleBaseHook):
:param update_mask: Required. Mask of fields to update.
If a dict is provided, it must be of the same form as the protobuf message
- :class:`~google.cloud.memcache_v1beta2.types.cloud_memcache.field_mask.FieldMask`
+ :class:`~google.protobuf.field_mask_pb2.FieldMask`
:type update_mask:
- Union[Dict, google.cloud.memcache_v1beta2.types.cloud_memcache.field_mask.FieldMask]
+ Union[Dict, google.protobuf.field_mask_pb2.FieldMask]
:param parameters: The parameters to apply to the instance.
If a dict is provided, it must be of the same form as the protobuf message
:class:`~google.cloud.memcache_v1beta2.types.cloud_memcache.MemcacheParameters`
diff --git a/setup.py b/setup.py
index 9dde824..46ff73d 100644
--- a/setup.py
+++ b/setup.py
@@ -292,7 +292,9 @@ google = [
'google-cloud-kms>=2.0.0,<3.0.0',
'google-cloud-language>=1.1.1,<2.0.0',
'google-cloud-logging>=2.1.1,<3.0.0',
- 'google-cloud-memcache>=0.2.0',
+ # 1.1.0 removed field_mask and broke import for released providers
+ # We can remove the <1.1.0 limitation after we release new Google Provider
+ 'google-cloud-memcache>=0.2.0,<1.1.0',
'google-cloud-monitoring>=2.0.0,<3.0.0',
'google-cloud-os-login>=2.0.0,<3.0.0',
'google-cloud-pubsub>=2.0.0,<3.0.0',
[airflow] 21/22: Install providers from sources in prod image only
on main (#17458)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 0948366ba47bc9bb529fb1c5c33f1a451426a923
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sat Aug 7 22:47:48 2021 +0200
Install providers from sources in prod image only on main (#17458)
When we build production image during test in main, we install
providers from current sources, to make sure that all the tests
including Helm Chart/Kubernetes tests are using latest sources for
providers.
However, when we build the prod image during v* branches, we
want to build the production image using latest released providers
instead, because this will be the way it will be built shortly when
we release it. We do not run providers test not helm chart tests in
this branch so it is more important to build the image in the way it
will be built for releases - we run verification then and install
dependencies in the very same way it will be done during release.
(cherry picked from commit 5790f72c663596711a344e4c0e5f1036159d7190)
---
.github/workflows/build-images.yml | 3 +++
1 file changed, 3 insertions(+)
diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml
index c2a9054..729d13f 100644
--- a/.github/workflows/build-images.yml
+++ b/.github/workflows/build-images.yml
@@ -74,6 +74,7 @@ jobs:
image-build: ${{ steps.dynamic-outputs.outputs.image-build }}
cacheDirective: ${{ steps.dynamic-outputs.outputs.cacheDirective }}
targetBranch: ${{ steps.dynamic-outputs.outputs.targetBranch }}
+ defaultBranch: ${{ steps.selective-checks.outputs.default-branch }}
steps:
# Retrieve it to be able to determine which files has changed in the incoming commit of the PR
# we checkout the target commit and it's parent to be able to compare them
@@ -231,6 +232,8 @@ jobs:
CHECK_IF_BASE_PYTHON_IMAGE_UPDATED: >
${{ github.event_name == 'pull_request_target' && 'false' || 'true' }}
VERSION_SUFFIX_FOR_PYPI: ".dev0"
+ INSTALL_PROVIDERS_FROM_SOURCES: >
+ ${{ needs.build-info.outputs.defaultBranch == 'main' && 'true' || 'false' }}
steps:
- name: Set envs
# Set pull image tag for CI image build, in order to pull the image pushed
[airflow] 14/22: Add timeout when asking whether to rebuild image
(#17412)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 7371f986b0eedec677c76dc7b6db28a996b9c525
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Wed Aug 4 15:44:06 2021 +0200
Add timeout when asking whether to rebuild image (#17412)
This PR adds timeout to answer the question, whether to rebuild
image when `breeze` is invoked or when pre-commit is run.
This reflects the typical use cases where rebuild is mostly not
needed, only in case of some tests which require new dependencies
to be included.
User has still 4 seconds to answer Y and have the images rebuilt
and just the presence of the question will be enough to get the
user trigger it from time to time.
(cherry picked from commit 2938acd817561c79674ca333b83ee1972248df98)
---
confirm | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/confirm b/confirm
index e796737..42316da 100755
--- a/confirm
+++ b/confirm
@@ -15,7 +15,7 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-set -euo pipefail
+set -uo pipefail
if [[ -n "${FORCE_ANSWER_TO_QUESTIONS=}" ]]; then
RESPONSE=${FORCE_ANSWER_TO_QUESTIONS}
@@ -31,8 +31,8 @@ if [[ -n "${FORCE_ANSWER_TO_QUESTIONS=}" ]]; then
esac
else
echo
- echo "Please confirm ${1}. Are you sure? [y/N/q]"
- read -r RESPONSE
+ echo "Please confirm ${1} (or wait 4 seconds to skip it). Are you sure? [y/N/q]"
+ read -t 4 -r RESPONSE
fi
case "${RESPONSE}" in
[airflow] 17/22: Switches to "/" convention in ghcr.io images
(#17356)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 8fc76971c80d1b52a32bc244548c058aaffe27dd
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Thu Aug 5 18:39:43 2021 +0200
Switches to "/" convention in ghcr.io images (#17356)
We are using ghcr.io as image cache for our CI builds and Breeze
and it seems ghcr.io is being "rebuilt" while running.
We had been using "airflow-.." image convention before,
bacause multiple nesting levels of images were not supported,
however we experienced errors recently with pushing 2.1 images
(https://issues.apache.org/jira/browse/INFRA-22124) and during
investigation it turned out, that it is possible now to use "/"
in the name of the image, and while it still does not introduce
multiple nesting levels and folder structure, the UI of GitHub
treats it like that and if you have image which starts wiht
"airflow/", the airflow prefix is stripped out and you can also
have even more "/" in then name to introduce further hierarchy.
Since we have to change image naming convention due to (still
unresolved) bug with no permission to push the v2-1-test image
we've decided to change naming convention for all our cache
images to follow this - now available - "/" connvention to make
it better structured and easier to manage/understand.
Some more optimisations are implemented - Python, prod-build and
ci-manifest images are only pushed when "latest" image is prepared.
They are not needed for the COMMIT builds because we only need
final images for those builds. This simplified the code quite
a bit.
The push of cache image in CI is done in one job for both
CI and PROD images and the image is rebuilt again with
latest constraints, to account for the latest constraints
but to make sure that UPGRADE_TO_NEWER_DEPENDENCIES
is not set during the build (which invalidates the cache
for next non-upgrade builds)
Backwards-compatibility was implemented to allow PRs that have
not been upgraded to continue building after this one is merged,
also a workaround has been implemented to make this change
to work even if it is not merged yet to main.
This "legacy" mode will be removed in ~week when everybody rebase
on top of main.
Documentation is updated reflecting those changes.
(cherry picked from commit 1bd3a5c68c88cf3840073d6276460a108f864187)
---
.github/workflows/build-images.yml | 18 +++
.github/workflows/ci.yml | 159 +++++++--------------
CI.rst | 51 ++++---
IMAGES.rst | 24 ++--
README.md | 127 +++++++++-------
breeze | 17 +--
dev/retag_docker_images.py | 9 +-
scripts/ci/images/ci_prepare_ci_image_on_ci.sh | 19 +--
scripts/ci/images/ci_prepare_prod_image_on_ci.sh | 29 +---
...ify_ci_image.sh => ci_push_legacy_ci_images.sh} | 35 +----
...y_ci_image.sh => ci_push_legacy_prod_images.sh} | 35 +----
.../images/ci_wait_for_and_verify_all_ci_images.sh | 2 +
.../ci_wait_for_and_verify_all_prod_images.sh | 2 +
.../ci/images/ci_wait_for_and_verify_ci_image.sh | 27 ++--
.../ci/images/ci_wait_for_and_verify_prod_image.sh | 32 +++--
scripts/ci/libraries/_build_images.sh | 109 ++++++++------
scripts/ci/libraries/_initialization.sh | 16 +--
scripts/ci/libraries/_kind.sh | 16 +--
scripts/ci/libraries/_parallel.sh | 7 +-
scripts/ci/libraries/_push_pull_remove_images.sh | 117 +++++++++------
scripts/ci/libraries/_script_init.sh | 2 +-
scripts/ci/selective_ci_checks.sh | 10 +-
22 files changed, 423 insertions(+), 440 deletions(-)
diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml
index ec8f435..c2a9054 100644
--- a/.github/workflows/build-images.yml
+++ b/.github/workflows/build-images.yml
@@ -203,6 +203,10 @@ jobs:
run: ./scripts/ci/images/ci_prepare_ci_image_on_ci.sh
- name: "Push CI images ${{ matrix.python-version }}:${{ env.TARGET_COMMIT_SHA }}"
run: ./scripts/ci/images/ci_push_ci_images.sh
+ # Remove me on 15th of August 2021 after all users had chance to rebase
+ - name: "Push Legacy CI images ${{ matrix.python-version }}:${{ env.TARGET_COMMIT_SHA }}"
+ run: ./scripts/ci/images/ci_push_legacy_ci_images.sh
+ if: github.event_name == 'pull_request_target'
build-prod-images:
permissions:
@@ -229,8 +233,11 @@ jobs:
VERSION_SUFFIX_FOR_PYPI: ".dev0"
steps:
- name: Set envs
+ # Set pull image tag for CI image build, in order to pull the image pushed
+ # Just a moment ago by build-ci-images job
run: |
echo "GITHUB_REGISTRY_PUSH_IMAGE_TAG=${TARGET_COMMIT_SHA}" >> "$GITHUB_ENV"
+ echo "GITHUB_REGISTRY_PULL_IMAGE_TAG=${TARGET_COMMIT_SHA}" >> "$GITHUB_ENV"
- uses: actions/checkout@v2
with:
ref: ${{ env.TARGET_COMMIT_SHA }}
@@ -278,10 +285,21 @@ jobs:
# Pull images built in the previous step
env:
GITHUB_REGISTRY_WAIT_FOR_IMAGE: "true"
+ # Here we are using PULL_IMAGE_TAG set in the environment variables above
- name: "Build PROD images ${{ matrix.python-version }}:${{ env.TARGET_COMMIT_SHA }}"
run: ./scripts/ci/images/ci_prepare_prod_image_on_ci.sh
+ env:
+ # GITHUB_REGISTRY_PULL_IMAGE_TAG is overriden to latest in order to build PROD image using "latest"
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
- name: "Push PROD images ${{ matrix.python-version }}:${{ env.TARGET_COMMIT_SHA }}"
run: ./scripts/ci/images/ci_push_production_images.sh
+ env:
+ # GITHUB_REGISTRY_PULL_IMAGE_TAG is overriden to latest in order to build PROD image using "latest"
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
+ # Remove me on 15th of August 2021 after all users had chance to rebase
+ - name: "Push Legacy PROD images ${{ matrix.python-version }}:${{ env.TARGET_COMMIT_SHA }}"
+ run: ./scripts/ci/images/ci_push_legacy_prod_images.sh
+ if: github.event_name == 'pull_request_target'
cancel-on-ci-build:
permissions:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 22634cf..7228f37 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -1045,108 +1045,6 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
path: /tmp/kind_logs_*
retention-days: 7
- push-prod-images-to-github-registry:
- permissions:
- packages: write
- timeout-minutes: 10
- name: "Push PROD images as cache to GitHub Registry"
- runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
- needs:
- - build-info
- - static-checks
- - tests-sqlite
- - tests-postgres
- - tests-mysql
- - tests-kubernetes
- - prod-images
- - docs
- if: >
- (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/v1-10-test' ||
- github.ref == 'refs/heads/v2-0-test' || github.ref == 'refs/heads/v2-1-test') &&
- github.event_name != 'schedule'
- strategy:
- matrix:
- python-version: ${{ fromJson(needs.build-info.outputs.pythonVersions) }}
- env:
- RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
- PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
- GITHUB_REGISTRY_PUSH_IMAGE_TAG: "latest"
- steps:
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
- uses: actions/checkout@v2
- with:
- persist-credentials: false
- - name: "Setup python"
- uses: actions/setup-python@v2
- with:
- python-version: ${{ env.PYTHON_MAJOR_MINOR_VERSION }}
- - name: "Free space"
- run: ./scripts/ci/tools/free_space.sh
- - name: Set push-python-image
- id: push-python-image
- run: |
- if [[ "${REF}" == 'refs/head/main' || "${REF}" == 'refs/head/main' ]]; then
- echo "::set-output name=wanted::true"
- else
- echo "::set-output name=wanted::false"
- fi
- env:
- REF: ${{ github.ref }}
- - name:
- "Prepare PROD image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:${{ env.GITHUB_REGISTRY_PULL_IMAGE_TAG }}"
- run: ./scripts/ci/images/ci_prepare_prod_image_on_ci.sh
- env:
- # Since we are going to push both final image and build image segment, we need to pull the
- # build image, in case we are pulling from registry rather than building.
- WAIT_FOR_PROD_BUILD_IMAGE: "true"
- WAIT_FOR_PYTHON_BASE_IMAGE: ${{ steps.push-python-image.outputs.wanted}}
- - name: "Push PROD images ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
- run: ./scripts/ci/images/ci_push_production_images.sh
- env:
- PUSH_PYTHON_BASE_IMAGE: ${{ steps.push-python-image.outputs.wanted}}
-
- push-ci-images-to-github-registry:
- permissions:
- packages: write
- timeout-minutes: 10
- name: "Push CI images as cache to GitHub Registry"
- runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
- needs:
- - build-info
- - static-checks
- - tests-sqlite
- - tests-postgres
- - tests-mysql
- - tests-kubernetes
- - ci-images
- - docs
- if: >
- (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/v1-10-test' ||
- github.ref == 'refs/heads/v2-0-test' || github.ref == 'refs/heads/v2-1-test') &&
- github.event_name != 'schedule'
- strategy:
- matrix:
- python-version: ${{ fromJson(needs.build-info.outputs.pythonVersions) }}
- env:
- RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
- PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
- GITHUB_REGISTRY_PUSH_IMAGE_TAG: "latest"
- steps:
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
- uses: actions/checkout@v2
- with:
- persist-credentials: false
- - name: "Setup python"
- uses: actions/setup-python@v2
- with:
- python-version: ${{ env.PYTHON_MAJOR_MINOR_VERSION }}
- - name: "Free space"
- run: ./scripts/ci/tools/free_space.sh
- - name: "Prepare CI image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:${{ env.GITHUB_REGISTRY_PULL_IMAGE_TAG }}"
- run: ./scripts/ci/images/ci_prepare_ci_image_on_ci.sh
- - name: "Push CI image ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
- run: ./scripts/ci/images/ci_push_ci_images.sh
-
constraints:
permissions:
contents: write
@@ -1166,10 +1064,8 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
CURRENT_PYTHON_MAJOR_MINOR_VERSIONS_AS_STRING: ${{needs.build-info.outputs.pythonVersionsListAsString}}
- # Only run it for direct pushes
- if: >
- github.ref == 'refs/heads/main' || github.ref == 'refs/heads/v1-10-test' ||
- github.ref == 'refs/heads/v2-0-test' || github.ref == 'refs/heads/v2-1-test'
+ # Only run it for direct pushes and scheduled builds
+ if: github.event_name == 'push' || github.event_name == 'schedule'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
@@ -1203,17 +1099,68 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- name: "Set constraints branch name"
id: constraints-branch
run: ./scripts/ci/constraints/ci_branch_constraints.sh
+ # only actually push it when we are in apache/airflow repository
- name: Checkout ${{ steps.constraints-branch.outputs.branch }}
uses: actions/checkout@v2
+ if: github.repository == 'apache/airflow'
with:
path: "repo"
ref: ${{ steps.constraints-branch.outputs.branch }}
persist-credentials: false
- name: "Commit changed constraint files for ${{needs.build-info.outputs.pythonVersions}}"
run: ./scripts/ci/constraints/ci_commit_constraints.sh
+ if: github.repository == 'apache/airflow'
- name: "Push changes"
uses: ./.github/actions/github-push-action
+ if: github.repository == 'apache/airflow'
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
branch: ${{ steps.constraints-branch.outputs.branch }}
directory: "repo"
+
+ # Push images to GitHub Registry in Apache repository, if all tests are successful and build
+ # is executed as result of direct push to "main" or one of the "test" branches
+ # It actually rebuilds all images using just-pushed constraints if they changed
+ # It will also check if a new python image was released and will pull the latest one if needed
+ # Same as build-images.yaml
+ push-images-to-github-registry:
+ permissions:
+ packages: write
+ timeout-minutes: 10
+ name: "Push images as cache to GitHub Registry"
+ runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
+ needs:
+ - build-info
+ - constraints
+ - docs
+ # Only run it for direct pushes and scheduled builds
+ if: github.event_name == 'push' || github.event_name == 'schedule'
+ strategy:
+ matrix:
+ python-version: ${{ fromJson(needs.build-info.outputs.pythonVersions) }}
+ env:
+ RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
+ GITHUB_REGISTRY_PUSH_IMAGE_TAG: "latest"
+ PUSH_PYTHON_BASE_IMAGE: "true"
+ CHECK_IF_BASE_PYTHON_IMAGE_UPDATED: "true"
+ steps:
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ uses: actions/checkout@v2
+ with:
+ persist-credentials: false
+ - name: "Setup python"
+ uses: actions/setup-python@v2
+ with:
+ python-version: ${{ env.PYTHON_MAJOR_MINOR_VERSION }}
+ - name: "Free space"
+ run: ./scripts/ci/tools/free_space.sh
+ - name: "Prepare CI image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:latest"
+ run: ./scripts/ci/images/ci_prepare_ci_image_on_ci.sh
+ - name: "Prepare PROD image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:latest"
+ run: ./scripts/ci/images/ci_prepare_prod_image_on_ci.sh
+ - name: "Push CI image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:latest"
+ run: ./scripts/ci/images/ci_push_ci_images.sh
+ - name: "Push PROD images ${{ env.PYTHON_MAJOR_MINOR_VERSION }}:latest"
+ run: ./scripts/ci/images/ci_push_production_images.sh
diff --git a/CI.rst b/CI.rst
index fedd300..6a704d0 100644
--- a/CI.rst
+++ b/CI.rst
@@ -568,12 +568,11 @@ This workflow is a regular workflow that performs all checks of Airflow code.
+---------------------------+----------------------------------------------+-------+-------+------+
| Tests Kubernetes | Run Kubernetes test | Yes(2)| Yes | Yes |
+---------------------------+----------------------------------------------+-------+-------+------+
-| Push PROD images | Pushes PROD images to GitHub Registry (4) | - | Yes | - |
-+---------------------------+----------------------------------------------+-------+-------+------+
-| Push CI images | Pushes CI images to GitHub Registry (4) | - | Yes | - |
-+---------------------------+----------------------------------------------+-------+-------+------+
| Constraints | Upgrade constraints to latest ones (4) | - | Yes | Yes |
+---------------------------+----------------------------------------------+-------+-------+------+
+| Push images | Pushes latest images to GitHub Registry (4) | - | Yes | Yes |
++---------------------------+----------------------------------------------+-------+-------+------+
+
Comments:
@@ -584,8 +583,8 @@ Comments:
You can set it to "false" to disable using shared images - this is slower though as the images
are rebuilt in every job that needs them.
(4) PROD and CI images are pushed as "latest" to GitHub Container registry and constraints are upgraded
- only if all tests are successful. Note that images are not pushed in CRON jobs because they are rebuilt
- from scratch and we want to push incremental changes to the Github Container registry.
+ only if all tests are successful. The images are rebuilt in this step using constraints pushed
+ in the previous step.
CodeQL scan
-----------
@@ -620,7 +619,9 @@ with the COMMIT_SHA id for images that were used in particular build.
The image names follow the patterns (except the Python image, all the images are stored in
https://ghcr.io/ in ``apache`` organization.
-The packages are available under:
+The packages are available under (CONTAINER_NAME is url-encoded name of the image). Note that "/" are
+supported now in the ``ghcr.io`` as apart of the image name within ``apache`` organization, but they
+have to be percent-encoded when you access them via UI (/ = %2F)
``https://github.com/apache/airflow/pkgs/container/<CONTAINER_NAME>``
@@ -631,26 +632,30 @@ The packages are available under:
| (DockerHub) | | Python maintainer release new versions of those image |
| | | with security fixes every few weeks in DockerHub. |
+--------------+----------------------------------------------------------+----------------------------------------------------------+
-| Airflow | airflow-python-v2:<X.Y>-slim-buster | Version of python base image used in Airflow Builds |
-| python base | or | We keep the "latest" version there and also each build |
-| image | airflow-python-v2:<X.Y>-slim-buster-<COMMIT_SHA> | has an associated specific python version that was used. |
+| Airflow | airflow/<BRANCH>/python:<X.Y>-slim-buster | Version of python base image used in Airflow Builds |
+| python base | | We keep the "latest" version only to mark last "good" |
+| image | | python base that went through testing and was pushed. |
+--------------+----------------------------------------------------------+----------------------------------------------------------+
-| CI image | airflow-<BRANCH>-python<X.Y>-ci-v2:latest | CI image - this is the image used for most of the tests. |
-| | or | Contains all provider dependencies and tools useful |
-| | airflow-<BRANCH>-python<X.Y>-ci-v2:<COMMIT_SHA> | For testing. This image is used in Breeze. |
+| PROD Build | airflow/<BRANCH>/prod-build/python<X.Y>:latest | Production Build image - this is the "build" stage of |
+| image | | production image. It contains build-essentials and all |
+| | | necessary apt packages to build/install PIP packages. |
+| | | We keep the "latest" version only to speed up builds. |
+--------------+----------------------------------------------------------+----------------------------------------------------------+
-| Manifest | airflow-<BRANCH>-python<X.Y>-ci-v2-manifest:latest | CI manifest image - this is the image used to optimize |
-| CI image | or | pulls and builds for Breeze development environment |
-| | airflow-<BRANCH>-python<X.Y>-ci-v2-manifest:<COMMIT_SHA> | They store hash indicating whether the image will be |
+| Manifest | airflow/<BRANCH>/ci-manifest/python<X.Y>:latest | CI manifest image - this is the image used to optimize |
+| CI image | | pulls and builds for Breeze development environment |
+| | | They store hash indicating whether the image will be |
| | | faster to build or pull. |
+| | | We keep the "latest" version only to help breeze to |
+| | | check if new image should be pulled. |
+--------------+----------------------------------------------------------+----------------------------------------------------------+
-| PROD Build | airflow-<BRANCH>-python<X.Y>-build-v2:latest | Production Build image - this is the "build" segment of |
-| image | or | production image. It contains build-essentials and all |
-| | airflow-<BRANCH>-python<X.Y>-build-v2:<COMMIT_SHA> | necessary packages to install PIP packages. |
+| CI image | airflow/<BRANCH>/ci/python<X.Y>:latest | CI image - this is the image used for most of the tests. |
+| | or | Contains all provider dependencies and tools useful |
+| | airflow/<BRANCH>/ci/python<X.Y>:<COMMIT_SHA> | For testing. This image is used in Breeze. |
+--------------+----------------------------------------------------------+----------------------------------------------------------+
-| PROD image | airflow-<BRANCH>-python<X.Y>-v2:latest | Production image. This is the actual production image |
+| | | faster to build or pull. |
+| PROD image | airflow/<BRANCH>/prod/python<X.Y>:latest | Production image. This is the actual production image |
| | or | optimized for size. |
-| | airflow-<BRANCH>-python<X.Y>-v2:<COMMIT_SHA> | It contains only compiled libraries and minimal set of |
+| | airflow/<BRANCH>/prod/python<X.Y>:<COMMIT_SHA> | It contains only compiled libraries and minimal set of |
| | | dependencies to run Airflow. |
+--------------+----------------------------------------------------------+----------------------------------------------------------+
@@ -668,9 +673,9 @@ For example knowing that the CI build was for commit ``cd27124534b46c9688a1d89e7
.. code-block:: bash
- docker pull ghcr.io/apache/airflow-main-python3.6-ci:cd27124534b46c9688a1d89e75fcd137ab5137e3
+ docker pull ghcr.io/apache/airflow/main/ci/python3.6:cd27124534b46c9688a1d89e75fcd137ab5137e3
- docker run -it ghcr.io/apache/airflow-main-python3.6-ci:cd27124534b46c9688a1d89e75fcd137ab5137e3
+ docker run -it ghcr.io/apache/airflow/main/ci/python3.6:cd27124534b46c9688a1d89e75fcd137ab5137e3
But you usually need to pass more variables and complex setup if you want to connect to a database or
diff --git a/IMAGES.rst b/IMAGES.rst
index bc34e6c..a1b1ace 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -24,7 +24,7 @@ Airflow has two main images (build from Dockerfiles):
* Production image (Dockerfile) - that can be used to build your own production-ready Airflow installation
You can read more about building and using the production image in the
- `Production Deployments <https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html>`_ document.
+ `Docker stack <https://airflow.apache.org/docs/docker-stack/index.html>`_ documentation.
The image is built using `Dockerfile <Dockerfile>`_
* CI image (Dockerfile.ci) - used for running tests and local development. The image is built using
@@ -246,19 +246,21 @@ Images with a commit SHA (built for pull requests and pushes)
.. code-block:: bash
- ghcr.io/apache/airflow-<BRANCH>-pythonX.Y-ci-v2:<COMMIT_SHA> - for CI images
- ghcr.io/apache/airflow-<BRANCH>-pythonX.Y-v2:<COMMIT_SHA> - for production images
- ghcr.io/apache/airflow-<BRANCH>-pythonX.Y-build-v2:<COMMIT_SHA> - for production build stage
- ghcr.io/apache/airflow-python-v2:X.Y-slim-buster-<COMMIT_SHA> - for base Python images
+ ghcr.io/apache/airflow/<BRANCH>/ci/python<X.Y>:<COMMIT_SHA> - for CI images
+ ghcr.io/apache/airflow/<BRANCH>/prod/python<X.Y>:<COMMIT_SHA> - for production images
+
+We do not push Base Python images and prod-build images when we prepare COMMIT builds, because those
+images are never rebuilt locally, so there is no need to store base images specific for those builds.
Latest images (pushed when main merge succeeds):
.. code-block:: bash
- ghcr.io/apache/airflow-<BRANCH>-pythonX.Y-ci-v2:latest - for CI images
- ghcr.io/apache/airflow-<BRANCH>-pythonX.Y-v2:latest - for production images
- ghcr.io/apache/airflow-<BRANCH>-pythonX.Y-build-v2:latest - for production build stage
- ghcr.io/apache/airflow-python-v2:X.Y-slim-buster - for base Python images
+ ghcr.io/apache/airflow/<BRANCH>/python:<X.Y>-slim-buster - for base Python images
+ ghcr.io/apache/airflow/<BRANCH>/ci/python<X.Y>:latest - for CI images
+ ghcr.io/apache/airflow/<BRANCH>/ci-manifest/python<X.Y>:latest - for CI Manifest images
+ ghcr.io/apache/airflow/<BRANCH>/prod/python<X.Y>:latest - for production images
+ ghcr.io/apache/airflow/<BRANCH>/prod-build/python<X.Y>:latest - for production build stage
You can see all the current GitHub images at `<https://github.com/apache/airflow/packages>`_
@@ -552,8 +554,8 @@ way of querying image details via API. You really need to download the image to
We workaround it in the way that always when we build the image we build a very small image manifest
containing randomly generated UUID and push it to registry together with the main CI image.
The tag for the manifest image reflects the image it refers to with added ``-manifest`` suffix.
-The manifest image for ``ghcr.io/apache/airflow-main-python3.6-ci-v2`` is named
-``ghcr.io/apache/airflow-main-python3.6-ci-v2-manifest``.
+The manifest image for ``ghcr.io/apache/airflow/main/ci/python3.6`` is named
+``ghcr.io/apache/airflow/main/ci-manifest/python3.6``.
The image is quickly pulled (it is really, really small) when important files change and the content
of the randomly generated UUID is compared with the one in our image. If the contents are different
diff --git a/README.md b/README.md
index 9798136..8ab1d2b 100644
--- a/README.md
+++ b/README.md
@@ -27,6 +27,7 @@
[](https://hub.docker.com/r/apache/airflow)
[](https://hub.docker.com/r/apache/airflow)
[](https://pypi.org/project/apache-airflow/)
+[](https://artifacthub.io/packages/search?repo=apache-airflow)
[](https://github.com/psf/black)
[](https://twitter.com/ApacheAirflow)
[](https://s.apache.org/airflow-slack)
@@ -44,19 +45,21 @@ Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The
- [Project Focus](#project-focus)
- [Principles](#principles)
- [Requirements](#requirements)
-- [Support for Python and Kubernetes versions](#support-for-python-and-kubernetes-versions)
- [Getting started](#getting-started)
- [Installing from PyPI](#installing-from-pypi)
- [Official source code](#official-source-code)
- [Convenience packages](#convenience-packages)
- [User Interface](#user-interface)
-- [Support for Python and Kubernetes versions](#support-for-python-and-kubernetes-versions-1)
+- [Semantic versioning](#semantic-versioning)
+- [Version Life Cycle](#version-life-cycle)
+- [Support for Python and Kubernetes versions](#support-for-python-and-kubernetes-versions)
- [Contributing](#contributing)
- [Who uses Apache Airflow?](#who-uses-apache-airflow)
- [Who Maintains Apache Airflow?](#who-maintains-apache-airflow)
- [Can I use the Apache Airflow logo in my presentation?](#can-i-use-the-apache-airflow-logo-in-my-presentation)
- [Airflow merchandise](#airflow-merchandise)
- [Links](#links)
+- [Sponsors](#sponsors)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
@@ -79,9 +82,9 @@ Airflow is not a streaming solution, but it is often used to process real-time d
Apache Airflow is tested with:
-| | Main version (dev) | Stable version (2.1.0) |
+| | Main version (dev) | Stable version (2.1.2) |
| -------------------- | ------------------------- | ------------------------ |
-| Python | 3.6, 3.7, 3.8, 3.9 | 3.6, 3.7, 3.8 |
+| Python | 3.6, 3.7, 3.8, 3.9 | 3.6, 3.7, 3.8, 3.9 |
| Kubernetes | 1.20, 1.19, 1.18 | 1.20, 1.19, 1.18 |
| PostgreSQL | 9.6, 10, 11, 12, 13 | 9.6, 10, 11, 12, 13 |
| MySQL | 5.7, 8 | 5.7, 8 |
@@ -95,34 +98,6 @@ MariaDB is not tested/recommended.
**Note:** SQLite is used in Airflow tests. Do not use it in production. We recommend
using the latest stable version of SQLite for local development.
-## Support for Python and Kubernetes versions
-
-As of Airflow 2.0 we agreed to certain rules we follow for Python and Kubernetes support.
-They are based on the official release schedule of Python and Kubernetes, nicely summarized in the
-[Python Developer's Guide](https://devguide.python.org/#status-of-python-branches) and
-[Kubernetes version skew policy](https://kubernetes.io/docs/setup/release/version-skew-policy/).
-
-1. We drop support for Python and Kubernetes versions when they reach EOL. We drop support for those
- EOL versions in main right after EOL date, and it is effectively removed when we release the
- first new MINOR (Or MAJOR if there is no new MINOR version) of Airflow
- For example for Python 3.6 it means that we drop support in main right after 23.12.2021, and the first
- MAJOR or MINOR version of Airflow released after will not have it.
-
-2. The "oldest" supported version of Python/Kubernetes is the default one. "Default" is only meaningful
- in terms of "smoke tests" in CI PRs which are run using this default version and default reference
- image available in DockerHub. Currently ``apache/airflow:latest`` and ``apache/airflow:2.0.2`` images
- are both Python 3.6 images, however the first MINOR/MAJOR release of Airflow release after 23.12.2021 will
- become Python 3.7 images.
-
-3. We support a new version of Python/Kubernetes in main after they are officially released, as soon as we
- make them work in our CI pipeline (which might not be immediate due to dependencies catching up with
- new versions of Python mostly) we release a new images/support in Airflow based on the working CI setup.
-
-### Additional notes on Python version requirements
-
-* Previous version [requires](https://github.com/apache/airflow/issues/8162) at least Python 3.5.3
- when using Python 3
-
## Getting started
Visit the official Airflow website documentation (latest **stable** release) for help with
@@ -135,7 +110,7 @@ through a more complete [tutorial](https://airflow.apache.org/docs/apache-airflo
For more information on Airflow Improvement Proposals (AIPs), visit
the [Airflow Wiki](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals).
-Official Docker (container) images for Apache Airflow are described in [IMAGES.rst](IMAGES.rst).
+Documentation for dependent projects like provider packages, Docker image, Helm Chart, you'll find it in [the documentation index](https://airflow.apache.org/docs/).
## Installing from PyPI
@@ -155,9 +130,7 @@ correct Airflow tag/version/branch and Python versions in the URL.
1. Installing just Airflow:
-NOTE!!!
-
-Only `pip` installation is currently officially supported.
+> Note: Only `pip` installation is currently officially supported.
While they are some successes with using other tools like [poetry](https://python-poetry.org) or
[pip-tools](https://pypi.org/project/pip-tools), they do not share the same workflow as
@@ -169,15 +142,15 @@ them to appropriate format and workflow that your tool requires.
```bash
-pip install apache-airflow==2.0.2 \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.7.txt"
+pip install apache-airflow==2.1.2 \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.1.2/constraints-3.7.txt"
```
2. Installing with extras (for example postgres,google)
```bash
-pip install apache-airflow[postgres,google]==2.0.2 \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.7.txt"
+pip install apache-airflow[postgres,google]==2.1.2 \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.1.2/constraints-3.7.txt"
```
For information on installing provider packages check
@@ -197,6 +170,7 @@ and our official source code releases:
Following the ASF rules, the source packages released must be sufficient for a user to build and test the
release provided they have access to the appropriate platform and tools.
+
## Convenience packages
There are other ways of installing and using Airflow. Those are "convenience" methods - they are
@@ -209,8 +183,8 @@ Those are - in the order of most common ways people install Airflow:
- [Docker Images](https://hub.docker.com/r/apache/airflow) to install airflow via
`docker` tool, use them in Kubernetes, Helm Charts, `docker-compose`, `docker swarm` etc. You can
read more about using, customising, and extending the images in the
- [Latest docs](https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html), and
- learn details on the internals in the [IMAGES.rst](IMAGES.rst) document.
+ [Latest docs](https://airflow.apache.org/docs/docker-stack/index.html), and
+ learn details on the internals in the [IMAGES.rst](https://github.com/apache/airflow/blob/main/IMAGES.rst) document.
- [Tags in GitHub](https://github.com/apache/airflow/tags) to retrieve the git project sources that
were used to generate official source packages via git
@@ -222,31 +196,65 @@ following the ASF Policy.
- **DAGs**: Overview of all DAGs in your environment.
- 
+ 
-- **Tree View**: Tree representation of a DAG that spans across time.
+- **Tree**: Tree representation of a DAG that spans across time.
- 
+ 
-- **Graph View**: Visualization of a DAG's dependencies and their current status for a specific run.
+- **Graph**: Visualization of a DAG's dependencies and their current status for a specific run.
- 
+ 
- **Task Duration**: Total time spent on different tasks over time.
- 
+ 
+
+- **Gantt**: Duration and overlap of a DAG.
+
+ 
-- **Gantt View**: Duration and overlap of a DAG.
+- **Code**: Quick way to view source code of a DAG.
- 
+ 
-- **Code View**: Quick way to view source code of a DAG.
- 
+## Semantic versioning
+
+As of Airflow 2.0.0, we support strict [SemVer](https://semver.org/) approach for all packages released.
+
+There are few specific rules that we agreed to, that define details of versioning of the different
+packages:
+
+* **Airflow**: SemVer rules apply to core airflow only (excludes any changes to providers).
+ Changing limits for versions of Airflow dependencies is not a breaking change on its own.
+* **Airflow Providers**: SemVer rules apply to changes in the particular provider's code only.
+ SemVer MAJOR and MINOR versions for the packages are independent from Airflow version.
+ For example `google 4.1.0` and `amazon 3.0.3` providers can happily be installed
+ with `Airflow 2.1.2`. If there are limits of cross-dependencies between providers and Airflow packages,
+ they are present in providers as `install_requires` limitations. We aim to keep backwards
+ compatibility of providers with all previously released Airflow 2 versions but
+ there will be sometimes breaking changes that might make some, or all
+ providers, to have minimum Airflow version specified. Change of that minimum supported Airflow version
+ is a breaking change for provider, because installing the new provider might automatically
+ upgrade Airflow (which might be undesired side effect of upgrading provider).
+* **Airflow Helm Chart**: SemVer rules apply to changes in the chart only. SemVer MAJOR and MINOR
+ versions for the chart are independent from the Airflow version. We aim to keep backwards
+ compatibility of the Helm Chart with all released Airflow 2 versions, but some new features might
+ only work starting from specific Airlfow releases. We might however limit the Helm
+ Chart to depend on minimal Airflow version.
+* **Airflow API clients**: SemVer MAJOR and MINOR versions follow MAJOR and MINOR version of Airflow.
+ The first MAJOR or MINOR X.Y.0 release of Airflow should always be followed by X.Y.0 release of
+ all clients. The clients then can release their own PATCH releases with bugfixes,
+ independently of Airflow PATCH releases.
+
+## Version Life Cycle
+
+Apache Airflow version life cycle:
| Version | Current Patch/Minor | State | First Release | Limited Support | EOL/Terminated |
|---------|---------------------|-----------|---------------|-----------------|----------------|
-| 2 | 2.1.1 | Supported | Dec 17, 2020 | Dec 2021 | TBD |
+| 2 | 2.1.2 | Supported | Dec 17, 2020 | Dec 2021 | TBD |
| 1.10 | 1.10.15 | EOL | Aug 27, 2018 | Dec 17, 2020 | June 17, 2021 |
| 1.9 | 1.9.0 | EOL | Jan 03, 2018 | Aug 27, 2018 | Aug 27, 2018 |
| 1.8 | 1.8.2 | EOL | Mar 19, 2017 | Jan 03, 2018 | Jan 03, 2018 |
@@ -272,7 +280,7 @@ They are based on the official release schedule of Python and Kubernetes, nicely
2. The "oldest" supported version of Python/Kubernetes is the default one. "Default" is only meaningful
in terms of "smoke tests" in CI PRs which are run using this default version and default reference
- image available. Currently ``apache/airflow:latest`` and ``apache/airflow:2.1.1` images
+ image available. Currently ``apache/airflow:latest`` and ``apache/airflow:2.1.2` images
are both Python 3.6 images, however the first MINOR/MAJOR release of Airflow release after 23.12.2021 will
become Python 3.7 images.
@@ -289,6 +297,8 @@ They are based on the official release schedule of Python and Kubernetes, nicely
Want to help build Apache Airflow? Check out our [contributing documentation](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst).
+Official Docker (container) images for Apache Airflow are described in [IMAGES.rst](https://github.com/apache/airflow/blob/main/IMAGES.rst).
+
## Who uses Apache Airflow?
More than 400 organizations are using Apache Airflow
@@ -315,3 +325,12 @@ If you would love to have Apache Airflow stickers, t-shirt etc. then check out
- [Documentation](https://airflow.apache.org/docs/apache-airflow/stable/)
- [Chat](https://s.apache.org/airflow-slack)
+
+## Sponsors
+
+The CI infrastructure for Apache Airflow has been sponsored by:
+
+<!-- Orderd by most recently "funded" -->
+
+<a href="https://astronomer.io"><img src="https://assets2.astronomer.io/logos/logoForLIGHTbackground.png" alt="astronomer.io" width="250px"></a>
+<a href="https://aws.amazon.com/opensource/"><img src="docs/integration-logos/aws/AWS-Cloud-alt_light-bg@4x.png" alt="AWS OpenSource" width="130px"></a>
diff --git a/breeze b/breeze
index 1f45ed1..b8387f3 100755
--- a/breeze
+++ b/breeze
@@ -164,6 +164,9 @@ function breeze::setup_default_breeze_constants() {
# Can be overridden by '--force-build-images' flag.
export FORCE_BUILD_IMAGES="false"
+ # When we push from breeze we always want to push base python images
+ export PUSH_PYTHON_BASE_IMAGE="true"
+
# Determines whether to reinstall airflow at entering the image.
export USE_AIRFLOW_VERSION=""
# if set to true, the ci image will look for wheel packages in dist folder and will install them
@@ -569,8 +572,7 @@ EOF
# AIRFLOW_SOURCES
# AIRFLOW_CI_IMAGE
# AIRFLOW_PROD_IMAGE
-# AIRFLOW_PROD_IMAGE_KUBERNETES
-# AIRFLOW_PROD_BASE_TAG
+# AIRFLOW_IMAGE_KUBERNETES
# SQLITE_URL
#
# Arguments:
@@ -633,8 +635,7 @@ export MYSQL_VERSION="${MYSQL_VERSION}"
export AIRFLOW_SOURCES="${AIRFLOW_SOURCES}"
export AIRFLOW_CI_IMAGE="${AIRFLOW_CI_IMAGE}"
export AIRFLOW_PROD_IMAGE="${AIRFLOW_PROD_IMAGE}"
-export AIRFLOW_PROD_IMAGE_KUBERNETES="${AIRFLOW_PROD_IMAGE_KUBERNETES}"
-export AIRFLOW_PROD_BASE_TAG="${AIRFLOW_PROD_BASE_TAG}"
+export AIRFLOW_IMAGE_KUBERNETES="${AIRFLOW_IMAGE_KUBERNETES}"
export SQLITE_URL="${SQLITE_URL}"
export USE_AIRFLOW_VERSION="${USE_AIRFLOW_VERSION}"
export USE_PACKAGES_FROM_DIST="${USE_PACKAGES_FROM_DIST}"
@@ -650,7 +651,6 @@ EOF
#
# Global constants set:
#
-# PYTHON_BASE_IMAGE_VERSION
# PYTHON_BASE_IMAGE
# AIRFLOW_CI_IMAGE
# BUILT_CI_IMAGE_FLAG_FILE
@@ -934,7 +934,6 @@ function breeze::parse_arguments() {
echo
export DOCKER_CACHE="disabled"
# if not set here, docker cached is determined later, depending on type of image to be build
- readonly DOCKER_CACHE
export FORCE_BUILD_IMAGES="true"
shift
;;
@@ -950,7 +949,6 @@ function breeze::parse_arguments() {
echo
export DOCKER_CACHE="local"
# if not set here, docker cached is determined later, depending on type of image to be build
- readonly DOCKER_CACHE
shift
;;
-U | --build-cache-pulled)
@@ -958,14 +956,12 @@ function breeze::parse_arguments() {
echo
export DOCKER_CACHE="pulled"
# if not set here, docker cached is determined later, depending on type of image to be build
- readonly DOCKER_CACHE
shift
;;
-X | --build-cache-disabled)
echo "Use disabled cache to build images"
echo
export DOCKER_CACHE="disabled"
- readonly DOCKER_CACHE
# if not set here, docker cached is determined later, depending on type of image to be build
shift
;;
@@ -1126,7 +1122,6 @@ function breeze::parse_arguments() {
export CHECK_IMAGE_FOR_REBUILD="false"
export SKIP_BUILDING_PROD_IMAGE="true"
export SKIP_CHECK_REMOTE_IMAGE="true"
- export FAIL_ON_GITHUB_DOCKER_PULL_ERROR="true"
shift 2
;;
--init-script)
@@ -3562,7 +3557,7 @@ breeze::check_and_save_all_params
build_images::determine_docker_cache_strategy
-build_images::get_docker_image_names
+build_images::get_docker_cache_image_names
initialization::make_constants_read_only
diff --git a/dev/retag_docker_images.py b/dev/retag_docker_images.py
index 5eeda8e..f29ce1b 100755
--- a/dev/retag_docker_images.py
+++ b/dev/retag_docker_images.py
@@ -36,10 +36,11 @@ PYTHON_VERSIONS = ["3.6", "3.7", "3.8", "3.9"]
GHCR_IO_PREFIX = "ghcr.io/apache/airflow"
GHCR_IO_IMAGES = [
- "{prefix}-{branch}-python{python_version}-ci-v2-manifest:latest",
- "{prefix}-{branch}-python{python_version}-ci-v2:latest",
- "{prefix}-{branch}-python{python_version}-v2:latest",
- "{prefix}-{branch}-python{python_version}-build-v2:latest",
+ "{prefix}/{branch}/ci-manifest/python{python_version}:latest",
+ "{prefix}/{branch}/ci/python{python_version}:latest",
+ "{prefix}/{branch}/prod-build/python{python_version}-build-v2:latest",
+ "{prefix}/{branch}/prod/python{python_version}-build-v2:latest",
+ "{prefix}/{branch}/python:{python_version}-slim-buster",
]
diff --git a/scripts/ci/images/ci_prepare_ci_image_on_ci.sh b/scripts/ci/images/ci_prepare_ci_image_on_ci.sh
index a550038..de1aadc 100755
--- a/scripts/ci/images/ci_prepare_ci_image_on_ci.sh
+++ b/scripts/ci/images/ci_prepare_ci_image_on_ci.sh
@@ -29,24 +29,13 @@ function build_ci_image_on_ci() {
if [[ ${GITHUB_REGISTRY_WAIT_FOR_IMAGE} == "true" ]]; then
# Pretend that the image was build. We already have image with the right sources baked in!
+ # so all the checksums are assumed to be correct
md5sum::calculate_md5sum_for_all_files
- # Tries to wait for the images indefinitely
- # skips further image checks - since we already have the target image
-
- local python_tag_suffix=""
- if [[ ${GITHUB_REGISTRY_PULL_IMAGE_TAG} != "latest" ]]; then
- python_tag_suffix="-${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
- fi
- # first we pull base python image. We will need it to re-push it after main build
- # Becoming the new "latest" image for other builds
- build_images::wait_for_image_tag "${AIRFLOW_PYTHON_BASE_IMAGE}" \
- "${python_tag_suffix}"
-
- # And then the actual image
- build_images::wait_for_image_tag "${AIRFLOW_CI_IMAGE}" \
- ":${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
+ # Remove me on 15th of August 2021 after all users had chance to rebase
+ legacy_ci_image="ghcr.io/${GITHUB_REPOSITORY}-${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci-v2:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
+ build_images::wait_for_image_tag "${AIRFLOW_CI_IMAGE}" ":${GITHUB_REGISTRY_PULL_IMAGE_TAG}" "${legacy_ci_image}"
md5sum::update_all_md5_with_group
else
build_images::rebuild_ci_image_if_needed
diff --git a/scripts/ci/images/ci_prepare_prod_image_on_ci.sh b/scripts/ci/images/ci_prepare_prod_image_on_ci.sh
index dbcb07d..3b5f7a6 100755
--- a/scripts/ci/images/ci_prepare_prod_image_on_ci.sh
+++ b/scripts/ci/images/ci_prepare_prod_image_on_ci.sh
@@ -33,36 +33,13 @@ function build_prod_images_on_ci() {
if [[ ${GITHUB_REGISTRY_WAIT_FOR_IMAGE} == "true" ]]; then
# Tries to wait for the images indefinitely
- # skips further image checks - since we already have the target image
-
- local python_tag_suffix=""
- if [[ ${GITHUB_REGISTRY_PULL_IMAGE_TAG} != "latest" ]]; then
- python_tag_suffix="-${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
- fi
-
- if [[ "${WAIT_FOR_PYTHON_BASE_IMAGE=}" == "true" ]]; then
- # first we pull base python image. We will need it to re-push it after main build
- # Becoming the new "latest" image for other builds
- build_images::wait_for_image_tag "${AIRFLOW_PYTHON_BASE_IMAGE}" \
- "${python_tag_suffix}"
- fi
-
- # And then the actual image
- build_images::wait_for_image_tag "${AIRFLOW_PROD_IMAGE}" \
- ":${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
-
- # And the prod build image
- if [[ "${WAIT_FOR_PROD_BUILD_IMAGE=}" == "true" ]]; then
- # If specified in variable - also waits for the build image
- build_images::wait_for_image_tag "${AIRFLOW_PROD_BUILD_IMAGE}" \
- ":${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
- fi
-
+ # Remove me on 15th of August 2021 after all users had chance to rebase
+ legacy_prod_image="ghcr.io/${GITHUB_REPOSITORY}-${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-v2:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
+ build_images::wait_for_image_tag "${AIRFLOW_PROD_IMAGE}" ":${GITHUB_REGISTRY_PULL_IMAGE_TAG}" "${legacy_prod_image}"
else
build_images::build_prod_images_from_locally_built_airflow_packages
fi
-
# Disable force pulling forced above this is needed for the subsequent scripts so that
# They do not try to pull/build images again
unset FORCE_PULL_IMAGES
diff --git a/scripts/ci/images/ci_wait_for_and_verify_ci_image.sh b/scripts/ci/images/ci_push_legacy_ci_images.sh
similarity index 52%
copy from scripts/ci/images/ci_wait_for_and_verify_ci_image.sh
copy to scripts/ci/images/ci_push_legacy_ci_images.sh
index 105bcfb..aa6696b 100755
--- a/scripts/ci/images/ci_wait_for_and_verify_ci_image.sh
+++ b/scripts/ci/images/ci_push_legacy_ci_images.sh
@@ -15,38 +15,13 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
-if [[ $1 == "" ]]; then
- >&2 echo "Requires python MAJOR/MINOR version as first parameter"
- exit 1
-fi
-
-export PYTHON_MAJOR_MINOR_VERSION=$1
-shift
-
-
# shellcheck source=scripts/ci/libraries/_script_init.sh
. "$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh"
-function pull_ci_image() {
- local image_name_with_tag="${AIRFLOW_CI_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
- start_end::group_start "Pulling ${image_name_with_tag} image"
- push_pull_remove_images::pull_image_if_not_present_or_forced "${image_name_with_tag}"
- start_end::group_end
-
-}
-
-start_end::group_start "Configure Docker Registry"
-build_images::configure_docker_registry
-start_end::group_end
-
-start_end::group_start "Waiting for ${AIRFLOW_CI_IMAGE}"
-
-push_pull_remove_images::wait_for_image "${AIRFLOW_CI_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
+# This script pushes legacy images to old-naming-convention images
+# It should be removed ~ 7th of August, giving users time to rebase their old pull requests
build_images::prepare_ci_build
-pull_ci_image
-start_end::group_end
-if [[ ${VERIFY_IMAGE=} != "false" ]]; then
- verify_image::verify_ci_image "${AIRFLOW_CI_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
-fi
+legacy_ci_image="ghcr.io/${GITHUB_REPOSITORY}-${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci-v2:${GITHUB_REGISTRY_PUSH_IMAGE_TAG}"
+docker tag "${AIRFLOW_CI_IMAGE}" "${legacy_ci_image}"
+docker push "${legacy_ci_image}"
diff --git a/scripts/ci/images/ci_wait_for_and_verify_ci_image.sh b/scripts/ci/images/ci_push_legacy_prod_images.sh
similarity index 52%
copy from scripts/ci/images/ci_wait_for_and_verify_ci_image.sh
copy to scripts/ci/images/ci_push_legacy_prod_images.sh
index 105bcfb..3f74874 100755
--- a/scripts/ci/images/ci_wait_for_and_verify_ci_image.sh
+++ b/scripts/ci/images/ci_push_legacy_prod_images.sh
@@ -15,38 +15,13 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
-if [[ $1 == "" ]]; then
- >&2 echo "Requires python MAJOR/MINOR version as first parameter"
- exit 1
-fi
-
-export PYTHON_MAJOR_MINOR_VERSION=$1
-shift
-
-
# shellcheck source=scripts/ci/libraries/_script_init.sh
. "$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh"
-function pull_ci_image() {
- local image_name_with_tag="${AIRFLOW_CI_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
- start_end::group_start "Pulling ${image_name_with_tag} image"
- push_pull_remove_images::pull_image_if_not_present_or_forced "${image_name_with_tag}"
- start_end::group_end
-
-}
-
-start_end::group_start "Configure Docker Registry"
-build_images::configure_docker_registry
-start_end::group_end
-
-start_end::group_start "Waiting for ${AIRFLOW_CI_IMAGE}"
-
-push_pull_remove_images::wait_for_image "${AIRFLOW_CI_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
+# This script pushes legacy images to old-naming-convention images which will keep old PRs working before
+# Rebasing to main. It should be removed ~7th of August 2021, giving users time to rebase their old pull requests
build_images::prepare_ci_build
-pull_ci_image
-start_end::group_end
-if [[ ${VERIFY_IMAGE=} != "false" ]]; then
- verify_image::verify_ci_image "${AIRFLOW_CI_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
-fi
+legacy_prod_image="ghcr.io/${GITHUB_REPOSITORY}-${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-v2:${GITHUB_REGISTRY_PUSH_IMAGE_TAG}"
+docker tag "${AIRFLOW_PROD_IMAGE}" "${legacy_prod_image}"
+docker push "${legacy_prod_image}"
diff --git a/scripts/ci/images/ci_wait_for_and_verify_all_ci_images.sh b/scripts/ci/images/ci_wait_for_and_verify_all_ci_images.sh
index 4255374..39dfe8f 100755
--- a/scripts/ci/images/ci_wait_for_and_verify_all_ci_images.sh
+++ b/scripts/ci/images/ci_wait_for_and_verify_all_ci_images.sh
@@ -25,6 +25,8 @@ source "${LIBRARIES_DIR}/_all_libs.sh"
initialization::set_output_color_variables
+PARALLEL_TAIL_LENGTH=5
+
parallel::make_sure_gnu_parallel_is_installed
parallel::make_sure_python_versions_are_specified
diff --git a/scripts/ci/images/ci_wait_for_and_verify_all_prod_images.sh b/scripts/ci/images/ci_wait_for_and_verify_all_prod_images.sh
index 08ed54b..6786a31 100755
--- a/scripts/ci/images/ci_wait_for_and_verify_all_prod_images.sh
+++ b/scripts/ci/images/ci_wait_for_and_verify_all_prod_images.sh
@@ -25,6 +25,8 @@ source "${LIBRARIES_DIR}/_all_libs.sh"
initialization::set_output_color_variables
+PARALLEL_TAIL_LENGTH=5
+
parallel::make_sure_gnu_parallel_is_installed
parallel::make_sure_python_versions_are_specified
diff --git a/scripts/ci/images/ci_wait_for_and_verify_ci_image.sh b/scripts/ci/images/ci_wait_for_and_verify_ci_image.sh
index 105bcfb..3d99d5e 100755
--- a/scripts/ci/images/ci_wait_for_and_verify_ci_image.sh
+++ b/scripts/ci/images/ci_wait_for_and_verify_ci_image.sh
@@ -28,25 +28,34 @@ shift
# shellcheck source=scripts/ci/libraries/_script_init.sh
. "$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh"
+image_name_with_tag="${AIRFLOW_CI_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
+
+# Remove me on 15th of August 2021 after all users had chance to rebase
+legacy_ci_image="ghcr.io/${GITHUB_REPOSITORY}-${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci-v2:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
+
function pull_ci_image() {
- local image_name_with_tag="${AIRFLOW_CI_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
- start_end::group_start "Pulling ${image_name_with_tag} image"
- push_pull_remove_images::pull_image_if_not_present_or_forced "${image_name_with_tag}"
+ start_end::group_start "Pulling image: ${IMAGE_AVAILABLE}"
+ push_pull_remove_images::pull_image_if_not_present_or_forced "${IMAGE_AVAILABLE}"
+ # Remove me on 15th of August 2021 after all users had chance to rebase
+ if [[ ${IMAGE_AVAILABLE} != "${image_name_with_tag}" ]]; then
+ verbosity::print_info "Tagging the legacy ${IMAGE_AVAILABLE} with ${image_name_with_tag}"
+ docker tag "${IMAGE_AVAILABLE}" "${image_name_with_tag}"
+ fi
start_end::group_end
-
}
start_end::group_start "Configure Docker Registry"
build_images::configure_docker_registry
start_end::group_end
-start_end::group_start "Waiting for ${AIRFLOW_CI_IMAGE}"
-
-push_pull_remove_images::wait_for_image "${AIRFLOW_CI_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
+start_end::group_start "Waiting for ${image_name_with_tag}"
+# Remove me on 15th of August 2021 after all users had chance to rebase
+push_pull_remove_images::wait_for_image "${image_name_with_tag}" "${legacy_ci_image}"
build_images::prepare_ci_build
-pull_ci_image
start_end::group_end
+pull_ci_image
+
if [[ ${VERIFY_IMAGE=} != "false" ]]; then
- verify_image::verify_ci_image "${AIRFLOW_CI_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
+ verify_image::verify_ci_image "${image_name_with_tag}"
fi
diff --git a/scripts/ci/images/ci_wait_for_and_verify_prod_image.sh b/scripts/ci/images/ci_wait_for_and_verify_prod_image.sh
index 482b0a5..309e506 100755
--- a/scripts/ci/images/ci_wait_for_and_verify_prod_image.sh
+++ b/scripts/ci/images/ci_wait_for_and_verify_prod_image.sh
@@ -28,22 +28,34 @@ shift
# shellcheck source=scripts/ci/libraries/_script_init.sh
. "$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh"
-start_end::group_start "Configure Docker Registry"
-build_images::configure_docker_registry
-start_end::group_end
+image_name_with_tag="${AIRFLOW_PROD_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
+
+# Remove me on 15th of August 2021 after all users had chance to rebase
+legacy_prod_image="ghcr.io/${GITHUB_REPOSITORY}-${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-v2:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
-start_end::group_start "Waiting for ${AIRFLOW_PROD_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
+function pull_prod_image() {
+ start_end::group_start "Pulling image: ${IMAGE_AVAILABLE}"
+ push_pull_remove_images::pull_image_if_not_present_or_forced "${IMAGE_AVAILABLE}"
+ # Remove me on 15th of August 2021 after all users had chance to rebase
+ if [[ ${IMAGE_AVAILABLE} != "${image_name_with_tag}" ]]; then
+ verbosity::print_info "Tagging the legacy ${IMAGE_AVAILABLE} with ${image_name_with_tag}"
+ docker tag "${IMAGE_AVAILABLE}" "${image_name_with_tag}"
+ fi
+ start_end::group_end
+}
-push_pull_remove_images::wait_for_image "${AIRFLOW_PROD_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
+start_end::group_start "Configure Docker Registry"
+build_images::configure_docker_registry
start_end::group_end
-start_end::group_start "Pulling the PROD Image"
+start_end::group_start "Waiting for ${image_name_with_tag}"
+# Remove me on 15th of August 2021 after all users had chance to rebase
+push_pull_remove_images::wait_for_image "${image_name_with_tag}" "${legacy_prod_image}"
build_images::prepare_prod_build
-image_name_with_tag="${AIRFLOW_PROD_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
-verbosity::print_info "Pulling the ${image_name_with_tag} image and tagging with ${AIRFLOW_PROD_IMAGE}"
-push_pull_remove_images::pull_image_if_not_present_or_forced "${image_name_with_tag}"
start_end::group_end
+pull_prod_image
+
if [[ ${VERIFY_IMAGE=} != "false" ]]; then
- verify_image::verify_prod_image "${AIRFLOW_PROD_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
+ verify_image::verify_prod_image "${image_name_with_tag}"
fi
diff --git a/scripts/ci/libraries/_build_images.sh b/scripts/ci/libraries/_build_images.sh
index 4ead442..3ce020b 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -251,7 +251,6 @@ EOF
# Retrieves information about build cache hash random file from the local image
#
function build_images::get_local_build_cache_hash() {
-
set +e
# Remove the container just in case
docker_v rm --force "local-airflow-ci-container" 2>/dev/null >/dev/null
@@ -262,6 +261,7 @@ function build_images::get_local_build_cache_hash() {
LOCAL_MANIFEST_IMAGE_UNAVAILABLE="true"
export LOCAL_MANIFEST_IMAGE_UNAVAILABLE
touch "${LOCAL_IMAGE_BUILD_CACHE_HASH_FILE}"
+ set -e
return
fi
@@ -296,6 +296,7 @@ function build_images::get_remote_image_build_cache_hash() {
REMOTE_DOCKER_REGISTRY_UNREACHABLE="true"
export REMOTE_DOCKER_REGISTRY_UNREACHABLE
touch "${REMOTE_IMAGE_BUILD_CACHE_HASH_FILE}"
+ set -e
return
fi
set -e
@@ -358,49 +359,44 @@ function build_images::get_github_container_registry_image_prefix() {
echo "${GITHUB_REPOSITORY}" | tr '[:upper:]' '[:lower:]'
}
-function build_images::get_docker_image_names() {
- # python image version to use
- export PYTHON_BASE_IMAGE_VERSION=${PYTHON_BASE_IMAGE_VERSION:=${PYTHON_MAJOR_MINOR_VERSION}}
-
+function build_images::get_docker_cache_image_names() {
# Python base image to use
- export PYTHON_BASE_IMAGE="python:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
+ export PYTHON_BASE_IMAGE="python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
local image_name
image_name="${GITHUB_REGISTRY}/$(build_images::get_github_container_registry_image_prefix)"
- # CI image base tag
- export AIRFLOW_CI_BASE_TAG="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci"
+ # Example:
+ # ghcr.io/apache/airflow/main/python:3.8-slim-buster
+ export AIRFLOW_PYTHON_BASE_IMAGE="${image_name}/${BRANCH_NAME}/python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
# Example:
- # ghcr.io/apache/airflow-main-python3.8-ci-v2
- export AIRFLOW_CI_IMAGE="${image_name}-${AIRFLOW_CI_BASE_TAG}${GITHUB_REGISTRY_IMAGE_SUFFIX}"
+ # ghcr.io/apache/airflow/main/ci/python3.8
+ export AIRFLOW_CI_IMAGE="${image_name}/${BRANCH_NAME}/ci/python${PYTHON_MAJOR_MINOR_VERSION}"
- export AIRFLOW_CI_LOCAL_MANIFEST_IMAGE="local-airflow-ci-manifest:${AIRFLOW_CI_BASE_TAG}"
+ # Example:
+ # local-airflow-ci-manifest/main/python3.8
+ export AIRFLOW_CI_LOCAL_MANIFEST_IMAGE="local-airflow-ci-manifest/${BRANCH_NAME}/python${PYTHON_MAJOR_MINOR_VERSION}"
# Example:
- # ghcr.io/apache/airflow-main-python3.8-ci-v2-manifest
- export AIRFLOW_CI_REMOTE_MANIFEST_IMAGE="${image_name}-${AIRFLOW_CI_BASE_TAG}${GITHUB_REGISTRY_IMAGE_SUFFIX}-manifest"
+ # ghcr.io/apache/airflow/main/ci-manifest/python3.8
+ export AIRFLOW_CI_REMOTE_MANIFEST_IMAGE="${image_name}/${BRANCH_NAME}/ci-manifest/python${PYTHON_MAJOR_MINOR_VERSION}"
# File that is touched when the CI image is built for the first time locally
export BUILT_CI_IMAGE_FLAG_FILE="${BUILD_CACHE_DIR}/${BRANCH_NAME}/.built_${PYTHON_MAJOR_MINOR_VERSION}"
- # PROD image to build
- export AIRFLOW_PROD_BASE_TAG="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}"
-
# Example:
- # ghcr.io/apache/airflow-v2-1-test-python-v2:3.6-slim-buster
- export AIRFLOW_PROD_IMAGE="${image_name}-${AIRFLOW_PROD_BASE_TAG}${GITHUB_REGISTRY_IMAGE_SUFFIX}"
-
- # PROD Kubernetes image to build
- export AIRFLOW_PROD_IMAGE_KUBERNETES="${AIRFLOW_PROD_IMAGE}-kubernetes"
+ # ghcr.io/apache/airflow/main/prod/python3.8
+ export AIRFLOW_PROD_IMAGE="${image_name}/${BRANCH_NAME}/prod/python${PYTHON_MAJOR_MINOR_VERSION}"
# Example:
- # ghcr.io/apache/airflow-main-python3.6-build-v2
- export AIRFLOW_PROD_BUILD_IMAGE="${image_name}-${AIRFLOW_PROD_BASE_TAG}-build${GITHUB_REGISTRY_IMAGE_SUFFIX}"
+ # ghcr.io/apache/airflow/main/prod-build/python3.8
+ export AIRFLOW_PROD_BUILD_IMAGE="${image_name}/${BRANCH_NAME}/prod-build/python${PYTHON_MAJOR_MINOR_VERSION}"
+
+ # Kubernetes image to build
+ # ghcr.io/apache/airflow/main/kubernetes/python3.8
+ export AIRFLOW_IMAGE_KUBERNETES="${image_name}/${BRANCH_NAME}/kubernetes/python${PYTHON_MAJOR_MINOR_VERSION}"
- # Example:
- # ghcr.io/apache/airflow-python-v2:3.6-slim-buster
- export AIRFLOW_PYTHON_BASE_IMAGE="${image_name}-python${GITHUB_REGISTRY_IMAGE_SUFFIX}:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
}
@@ -865,17 +861,37 @@ function build_images::build_prod_images() {
fi
}
+# Tags source image with names provided
+# $1 source image
+# $2, $3 - target image names
+function build_images::tag_image() {
+ local source_image_name="$1"
+ shift
+ local target_image_name
+ for target_image_name in "${@}"; do
+ echo
+ echo "Tagging ${source_image_name} as ${target_image_name}."
+ echo
+ docker_v tag "${source_image_name}" "${target_image_name}"
+ done
+}
+
# Waits for image tag to appear in GitHub Registry, pulls it and tags with the target tag
# Parameters:
# $1 - image name to wait for
-# $2 - suffix of the image to wait for
-# $3, $4, ... - target tags to tag the image with
+# $2 - fallback image to wait for
+# $3 - suffix of the image to wait for (Remove me on 15th of August 2021 after all users had chance to rebase)
+# $4, $5, ... - target tags to tag the image with
function build_images::wait_for_image_tag() {
local image_name="${1}"
local image_suffix="${2}"
shift 2
+ # Remove me 7th of August 2021
+ local legacy_image_to_pull="${1}"
+ shift
+
local image_to_wait_for="${image_name}${image_suffix}"
start_end::group_start "Wait for image tag ${image_to_wait_for}"
while true; do
@@ -888,26 +904,34 @@ function build_images::wait_for_image_tag() {
image_hash="$(docker images -q "${image_to_wait_for}" 2>>"${OUTPUT_LOG}" || true)"
if [[ -z "${image_hash}" ]]; then
echo
- echo "The image ${image_to_wait_for} is not yet available. No local hash for the image. Waiting."
+ echo "The image ${image_to_wait_for} is not yet available. No local hash for the image. Falling bacl to legacy."
echo
echo "Last log:"
cat "${OUTPUT_LOG}" || true
echo
- sleep 10
- else
- echo
- echo "The image ${image_to_wait_for} with '${image_name}' tag"
- echo
- echo
- echo "Tagging ${image_to_wait_for} as ${image_name}."
- echo
- docker_v tag "${image_to_wait_for}" "${image_name}"
- for TARGET_TAG in "${@}"; do
+ echo "Checking Legacy image!"
+ # Legacy - Remove me 7th of August 2021
+ set +e
+ echo "${COLOR_BLUE}Docker pull ${legacy_image_to_pull} ${COLOR_RESET}" >"${OUTPUT_LOG}"
+ docker_v pull "${legacy_image_to_pull}" >>"${OUTPUT_LOG}" 2>&1
+ set -e
+ echo "${COLOR_BLUE} Docker images -q ${legacy_image_to_pull}${COLOR_RESET}" >>"${OUTPUT_LOG}"
+ image_hash="$(docker images -q "${legacy_image_to_pull}" 2>>"${OUTPUT_LOG}" || true)"
+ if [[ -z "${image_hash}" ]]; then
echo
- echo "Tagging ${image_to_wait_for} as ${TARGET_TAG}."
+ echo "The image ${legacy_image_to_pull} is not yet available. No local hash for the image. Waiting."
echo
- docker_v tag "${image_to_wait_for}" "${TARGET_TAG}"
- done
+ echo "Last log:"
+ cat "${OUTPUT_LOG}" || true
+ sleep 10
+ else
+ # Legacy - Rremove me 7th of August 2021
+ # Pretend that the image we waited for was downloaded :)
+ build_images::tag_image "${legacy_image_to_pull}" "${image_to_wait_for}" "${image_name}:latest" "${@}"
+ break
+ fi
+ else
+ build_images::tag_image "${image_to_wait_for}" "${image_name}:latest" "${@}"
break
fi
done
@@ -924,7 +948,6 @@ function build_images::determine_docker_cache_strategy() {
export DOCKER_CACHE="pulled"
fi
fi
- readonly DOCKER_CACHE
verbosity::print_info
verbosity::print_info "Using ${DOCKER_CACHE} cache strategy for the build."
verbosity::print_info
diff --git a/scripts/ci/libraries/_initialization.sh b/scripts/ci/libraries/_initialization.sh
index 2cebe7d..a51cdd0 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -153,6 +153,10 @@ function initialization::initialize_base_variables() {
# Dry run - only show docker-compose and docker commands but do not execute them
export DRY_RUN_DOCKER=${DRY_RUN_DOCKER:="false"}
+
+ # By default we only push built ci/prod images - base python images are only pushed
+ # When requested
+ export PUSH_PYTHON_BASE_IMAGE=${PUSH_PYTHON_BASE_IMAGE:="false"}
}
# Determine current branch
@@ -282,9 +286,6 @@ function initialization::initialize_force_variables() {
# Can be set to true to skip if the image is newer in registry
export SKIP_CHECK_REMOTE_IMAGE=${SKIP_CHECK_REMOTE_IMAGE:="false"}
-
- # Should be set to true if you expect image frm GitHub to be present and downloaded
- export FAIL_ON_GITHUB_DOCKER_PULL_ERROR=${FAIL_ON_GITHUB_DOCKER_PULL_ERROR:="false"}
}
# Determine information about the host
@@ -532,7 +533,6 @@ function initialization::initialize_git_variables() {
function initialization::initialize_github_variables() {
# Defaults for interacting with GitHub
export GITHUB_REGISTRY="ghcr.io"
- export GITHUB_REGISTRY_IMAGE_SUFFIX=${GITHUB_REGISTRY_IMAGE_SUFFIX:="-v2"}
export GITHUB_REGISTRY_WAIT_FOR_IMAGE=${GITHUB_REGISTRY_WAIT_FOR_IMAGE:="false"}
export GITHUB_REGISTRY_PULL_IMAGE_TAG=${GITHUB_REGISTRY_PULL_IMAGE_TAG:="latest"}
export GITHUB_REGISTRY_PUSH_IMAGE_TAG=${GITHUB_REGISTRY_PUSH_IMAGE_TAG:="latest"}
@@ -631,7 +631,6 @@ Force variables:
FORCE_BUILD_IMAGES: ${FORCE_BUILD_IMAGES}
FORCE_ANSWER_TO_QUESTIONS: ${FORCE_ANSWER_TO_QUESTIONS}
SKIP_CHECK_REMOTE_IMAGE: ${SKIP_CHECK_REMOTE_IMAGE}
- FAIL_ON_GITHUB_DOCKER_PULL_ERROR: ${FAIL_ON_GITHUB_DOCKER_PULL_ERROR}
Host variables:
@@ -826,8 +825,6 @@ function initialization::make_constants_read_only() {
readonly ADDITIONAL_RUNTIME_APT_DEPS
readonly ADDITIONAL_RUNTIME_APT_ENV
- readonly DOCKER_CACHE
-
readonly GITHUB_REGISTRY
readonly GITHUB_REGISTRY_WAIT_FOR_IMAGE
readonly GITHUB_REGISTRY_PULL_IMAGE_TAG
@@ -843,11 +840,8 @@ function initialization::make_constants_read_only() {
readonly VERSION_SUFFIX_FOR_PYPI
- readonly PYTHON_BASE_IMAGE_VERSION
readonly PYTHON_BASE_IMAGE
- readonly AIRFLOW_CI_BASE_TAG
- readonly AIRFLOW_PROD_BASE_TAG
- readonly AIRFLOW_PROD_IMAGE_KUBERNETES
+ readonly AIRFLOW_IMAGE_KUBERNETES
readonly BUILT_CI_IMAGE_FLAG_FILE
readonly INIT_SCRIPT_FILE
diff --git a/scripts/ci/libraries/_kind.sh b/scripts/ci/libraries/_kind.sh
index d4910d9..1fb77eb 100644
--- a/scripts/ci/libraries/_kind.sh
+++ b/scripts/ci/libraries/_kind.sh
@@ -262,8 +262,8 @@ function kind::build_image_for_kubernetes_tests() {
if [[ -n ${GITHUB_REGISTRY_PULL_IMAGE_TAG=} ]]; then
image_tag="${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
fi
- echo "Building ${AIRFLOW_PROD_IMAGE_KUBERNETES}:latest from ${AIRFLOW_PROD_IMAGE}:${image_tag}"
- docker_v build --tag "${AIRFLOW_PROD_IMAGE_KUBERNETES}:latest" . -f - <<EOF
+ echo "Building ${AIRFLOW_IMAGE_KUBERNETES}:latest from ${AIRFLOW_PROD_IMAGE}:${image_tag}"
+ docker_v build --tag "${AIRFLOW_IMAGE_KUBERNETES}:latest" . -f - <<EOF
FROM ${AIRFLOW_PROD_IMAGE}:${image_tag}
COPY airflow/example_dags/ \${AIRFLOW_HOME}/dags/
@@ -271,11 +271,11 @@ COPY airflow/example_dags/ \${AIRFLOW_HOME}/dags/
COPY airflow/kubernetes_executor_templates/ \${AIRFLOW_HOME}/pod_templates/
EOF
- echo "The ${AIRFLOW_PROD_IMAGE_KUBERNETES}:${image_tag} is prepared for test kubernetes deployment."
+ echo "The ${AIRFLOW_IMAGE_KUBERNETES}:${image_tag} is prepared for test kubernetes deployment."
}
function kind::load_image_to_kind_cluster() {
- kind load docker-image --name "${KIND_CLUSTER_NAME}" "${AIRFLOW_PROD_IMAGE_KUBERNETES}:latest"
+ kind load docker-image --name "${KIND_CLUSTER_NAME}" "${AIRFLOW_IMAGE_KUBERNETES}:latest"
}
MAX_NUM_TRIES_FOR_HEALTH_CHECK=12
@@ -343,8 +343,8 @@ function kind::deploy_airflow_with_helm() {
helm install airflow . \
--timeout 10m0s \
--namespace "${HELM_AIRFLOW_NAMESPACE}" \
- --set "defaultAirflowRepository=${AIRFLOW_PROD_IMAGE_KUBERNETES}" \
- --set "images.airflow.repository=${AIRFLOW_PROD_IMAGE_KUBERNETES}" \
+ --set "defaultAirflowRepository=${AIRFLOW_IMAGE_KUBERNETES}" \
+ --set "images.airflow.repository=${AIRFLOW_IMAGE_KUBERNETES}" \
--set "images.airflow.tag=latest" -v 1 \
--set "defaultAirflowTag=latest" -v 1 \
--set "config.api.auth_backend=airflow.api.auth.backend.basic_auth" \
@@ -376,8 +376,8 @@ function kind::upgrade_airflow_with_helm() {
helm repo add stable https://charts.helm.sh/stable/
helm dep update
helm upgrade airflow . --namespace "${HELM_AIRFLOW_NAMESPACE}" \
- --set "defaultAirflowRepository=${AIRFLOW_PROD_IMAGE_KUBERNETES}" \
- --set "images.airflow.repository=${AIRFLOW_PROD_IMAGE_KUBERNETES}" \
+ --set "defaultAirflowRepository=${AIRFLOW_IMAGE_KUBERNETES}" \
+ --set "images.airflow.repository=${AIRFLOW_IMAGE_KUBERNETES}" \
--set "images.airflow.tag=latest" -v 1 \
--set "defaultAirflowTag=latest" -v 1 \
--set "config.api.auth_backend=airflow.api.auth.backend.basic_auth" \
diff --git a/scripts/ci/libraries/_parallel.sh b/scripts/ci/libraries/_parallel.sh
index f81fee0..69d362a 100644
--- a/scripts/ci/libraries/_parallel.sh
+++ b/scripts/ci/libraries/_parallel.sh
@@ -22,6 +22,9 @@
function parallel::initialize_monitoring() {
PARALLEL_MONITORED_DIR="$(mktemp -d)"
export PARALLEL_MONITORED_DIR
+
+ PARALLEL_TAIL_LENGTH=${PARALLEL_TAIL_LENGTH:=2}
+ export PARALLEL_TAIL_LENGTH
}
function parallel::make_sure_gnu_parallel_is_installed() {
@@ -81,9 +84,9 @@ function parallel::monitor_loop() {
continue
fi
- echo "${COLOR_BLUE}### The last lines for ${parallel_process} process: ${directory}/stdout ###${COLOR_RESET}"
+ echo "${COLOR_BLUE}### The last ${PARALLEL_TAIL_LENGTH} lines for ${parallel_process} process: ${directory}/stdout ###${COLOR_RESET}"
echo
- tail -2 "${directory}/stdout" || true
+ tail "-${PARALLEL_TAIL_LENGTH}" "${directory}/stdout" || true
echo
if [[ -s "${directory}/status" ]]; then
diff --git a/scripts/ci/libraries/_push_pull_remove_images.sh b/scripts/ci/libraries/_push_pull_remove_images.sh
index 0e99b10..2d8e1c4 100644
--- a/scripts/ci/libraries/_push_pull_remove_images.sh
+++ b/scripts/ci/libraries/_push_pull_remove_images.sh
@@ -48,6 +48,7 @@ function push_pull_remove_images::push_image_with_retries() {
# Should be run with set +e
# Parameters:
# $1 -> image to pull
+# $2 - fallback image
function push_pull_remove_images::pull_image_if_not_present_or_forced() {
local image_to_pull="${1}"
local image_hash
@@ -62,25 +63,6 @@ function push_pull_remove_images::pull_image_if_not_present_or_forced() {
echo "Pulling the image ${image_to_pull}"
echo
docker_v pull "${image_to_pull}"
- local exit_value="$?"
- if [[ ${exit_value} != "0" && ${FAIL_ON_GITHUB_DOCKER_PULL_ERROR} == "true" ]]; then
- echo
- echo """
-${COLOR_RED}ERROR: Exiting on docker pull error
-
-If you have authorisation problems, you might want to run:
-
-docker login ${image_to_pull%%\/*}
-
-You need to use generate token as the password, not your personal password.
-You can generate one at https://github.com/settings/tokens
-Make sure to choose 'read:packages' scope.
-${COLOR_RESET}
-"""
- exit ${exit_value}
- fi
- echo
- return ${exit_value}
fi
}
@@ -90,7 +72,7 @@ function push_pull_remove_images::check_and_rebuild_python_base_image_if_needed(
local dockerhub_python_version
dockerhub_python_version=$(docker run "${PYTHON_BASE_IMAGE}" python -c 'import sys; print(sys.version)')
local local_python_version
- local_python_version=$(docker run "${AIRFLOW_PYTHON_BASE_IMAGE}" python -c 'import sys; print(sys.version)')
+ local_python_version=$(docker run "${AIRFLOW_PYTHON_BASE_IMAGE}" python -c 'import sys; print(sys.version)' || true)
if [[ ${local_python_version} != "${dockerhub_python_version}" ]]; then
echo
echo "There is a new Python Base image updated!"
@@ -102,6 +84,10 @@ function push_pull_remove_images::check_and_rebuild_python_base_image_if_needed(
docker_v build \
--label "org.opencontainers.image.source=https://github.com/${GITHUB_REPOSITORY}" \
-t "${AIRFLOW_PYTHON_BASE_IMAGE}" -
+ else
+ echo
+ echo "Not rebuilding the base python image - the image has the same python version ${dockerhub_python_version}"
+ echo
fi
}
@@ -116,10 +102,10 @@ function push_pull_remove_images::check_and_rebuild_python_base_image_if_needed(
# it will pull the right image using the specified suffix
function push_pull_remove_images::pull_base_python_image() {
echo
- echo "Docker pulling base python image. Upgrade to newer deps: ${UPGRADE_TO_NEWER_DEPENDENCIES}"
+ echo "Docker pull base python image. Upgrade to newer deps: ${UPGRADE_TO_NEWER_DEPENDENCIES}"
echo
if [[ -n ${DETECTED_TERMINAL=} ]]; then
- echo -n "Docker pulling base python image. Upgrade to newer deps: ${UPGRADE_TO_NEWER_DEPENDENCIES}
+ echo -n "Docker pull base python image. Upgrade to newer deps: ${UPGRADE_TO_NEWER_DEPENDENCIES}
" > "${DETECTED_TERMINAL}"
fi
if [[ ${GITHUB_REGISTRY_PULL_IMAGE_TAG} != "latest" ]]; then
@@ -132,8 +118,14 @@ function push_pull_remove_images::pull_base_python_image() {
return 1
fi
else
+ set +e
push_pull_remove_images::pull_image_if_not_present_or_forced "${AIRFLOW_PYTHON_BASE_IMAGE}"
- if [[ ${CHECK_IF_BASE_PYTHON_IMAGE_UPDATED} == "true" ]] ; then
+ local res="$?"
+ set -e
+ if [[ ${CHECK_IF_BASE_PYTHON_IMAGE_UPDATED} == "true" || ${res} != "0" ]] ; then
+ # Rebuild the base python image using DockerHub - either when we explicitly want it
+ # or when there is no image available yet in ghcr.io (usually when you build it for the
+ # first time in your repository
push_pull_remove_images::check_and_rebuild_python_base_image_if_needed
fi
fi
@@ -151,8 +143,26 @@ function push_pull_remove_images::pull_ci_images_if_needed() {
fi
fi
if [[ "${DOCKER_CACHE}" == "pulled" ]]; then
+ set +e
push_pull_remove_images::pull_image_if_not_present_or_forced \
"${AIRFLOW_CI_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
+ local res="$?"
+ set -e
+ if [[ ${res} != "0" ]]; then
+ if [[ ${GITHUB_REGISTRY_PULL_IMAGE_TAG} == "latest" ]] ; then
+ echo
+ echo "The CI image cache does not exist. This is likely the first time you build the image"
+ echo "Switching to 'local' cache for docker images"
+ echo
+ DOCKER_CACHE="local"
+ else
+ echo
+ echo "The CI image cache does not exist and we want to pull tag ${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
+ echo "Failing as we have to pull the tagged image in order to continue"
+ echo
+ return "${res}"
+ fi
+ fi
fi
}
@@ -169,12 +179,33 @@ function push_pull_remove_images::pull_prod_images_if_needed() {
fi
fi
if [[ "${DOCKER_CACHE}" == "pulled" ]]; then
+ set +e
# "Build" segment of production image
push_pull_remove_images::pull_image_if_not_present_or_forced \
"${AIRFLOW_PROD_BUILD_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
- # "Main" segment of production image
- push_pull_remove_images::pull_image_if_not_present_or_forced \
- "${AIRFLOW_PROD_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
+ local res="$?"
+ if [[ ${res} == "0" ]]; then
+ # "Main" segment of production image
+ push_pull_remove_images::pull_image_if_not_present_or_forced \
+ "${AIRFLOW_PROD_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
+ res="$?"
+ fi
+ set -e
+ if [[ ${res} != "0" ]]; then
+ if [[ ${GITHUB_REGISTRY_PULL_IMAGE_TAG} == "latest" ]] ; then
+ echo
+ echo "The PROD image cache does not exist. This is likely the first time you build the image"
+ echo "Switching to 'local' cache for docker images"
+ echo
+ DOCKER_CACHE="local"
+ else
+ echo
+ echo "The PROD image cache does not exist and we want to pull tag ${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
+ echo "Failing as we have to pull the tagged image in order to continue"
+ echo
+ return "${res}"
+ fi
+ fi
fi
}
@@ -203,17 +234,12 @@ function push_pull_remove_images::push_ci_images_to_github() {
local airflow_ci_tagged_image="${AIRFLOW_CI_IMAGE}:${GITHUB_REGISTRY_PUSH_IMAGE_TAG}"
docker_v tag "${AIRFLOW_CI_IMAGE}" "${airflow_ci_tagged_image}"
push_pull_remove_images::push_image_with_retries "${airflow_ci_tagged_image}"
+ # Also push ci manifest image if GITHUB_REGISTRY_PUSH_IMAGE_TAG is "latest"
if [[ ${GITHUB_REGISTRY_PUSH_IMAGE_TAG} == "latest" ]]; then
- local airflow_ci_manifest_tagged_image="${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}:${GITHUB_REGISTRY_PUSH_IMAGE_TAG}"
+ local airflow_ci_manifest_tagged_image="${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}:latest"
docker_v tag "${AIRFLOW_CI_LOCAL_MANIFEST_IMAGE}" "${airflow_ci_manifest_tagged_image}"
push_pull_remove_images::push_image_with_retries "${airflow_ci_manifest_tagged_image}"
fi
- if [[ -n ${GITHUB_SHA=} ]]; then
- # Also push image to GitHub registry with commit SHA
- local airflow_ci_sha_image="${AIRFLOW_CI_IMAGE}:${COMMIT_SHA}"
- docker_v tag "${AIRFLOW_CI_IMAGE}" "${airflow_ci_sha_image}"
- push_pull_remove_images::push_image_with_retries "${airflow_ci_sha_image}"
- fi
}
# Pushes PROD image to registry in GitHub
@@ -222,19 +248,18 @@ function push_pull_remove_images::push_ci_images_to_github() {
# "${COMMIT_SHA}" - in case of pull-request triggered 'workflow_run' builds
# "latest" - in case of push builds
function push_pull_remove_images::push_prod_images_to_github () {
+ if [[ "${PUSH_PYTHON_BASE_IMAGE=}" != "false" ]]; then
+ push_pull_remove_images::push_python_image_to_github
+ fi
local airflow_prod_tagged_image="${AIRFLOW_PROD_IMAGE}:${GITHUB_REGISTRY_PUSH_IMAGE_TAG}"
docker_v tag "${AIRFLOW_PROD_IMAGE}" "${airflow_prod_tagged_image}"
push_pull_remove_images::push_image_with_retries "${airflow_prod_tagged_image}"
- if [[ -n ${COMMIT_SHA=} ]]; then
- # Also push image to GitHub registry with commit SHA
- local airflow_prod_sha_image="${AIRFLOW_PROD_IMAGE}:${COMMIT_SHA}"
- docker_v tag "${AIRFLOW_PROD_IMAGE}" "${airflow_prod_sha_image}"
- push_pull_remove_images::push_image_with_retries "${airflow_prod_sha_image}"
+ # Also push prod build image if GITHUB_REGISTRY_PUSH_IMAGE_TAG is "latest"
+ if [[ ${GITHUB_REGISTRY_PUSH_IMAGE_TAG} == "latest" ]]; then
+ local airflow_prod_build_tagged_image="${AIRFLOW_PROD_BUILD_IMAGE}:latest"
+ docker_v tag "${AIRFLOW_PROD_BUILD_IMAGE}" "${airflow_prod_build_tagged_image}"
+ push_pull_remove_images::push_image_with_retries "${airflow_prod_build_tagged_image}"
fi
- # Also push prod build image
- local airflow_prod_build_tagged_image="${AIRFLOW_PROD_BUILD_IMAGE}:${GITHUB_REGISTRY_PUSH_IMAGE_TAG}"
- docker_v tag "${AIRFLOW_PROD_BUILD_IMAGE}" "${airflow_prod_build_tagged_image}"
- push_pull_remove_images::push_image_with_retries "${airflow_prod_build_tagged_image}"
}
# waits for an image to be available in GitHub Container Registry. Should be run with `set +e`
@@ -253,12 +278,18 @@ function push_pull_remove_images::check_image_manifest() {
}
# waits for an image to be available in the GitHub registry
+# Remove the fallback on 7th of August 2021
function push_pull_remove_images::wait_for_image() {
set +e
- echo " Waiting for github registry image: " "$1"
+ echo " Waiting for github registry image: $1 with $2 fallback"
while true
do
if push_pull_remove_images::check_image_manifest "$1"; then
+ export IMAGE_AVAILABLE="$1"
+ break
+ fi
+ if push_pull_remove_images::check_image_manifest "$2"; then
+ export IMAGE_AVAILABLE="$2"
break
fi
sleep 30
diff --git a/scripts/ci/libraries/_script_init.sh b/scripts/ci/libraries/_script_init.sh
index 0f3c862..dc79fd5 100755
--- a/scripts/ci/libraries/_script_init.sh
+++ b/scripts/ci/libraries/_script_init.sh
@@ -41,7 +41,7 @@ build_images::determine_docker_cache_strategy
initialization::get_environment_for_builds_on_ci
-build_images::get_docker_image_names
+build_images::get_docker_cache_image_names
initialization::make_constants_read_only
diff --git a/scripts/ci/selective_ci_checks.sh b/scripts/ci/selective_ci_checks.sh
index e7a9144..ee3b506 100755
--- a/scripts/ci/selective_ci_checks.sh
+++ b/scripts/ci/selective_ci_checks.sh
@@ -46,10 +46,12 @@ function check_upgrade_to_newer_dependencies_needed() {
# shellcheck disable=SC2153
if [[ "${UPGRADE_TO_NEWER_DEPENDENCIES}" != "false" ||
${GITHUB_EVENT_NAME=} == 'push' || ${GITHUB_EVENT_NAME=} == "scheduled" ]]; then
- # Trigger upgrading to latest constraints where label is set or when
- # SHA of the merge commit triggers rebuilding layer in the docker image
+ # Trigger upgrading to latest constraints when we are in push or schedule event or when it is forced
+ # By UPGRADE_TO_NEWER_DEPENDENCIES set to non-false. The variable is set to
+ # SHA of the merge commit - so that it always triggers rebuilding layer in the docker image
# Each build that upgrades to latest constraints will get truly latest constraints, not those
- # Cached in the image this way
+ # Cached in the image if we set it to "true". This upgrade_to_newer_dependencies variable
+ # can later be overridden in case we find that setup.* files changed (see below)
upgrade_to_newer_dependencies="${INCOMING_COMMIT_SHA}"
fi
}
@@ -338,6 +340,8 @@ function check_if_setup_files_changed() {
show_changed_files
if [[ $(count_changed_files) != "0" ]]; then
+ # In case the setup files changed, we automatically force upgrading to newer dependencies
+ # no matter what was set before
upgrade_to_newer_dependencies="${INCOMING_COMMIT_SHA}"
fi
start_end::group_end
[airflow] 19/22: Disable Helm tests when branch is not main
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 562be8cac469f9c269616baf1d7379041a322e8c
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Fri Aug 6 10:19:26 2021 +0200
Disable Helm tests when branch is not main
We are preparing Helm chart from main branch only and we never
run it from airflow version branches (similarly as providers)
This change disables Helm Chart tests in case default branch
is different than main.
---
.github/workflows/ci.yml | 16 ++++++++++------
1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 7228f37..0d65286 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -574,7 +574,8 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
PYTHON_MAJOR_MINOR_VERSION: ${{needs.build-info.outputs.defaultPythonVersion}}
if: >
needs.build-info.outputs.needs-helm-tests == 'true' &&
- (github.repository == 'apache/airflow' || github.event_name != 'schedule')
+ (github.repository == 'apache/airflow' || github.event_name != 'schedule') &&
+ needs.build-info.outputs.default-branch == 'main'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
@@ -863,7 +864,8 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
# Only upload coverage on merges to main
if: >
github.ref == 'refs/heads/main' && github.repository == 'apache/airflow' &&
- github.event_name == 'push'
+ github.event_name == 'push' &&
+ needs.build-info.outputs.default-branch == 'main'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
@@ -945,8 +947,9 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
CURRENT_KUBERNETES_VERSIONS_AS_STRING: >
${{needs.build-info.outputs.kubernetesVersionsListAsString}}
if: >
- needs.build-info.outputs.run-kubernetes-tests == 'true' ||
- needs.build-info.outputs.needs-helm-tests == 'true'
+ ( needs.build-info.outputs.run-kubernetes-tests == 'true' ||
+ needs.build-info.outputs.needs-helm-tests == 'true' ) &&
+ needs.build-info.outputs.default-branch == 'main'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
@@ -999,7 +1002,9 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
${{needs.build-info.outputs.pythonVersionsListAsString}}
CURRENT_KUBERNETES_VERSIONS_AS_STRING: >
${{needs.build-info.outputs.kubernetesVersionsListAsString}}
- if: needs.build-info.outputs.run-kubernetes-tests == 'true'
+ if: >
+ needs.build-info.outputs.run-kubernetes-tests == 'true' &&
+ needs.build-info.outputs.default-branch == 'main'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
@@ -1059,7 +1064,6 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- tests-sqlite
- tests-mysql
- tests-postgres
- - tests-kubernetes
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
[airflow] 16/22: Fix failing static checks in main (#17424)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit f1966908e0f9b7dee4415046c19a213781271fe4
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Wed Aug 4 23:33:29 2021 +0200
Fix failing static checks in main (#17424)
(cherry picked from commit 60848a6ea78e29f20004a5e9f177335e6b99c706)
---
airflow/utils/log/file_task_handler.py | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/airflow/utils/log/file_task_handler.py b/airflow/utils/log/file_task_handler.py
index 56b9d23..2387633 100644
--- a/airflow/utils/log/file_task_handler.py
+++ b/airflow/utils/log/file_task_handler.py
@@ -187,10 +187,14 @@ class FileTaskHandler(logging.Handler):
response.encoding = "utf-8"
if response.status_code == 403:
- log += "*** !!!! Please make sure that all your webservers and workers have" \
- " the same 'secret_key' configured in 'webserver' section !!!!!\n***"
- log += "*** See more at https://airflow.apache.org/docs/apache-airflow/" \
- "stable/configurations-ref.html#secret-key\n***"
+ log += (
+ "*** !!!! Please make sure that all your Airflow components (e.g. schedulers, webservers and workers) have"
+ " the same 'secret_key' configured in 'webserver' section !!!!!\n***"
+ )
+ log += (
+ "*** See more at https://airflow.apache.org/docs/apache-airflow/"
+ "stable/configurations-ref.html#secret-key\n***"
+ )
# Check if the resource was properly fetched
response.raise_for_status()
[airflow] 15/22: Improve diagnostics message when users have
secret_key misconfigured (#17410)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 2df7e6e41cf6968b48658c21e989868cb3960027
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Wed Aug 4 15:15:38 2021 +0200
Improve diagnostics message when users have secret_key misconfigured (#17410)
* Improve diagnostics message when users have secret_key misconfigured
Recently fixed log open-access vulnerability have caused
quite a lot of questions and issues from the affected users who
did not have webserver/secret_key configured for their workers
(effectively leading to random value for those keys for workers)
This PR explicitly explains the possible reason for the problem and
encourages the user to configure their webserver's secret_key
in both - workers and webserver.
Related to: #17251 and a number of similar slack discussions.
(cherry picked from commit 2321020e29511f3741940440739e4cc01c0a7ba2)
---
airflow/utils/log/file_task_handler.py | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/airflow/utils/log/file_task_handler.py b/airflow/utils/log/file_task_handler.py
index 2dc9beb..56b9d23 100644
--- a/airflow/utils/log/file_task_handler.py
+++ b/airflow/utils/log/file_task_handler.py
@@ -186,6 +186,11 @@ class FileTaskHandler(logging.Handler):
)
response.encoding = "utf-8"
+ if response.status_code == 403:
+ log += "*** !!!! Please make sure that all your webservers and workers have" \
+ " the same 'secret_key' configured in 'webserver' section !!!!!\n***"
+ log += "*** See more at https://airflow.apache.org/docs/apache-airflow/" \
+ "stable/configurations-ref.html#secret-key\n***"
# Check if the resource was properly fetched
response.raise_for_status()
[airflow] 18/22: Fixes some static check errors (#17432)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 2ba71c9faddf7343f3ba079b280c8aeb269482e9
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Thu Aug 5 08:43:51 2021 +0200
Fixes some static check errors (#17432)
Disabling pre-commit even for a while and usin GitHub to
edit .rst files can have disastrous consequences for the static checks.
This one fixes my own mistakes from 3 PRs (!) after being far too
hasty in adding/merging them.
Will do better next time.
(cherry picked from commit 02586e3edd3260f2bd2c5ef9c646770d38f50069)
---
airflow/utils/log/file_task_handler.py | 3 ++-
docs/apache-airflow/start/docker.rst | 6 +++---
.../docker-stack/docker-examples/extending/add-providers/Dockerfile | 1 -
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/airflow/utils/log/file_task_handler.py b/airflow/utils/log/file_task_handler.py
index 2387633..e15e375 100644
--- a/airflow/utils/log/file_task_handler.py
+++ b/airflow/utils/log/file_task_handler.py
@@ -188,7 +188,8 @@ class FileTaskHandler(logging.Handler):
if response.status_code == 403:
log += (
- "*** !!!! Please make sure that all your Airflow components (e.g. schedulers, webservers and workers) have"
+ "*** !!!! Please make sure that all your Airflow components (e.g. "
+ "schedulers, webservers and workers) have"
" the same 'secret_key' configured in 'webserver' section !!!!!\n***"
)
log += (
diff --git a/docs/apache-airflow/start/docker.rst b/docs/apache-airflow/start/docker.rst
index 747f87c..78d5c60 100644
--- a/docs/apache-airflow/start/docker.rst
+++ b/docs/apache-airflow/start/docker.rst
@@ -73,9 +73,9 @@ Using custom images
When you want to run Airflow locally, you might want to use an extended image, containing some additional dependencies - for
example you might add new python packages, or upgrade airflow providers to a later version. This can be done very easily
-by placing a custom Dockerfile alongside your `docker-compose.yaml`. Then you can use `docker-compose build` command to build your image (you need to
-do it only once). You can also add the `--build` flag to your `docker-compose` commands to rebuild the images
-on-the-fly when you run other `docker-compose` commands.
+by placing a custom Dockerfile alongside your ``docker-compose.yaml``. Then you can use ``docker-compose build`` command
+to build your image (you need to do it only once). You can also add the ``--build`` flag to your ``docker-compose`` commands
+to rebuild the images on-the-fly when you run other ``docker-compose`` commands.
Examples of how you can extend the image with custom providers, python packages,
apt packages and more can be found in :doc:`Building the image <docker-stack:build>`.
diff --git a/docs/docker-stack/docker-examples/extending/add-providers/Dockerfile b/docs/docker-stack/docker-examples/extending/add-providers/Dockerfile
index cdf7a42..b65262f 100644
--- a/docs/docker-stack/docker-examples/extending/add-providers/Dockerfile
+++ b/docs/docker-stack/docker-examples/extending/add-providers/Dockerfile
@@ -18,4 +18,3 @@
FROM apache/airflow:2.1.2
RUN pip install --no-cache-dir apache-airflow-providers-docker==2.1.0
# [END Dockerfile]
-
[airflow] 11/22: Increases timeout for helm chart builds (#17417)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 38c7115599884e1b927fc3c06d8035fb59aff3f2
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Wed Aug 4 22:02:46 2021 +0200
Increases timeout for helm chart builds (#17417)
(cherry picked from commit 4348239686b3e2d3df17e5e8ed6462dfc6b98164)
---
.github/workflows/ci.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 8c31784..22634cf 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -561,7 +561,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
PACKAGE_FORMAT: "sdist"
tests-helm:
- timeout-minutes: 20
+ timeout-minutes: 40
name: "Python unit tests for helm chart"
runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
needs: [build-info, ci-images]
[airflow] 09/22: Enhancement to bash scripts (#17098)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 7a43960a5c3c5822c56149ece261ab99c6a92282
Author: Shraman Basyal <49...@users.noreply.github.com>
AuthorDate: Mon Aug 2 14:49:11 2021 -0500
Enhancement to bash scripts (#17098)
(cherry picked from commit e544ffc2241a067b7433a954f51d9fa91f345c29)
---
scripts/in_container/prod/airflow_scheduler_autorestart.sh | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/scripts/in_container/prod/airflow_scheduler_autorestart.sh b/scripts/in_container/prod/airflow_scheduler_autorestart.sh
index 09e0344..0fdc644 100755
--- a/scripts/in_container/prod/airflow_scheduler_autorestart.sh
+++ b/scripts/in_container/prod/airflow_scheduler_autorestart.sh
@@ -18,7 +18,11 @@
while echo "Running"; do
airflow scheduler -n 5
- echo "Scheduler crashed with exit code $?. Respawning.." >&2
- date >> /tmp/airflow_scheduler_errors.txt
+ return_code=$?
+ if (( return_code != 0 )); then
+ echo "Scheduler crashed with exit code $return_code. Respawning.." >&2
+ date >> /tmp/airflow_scheduler_errors.txt
+ fi
+
sleep 1
done
[airflow] 22/22: Better diagnostics and self-healing of
docker-compose (#17484)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-1-test-jarek-cherry-picks
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 3833fb724d81f05afc8940444616c4fd83a4adad
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Mon Aug 9 10:41:57 2021 +0200
Better diagnostics and self-healing of docker-compose (#17484)
There are several ways people might get the quick-start
docker-compose running messed up (especially on linux):
1) they do not run initialization steps and run docker-compose-up
2) they do not run docker-compose-init first
Also on MacOS/Windows default memory/disk settings are not
enough to run Airflow via docker-compose and people are reporting
"Airflow not working" where they simply do not allocate enough
resources.
Finally the docker compose does not support all versions of airflow
and various problems might occur when you use this
docker compose with old version of airflow.
This change adds the following improvements:
* automated check of minimum version of airflow supported
* mkdir -p in the directories creation in instructions
* automated checking if AIRFLOW_UID has been set (and printing
error and instruction link in case it is not)
* prints warning about too-low memory, cpu, disk allocation
and instruction link where to read about it
* automated fixing of ownership of the directories created in
case they were not created initially and ended up owned by
root user
(cherry picked from commit 763860c8109fd3326fba92abc52fb8ffb1d39834)
---
docs/apache-airflow/start/docker-compose.yaml | 57 ++++++++++++++++++++++++++-
docs/apache-airflow/start/docker.rst | 23 +++++++++--
2 files changed, 74 insertions(+), 6 deletions(-)
diff --git a/docs/apache-airflow/start/docker-compose.yaml b/docs/apache-airflow/start/docker-compose.yaml
index 06991e7..b74f5ca 100644
--- a/docs/apache-airflow/start/docker-compose.yaml
+++ b/docs/apache-airflow/start/docker-compose.yaml
@@ -28,7 +28,7 @@
# AIRFLOW_UID - User ID in Airflow containers
# Default: 50000
# AIRFLOW_GID - Group ID in Airflow containers
-# Default: 50000
+# Default: 0
#
# Those configurations are useful mostly in case of standalone testing/running Airflow in test/try-out mode
#
@@ -133,13 +133,66 @@ services:
airflow-init:
<<: *airflow-common
- command: version
+ entrypoint: /bin/bash
+ command:
+ - -c
+ - |
+ function ver() {
+ printf "%04d%04d%04d%04d" $${1//./ }
+ }
+ airflow_version=$$(gosu airflow airflow version)
+ airflow_version_comparable=$$(ver $${airflow_version})
+ min_airflow_version=2.1.0
+ min_airlfow_version_comparable=$$(ver $${min_airflow_version})
+ if (( airflow_version_comparable < min_airlfow_version_comparable )); then
+ echo -e "\033[1;31mERROR!!!: Too old Airflow version $${airflow_version}!\e[0m"
+ echo "The minimum Airflow version supported: $${min_airflow_version}. Only use this or higher!"
+ exit 1
+ fi
+ if [[ -z "${AIRFLOW_UID}" ]]; then
+ echo -e "\033[1;31mERROR!!!: AIRFLOW_UID not set!\e[0m"
+ echo "Please follow these instructions to set AIRFLOW_UID and AIRFLOW_GID environment variables:
+ https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#initializing-environment"
+ exit 1
+ fi
+ one_meg=1048576
+ mem_available=$$(($$(getconf _PHYS_PAGES) * $$(getconf PAGE_SIZE) / one_meg))
+ cpus_available=$$(grep -cE 'cpu[0-9]+' /proc/stat)
+ disk_available=$$(df / | tail -1 | awk '{print $$4}')
+ warning_resources="false"
+ if (( mem_available < 4000 )) ; then
+ echo -e "\033[1;33mWARNING!!!: Not enough memory available for Docker.\e[0m"
+ echo "At least 4GB of memory required. You have $$(numfmt --to iec $$((mem_available * one_meg)))"
+ warning_resources="true"
+ fi
+ if (( cpus_available < 2 )); then
+ echo -e "\033[1;33mWARNING!!!: Not enough CPUS available for Docker.\e[0m"
+ echo "At least 2 CPUs recommended. You have $${cpus_available}"
+ warning_resources="true"
+ fi
+ if (( disk_available < one_meg * 10 )); then
+ echo -e "\033[1;33mWARNING!!!: Not enough Disk space available for Docker.\e[0m"
+ echo "At least 10 GBs recommended. You have $$(numfmt --to iec $$((disk_available * 1024 )))"
+ warning_resources="true"
+ fi
+ if [[ $${warning_resources} == "true" ]]; then
+ echo
+ echo -e "\033[1;33mWARNING!!!: You have not enough resources to run Airflow (see above)!\e[0m"
+ echo "Please follow the instructions to increase amount of resources available:"
+ echo " https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#before-you-begin"
+ fi
+ mkdir -p /sources/logs /sources/dags /sources/plugins
+ chown -R "${AIRFLOW_UID}:${AIRFLOW_GID}" /sources/{logs,dags,plugins}
+ exec /entrypoint airflow version
environment:
<<: *airflow-common-env
_AIRFLOW_DB_UPGRADE: 'true'
_AIRFLOW_WWW_USER_CREATE: 'true'
_AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
_AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
+ user: "0:${AIRFLOW_GID:-0}"
+ volumes:
+ - .:/sources
flower:
<<: *airflow-common
diff --git a/docs/apache-airflow/start/docker.rst b/docs/apache-airflow/start/docker.rst
index 78d5c60..d37f33b 100644
--- a/docs/apache-airflow/start/docker.rst
+++ b/docs/apache-airflow/start/docker.rst
@@ -65,9 +65,6 @@ Some directories in the container are mounted, which means that their contents a
This file uses the latest Airflow image (`apache/airflow <https://hub.docker.com/r/apache/airflow>`__).
If you need install a new Python library or system library, you can :doc:`build your image <docker-stack:index>`.
-.. _initializing_docker_compose_environment:
-
-
Using custom images
===================
@@ -80,6 +77,8 @@ to rebuild the images on-the-fly when you run other ``docker-compose`` commands.
Examples of how you can extend the image with custom providers, python packages,
apt packages and more can be found in :doc:`Building the image <docker-stack:build>`.
+.. _initializing_docker_compose_environment:
+
Initializing Environment
========================
@@ -89,7 +88,7 @@ On **Linux**, the mounted volumes in container use the native Linux filesystem u
.. code-block:: bash
- mkdir ./dags ./logs ./plugins
+ mkdir -p ./dags ./logs ./plugins
echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env
See :ref:`Docker Compose environment variables <docker-compose-env-variables>`
@@ -111,6 +110,22 @@ After initialization is complete, you should see a message like below.
The account created has the login ``airflow`` and the password ``airflow``.
+Cleaning-up the environment
+===========================
+
+The docker-compose we prepare is a "Quick-start" one. It is not intended to be used in production
+and it has a number of caveats - one of them that the best way to recover from any problem is to clean it
+up and restart from the scratch.
+
+The best way to do it is to:
+
+* Run ``docker-compose down --volumes --remove-orphans`` command in the directory you downloaded the
+ ``docker-compose.yaml`` file
+* remove the whole directory where you downloaded the ``docker-compose.yaml`` file
+ ``rm -rf '<DIRECTORY>'``
+* re-download the ``docker-compose.yaml`` file
+* re-start following the instructions from the very beginning in this guide
+
Running Airflow
===============