You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ep...@apache.org on 2022/10/18 15:08:35 UTC
[airflow] branch v2-4-test updated (5bf2850490 -> 0196f1c0ab)
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a change to branch v2-4-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
from 5bf2850490 Add separate error handler for 405(Method not allowed) errors (#26880)
new 5c073b52f6 Respect "common" options value in breeze sub-commands. (#26264)
new 56ba1eb715 Update CI documentation, renaming runs to "Canary" (#26151)
new e574aa78f9 Create a workflow to sync the list of authors (#26275)
new 6cb03bf216 Fix static checks from sync_authors script (#26365)
new 26d136cec9 Actually run the "quick image build" (#26351)
new 7c7ce3c7bf Use the full path to the workflow being edited (#26386)
new d190ea7973 Fix params reference for default_constraints_branch (#26406)
new a4e2037f24 Attempt to create k8s venv without constraints if constraint build fails (#26407)
new 6d2b39470f Credentials are needed in order to push branch (#26413)
new 79e017f558 Manual triggering of author sync workflow (#26414)
new e271d16080 Fix problem with building providers where constraints are conflicting (#26420)
new bf70e1e6ed Add workflows permission (#26416)
new 0196f1c0ab Revert "Add workflows permission (#26416)" (#26422)
The 13 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/build-images.yml | 2 -
.github/workflows/ci.yml | 64 ++---
.github/workflows/sync_authors.yml | 65 +++++
.pre-commit-config.yaml | 1 +
CI.rst | 308 ++++++++-------------
CI_DIAGRAMS.md | 8 +-
.../src/airflow_breeze/commands/ci_commands.py | 4 +-
dev/breeze/src/airflow_breeze/global_constants.py | 6 -
.../src/airflow_breeze/utils/common_options.py | 30 +-
.../airflow_breeze/utils/docker_command_utils.py | 3 +-
.../src/airflow_breeze/utils/kubernetes_utils.py | 60 ++--
dev/breeze/tests/test_pr_info.py | 16 +-
scripts/ci/docker-compose/_docker.env | 1 -
scripts/ci/docker-compose/base.yml | 1 -
scripts/ci/docker-compose/devcontainer.env | 1 -
scripts/ci/libraries/_all_libs.sh | 2 -
scripts/ci/libraries/_initialization.sh | 14 +-
scripts/ci/libraries/_md5sum.sh | 154 -----------
scripts/ci/runners/sync_authors.py | 73 +++++
scripts/in_container/_in_container_utils.sh | 18 ++
20 files changed, 396 insertions(+), 435 deletions(-)
create mode 100644 .github/workflows/sync_authors.yml
delete mode 100644 scripts/ci/libraries/_md5sum.sh
create mode 100644 scripts/ci/runners/sync_authors.py
[airflow] 11/13: Fix problem with building providers where constraints are conflicting (#26420)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-4-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e271d16080ef5a5c0fcc765860e9edb04f9677ae
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Thu Sep 15 17:37:08 2022 +0200
Fix problem with building providers where constraints are conflicting (#26420)
When constraints are conflicting with Airflow's requirement,
the provider build fails. This change makes it retry the installation
without constraints in such case.
(cherry picked from commit 954a5510e5324cb78c9b386c750eb4168f1c8f53)
---
scripts/in_container/_in_container_utils.sh | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/scripts/in_container/_in_container_utils.sh b/scripts/in_container/_in_container_utils.sh
index 7cb8837112..3634d1dea8 100644
--- a/scripts/in_container/_in_container_utils.sh
+++ b/scripts/in_container/_in_container_utils.sh
@@ -202,8 +202,17 @@ function install_airflow_from_wheel() {
if [[ ${constraints_reference} == "none" ]]; then
pip install "${airflow_package}${extras}"
else
+ set +e
pip install "${airflow_package}${extras}" --constraint \
"https://raw.githubusercontent.com/apache/airflow/${constraints_reference}/constraints-${PYTHON_MAJOR_MINOR_VERSION}.txt"
+ res=$?
+ set -e
+ if [[ ${res} != "0" ]]; then
+ >&2 echo
+ >&2 echo "WARNING! Could not install provider packages with constraints, falling back to no-constraints mode"
+ >&2 echo
+ pip install "${airflow_package}${extras}"
+ fi
fi
}
@@ -232,8 +241,17 @@ function install_airflow_from_sdist() {
if [[ ${constraints_reference} == "none" ]]; then
pip install "${airflow_package}${extras}"
else
+ set +e
pip install "${airflow_package}${extras}" --constraint \
"https://raw.githubusercontent.com/apache/airflow/${constraints_reference}/constraints-${PYTHON_MAJOR_MINOR_VERSION}.txt"
+ res=$?
+ set -e
+ if [[ ${res} != "0" ]]; then
+ >&2 echo
+ >&2 echo "WARNING! Could not install provider packages with constraints, falling back to no-constraints mode"
+ >&2 echo
+ pip install "${airflow_package}${extras}"
+ fi
fi
}
[airflow] 13/13: Revert "Add workflows permission (#26416)" (#26422)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-4-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 0196f1c0ab3122849128922dbddddd2358e0b368
Author: Ash Berlin-Taylor <as...@firemirror.com>
AuthorDate: Fri Sep 16 08:35:28 2022 +0100
Revert "Add workflows permission (#26416)" (#26422)
This reverts commit 2382c12cc3aa5d819fd089c73e62f8849a567a0a.
(cherry picked from commit 7a3a752c64a0f96f0627f08061e01cbc71494df4)
---
.github/workflows/sync_authors.yml | 1 -
1 file changed, 1 deletion(-)
diff --git a/.github/workflows/sync_authors.yml b/.github/workflows/sync_authors.yml
index d136d7063d..5ef30ba9c4 100644
--- a/.github/workflows/sync_authors.yml
+++ b/.github/workflows/sync_authors.yml
@@ -29,7 +29,6 @@ on: # yamllint disable-line rule:truthy
permissions:
contents: write
pull-requests: write
- workflows: write
jobs:
sync:
[airflow] 09/13: Credentials are needed in order to push branch (#26413)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-4-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6d2b39470fe609c277d951efd18fbd9c55f5c2ea
Author: Mark Norman Francis <no...@201created.com>
AuthorDate: Thu Sep 15 10:11:41 2022 +0100
Credentials are needed in order to push branch (#26413)
(cherry picked from commit 9115488a68f7c3112f0c4943c32d318a8222cf96)
---
.github/workflows/sync_authors.yml | 2 --
1 file changed, 2 deletions(-)
diff --git a/.github/workflows/sync_authors.yml b/.github/workflows/sync_authors.yml
index 8405292859..52cb6db782 100644
--- a/.github/workflows/sync_authors.yml
+++ b/.github/workflows/sync_authors.yml
@@ -35,8 +35,6 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@v3
- with:
- persist-credentials: false
- name: Python
uses: actions/setup-python@v4
[airflow] 07/13: Fix params reference for default_constraints_branch (#26406)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-4-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d190ea79730b6ccaddf8f13387ba858c07fad355
Author: Gabriel Machado <ga...@hotmail.com>
AuthorDate: Wed Sep 14 23:56:14 2022 +0200
Fix params reference for default_constraints_branch (#26406)
Fixes breeze commands that depends on the default constraints branch,
such as the generated diff for
`breeze release-management generate-constraints`
(cherry picked from commit e458eb689d70e4857cdc3cdaaaafd4bfdbcc8ed7)
---
dev/breeze/src/airflow_breeze/utils/docker_command_utils.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py b/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
index f276051008..868499611a 100644
--- a/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
+++ b/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
@@ -610,7 +610,7 @@ DERIVE_ENV_VARIABLES_FROM_ATTRIBUTES = {
"AIRFLOW_CI_IMAGE": "airflow_image_name",
"AIRFLOW_CI_IMAGE_WITH_TAG": "airflow_image_name_with_tag",
"AIRFLOW_EXTRAS": "airflow_extras",
- "DEFAULT_CONSTRAINTS_BRANCH": "default-constraints-branch",
+ "DEFAULT_CONSTRAINTS_BRANCH": "default_constraints_branch",
"AIRFLOW_CONSTRAINTS_MODE": "airflow_constraints_mode",
"AIRFLOW_CONSTRAINTS_REFERENCE": "airflow_constraints_reference",
"AIRFLOW_IMAGE_KUBERNETES": "airflow_image_kubernetes",
[airflow] 05/13: Actually run the "quick image build" (#26351)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-4-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 26d136cec9bd6108cfaea03ed69939d0c9c9346b
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Wed Sep 14 14:38:18 2022 +0200
Actually run the "quick image build" (#26351)
The check does not use matrix build but we left the check for
matrix so the check was always skipped.
(cherry picked from commit c89e92f63aca7b1b008604883ce08c0ce032c80b)
---
.github/workflows/ci.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 742583271c..d8849e2c4f 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -339,7 +339,7 @@ jobs:
if: needs.build-info.outputs.canary-run == 'true'
- name: "Check that image builds quickly"
run: breeze shell --max-time 120
- if: matrix.platform == 'linux/amd64' && needs.build-info.outputs.canary-run == 'true'
+ if: needs.build-info.outputs.canary-run == 'true'
- name: "Fix ownership"
run: breeze ci fix-ownership
if: always() && needs.build-info.outputs.canary-run == 'true'
[airflow] 04/13: Fix static checks from sync_authors script (#26365)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-4-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6cb03bf216d7b9fff1d83dc973af22177f30199f
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Tue Sep 13 17:52:21 2022 +0200
Fix static checks from sync_authors script (#26365)
The #26275 introduced static check problem (isort). This PR fixes it
(cherry picked from commit aa0118b0d2a4085f80663b14c1d58b54cab81319)
---
scripts/ci/runners/sync_authors.py | 2 --
1 file changed, 2 deletions(-)
diff --git a/scripts/ci/runners/sync_authors.py b/scripts/ci/runners/sync_authors.py
index 6df09305a5..8fc9717e5b 100644
--- a/scripts/ci/runners/sync_authors.py
+++ b/scripts/ci/runners/sync_authors.py
@@ -22,7 +22,6 @@ import re
import requests
import toml
-
# The list of users in the 'build-info' job looks like:
#
# contains(fromJSON('[
@@ -53,7 +52,6 @@ authors = authors[:-2]
with open('ci.yml') as handle:
-
new_ci = re.sub(
r'''
^
[airflow] 12/13: Add workflows permission (#26416)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-4-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit bf70e1e6ed3960f8e86e5955b90d7768ebc92faa
Author: Mark Norman Francis <no...@201created.com>
AuthorDate: Thu Sep 15 20:00:54 2022 +0100
Add workflows permission (#26416)
Co-authored-by: Jed Cunningham <66...@users.noreply.github.com>
(cherry picked from commit 2382c12cc3aa5d819fd089c73e62f8849a567a0a)
---
.github/workflows/sync_authors.yml | 1 +
1 file changed, 1 insertion(+)
diff --git a/.github/workflows/sync_authors.yml b/.github/workflows/sync_authors.yml
index 5ef30ba9c4..d136d7063d 100644
--- a/.github/workflows/sync_authors.yml
+++ b/.github/workflows/sync_authors.yml
@@ -29,6 +29,7 @@ on: # yamllint disable-line rule:truthy
permissions:
contents: write
pull-requests: write
+ workflows: write
jobs:
sync:
[airflow] 08/13: Attempt to create k8s venv without constraints if constraint build fails (#26407)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-4-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit a4e2037f2404e628d8c8257bfa035d2b543ea2b0
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Thu Sep 15 08:24:06 2022 +0200
Attempt to create k8s venv without constraints if constraint build fails (#26407)
There is an issue with trying to install the k8s env where the
current requirements conflict with constraints. The k8s env
creation is failing in this case.
This PR will not fail hard when the constraint-build fails, instead
it will attempt to build the venv without constraints. This will work
in vast majority of cases.
(cherry picked from commit 6f390fa3c2b5d56de1454f1ba802a7b81853142e)
---
.../src/airflow_breeze/utils/kubernetes_utils.py | 60 ++++++++++++++--------
1 file changed, 40 insertions(+), 20 deletions(-)
diff --git a/dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py b/dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py
index 31bf7f6e45..353564b569 100644
--- a/dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py
+++ b/dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py
@@ -305,6 +305,38 @@ def _requirements_changed() -> bool:
return False
+def _install_packages_in_k8s_virtualenv(dry_run: bool, verbose: bool, with_constraints: bool):
+ install_command = [
+ str(PYTHON_BIN_PATH),
+ "-m",
+ "pip",
+ "install",
+ "-r",
+ str(K8S_REQUIREMENTS.resolve()),
+ ]
+ if with_constraints:
+ install_command.extend(
+ [
+ "--constraint",
+ f"https://raw.githubusercontent.com/apache/airflow/{DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH}/"
+ f"constraints-{sys.version_info.major}.{sys.version_info.minor}.txt",
+ ]
+ )
+ install_packages_result = run_command(
+ install_command,
+ verbose=verbose,
+ dry_run=dry_run,
+ check=False,
+ capture_output=True,
+ )
+ if install_packages_result.returncode != 0:
+ get_console().print(
+ f'[error]Error when updating pip to {PIP_VERSION}:[/]\n'
+ f'{install_packages_result.stdout}\n{install_packages_result.stderr}'
+ )
+ return install_packages_result
+
+
def create_virtualenv(force: bool, verbose: bool, dry_run: bool) -> RunCommandResult:
K8S_CLUSTERS_PATH.mkdir(parents=True, exist_ok=True)
if not force and not _requirements_changed():
@@ -354,29 +386,17 @@ def create_virtualenv(force: bool, verbose: bool, dry_run: bool) -> RunCommandRe
)
return pip_reinstall_result
get_console().print(f'[info]Installing necessary packages in {K8S_ENV_PATH}')
- install_packages_result = run_command(
- [
- str(PYTHON_BIN_PATH),
- "-m",
- "pip",
- "install",
- "-r",
- str(K8S_REQUIREMENTS.resolve()),
- "--constraint",
- f"https://raw.githubusercontent.com/apache/airflow/{DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH}/"
- f"constraints-{sys.version_info.major}.{sys.version_info.minor}.txt",
- ],
- verbose=verbose,
- dry_run=dry_run,
- check=False,
- capture_output=True,
+
+ install_packages_result = _install_packages_in_k8s_virtualenv(
+ dry_run=dry_run, verbose=verbose, with_constraints=True
)
if install_packages_result.returncode != 0:
- get_console().print(
- f'[error]Error when updating pip to {PIP_VERSION}:[/]\n'
- f'{install_packages_result.stdout}\n{install_packages_result.stderr}'
+ # if the first installation fails, attempt to install it without constraints
+ install_packages_result = _install_packages_in_k8s_virtualenv(
+ dry_run=dry_run, verbose=verbose, with_constraints=False
)
- CACHED_K8S_REQUIREMENTS.write_text(K8S_REQUIREMENTS.read_text())
+ if install_packages_result.returncode == 0:
+ CACHED_K8S_REQUIREMENTS.write_text(K8S_REQUIREMENTS.read_text())
return install_packages_result
[airflow] 02/13: Update CI documentation, renaming runs to "Canary" (#26151)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-4-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 56ba1eb715f8e870dc6d2e7a92f0940e3fe6ab8b
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Fri Sep 9 17:56:50 2022 +0200
Update CI documentation, renaming runs to "Canary" (#26151)
For quite some time we did not have the name of the main builds
However more and more the "main" builds are used to provide early
warnigns for some problems:
* 3rd-party dependencies breaking our builds
* our own dependencies breaking the constraints
* building ARM images
* building breeze images quickly
* running complete matrix of tests
* finding flaky tests
So effectively, those main builds are really "Canary" builds - when
those builds are failing, they give us a chance to react quickly,
without affecting the regular PR builds.
This PR clarifies the meaning and reasoning for those builds
and introduces "Canary" name for them.
During related documentation review, it also turned out that a number
of old environment variables are not used any more (after the breeze
changing to Python) and this PR also removes them from documentation
and removes the variables from all the scripts (including removal of
some unused scripts)
The new documentation also mentions somethign that we've learned
recently - that in case you use Breeze in non-airflow workflows in
GitHub Actions, you need to override the variables through command
line parameters rather than through environment variables, because
GitHub actions treats GITHUB_* variables as immutable for security.
(cherry picked from commit eb03959e437e11891b8c3696b76f664a991a37a4)
---
.github/workflows/build-images.yml | 2 -
.github/workflows/ci.yml | 52 ++--
CI.rst | 308 ++++++++-------------
CI_DIAGRAMS.md | 8 +-
.../src/airflow_breeze/commands/ci_commands.py | 4 +-
dev/breeze/src/airflow_breeze/global_constants.py | 6 -
.../airflow_breeze/utils/docker_command_utils.py | 1 -
dev/breeze/tests/test_pr_info.py | 16 +-
scripts/ci/docker-compose/_docker.env | 1 -
scripts/ci/docker-compose/base.yml | 1 -
scripts/ci/docker-compose/devcontainer.env | 1 -
scripts/ci/libraries/_all_libs.sh | 2 -
scripts/ci/libraries/_initialization.sh | 14 +-
scripts/ci/libraries/_md5sum.sh | 154 -----------
14 files changed, 164 insertions(+), 406 deletions(-)
diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml
index 5c3be9270f..f8433ab92e 100644
--- a/.github/workflows/build-images.yml
+++ b/.github/workflows/build-images.yml
@@ -26,8 +26,6 @@ permissions:
env:
MOUNT_SELECTED_LOCAL_SOURCES: "false"
ANSWER: "yes"
- CHECK_IMAGE_FOR_REBUILD: "true"
- SKIP_CHECK_REMOTE_IMAGE: "true"
DB_RESET: "true"
VERBOSE: "true"
GITHUB_REPOSITORY: ${{ github.repository }}
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 51fd6ac9b0..ece19a4c8f 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -31,8 +31,6 @@ permissions:
env:
MOUNT_SELECTED_LOCAL_SOURCES: "false"
ANSWER: "yes"
- CHECK_IMAGE_FOR_REBUILD: "true"
- SKIP_CHECK_REMOTE_IMAGE: "true"
DB_RESET: "true"
VERBOSE: "true"
GITHUB_REPOSITORY: ${{ github.repository }}
@@ -168,7 +166,7 @@ jobs:
in-workflow-build: ${{ steps.source-run-info.outputs.in-workflow-build }}
build-job-description: ${{ steps.source-run-info.outputs.build-job-description }}
runs-on: ${{ steps.source-run-info.outputs.runs-on }}
- merge-run: ${{ steps.source-run-info.outputs.merge-run }}
+ canary-run: ${{ steps.source-run-info.outputs.canary-run }}
run-coverage: ${{ steps.source-run-info.outputs.run-coverage }}
steps:
- name: Cleanup repo
@@ -261,27 +259,27 @@ jobs:
steps:
- name: Cleanup repo
run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
- if: needs.build-info.outputs.merge-run == 'true'
+ if: needs.build-info.outputs.canary-run == 'true'
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v3
with:
persist-credentials: false
- if: needs.build-info.outputs.merge-run == 'true'
+ if: needs.build-info.outputs.canary-run == 'true'
- name: "Setup python"
uses: actions/setup-python@v4
with:
python-version: ${{ needs.build-info.outputs.default-python-version }}
cache: 'pip'
cache-dependency-path: ./dev/breeze/setup*
- if: needs.build-info.outputs.merge-run == 'true'
+ if: needs.build-info.outputs.canary-run == 'true'
- run: ./scripts/ci/install_breeze.sh
- if: needs.build-info.outputs.merge-run == 'true'
+ if: needs.build-info.outputs.canary-run == 'true'
- name: "Free space"
run: breeze ci free-space
- if: needs.build-info.outputs.merge-run == 'true'
+ if: needs.build-info.outputs.canary-run == 'true'
- name: "Start ARM instance"
run: ./scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh
- if: matrix.platform == 'linux/arm64' && needs.build-info.outputs.merge-run == 'true'
+ if: matrix.platform == 'linux/arm64' && needs.build-info.outputs.canary-run == 'true'
- name: "Push CI cache ${{ matrix.platform }}"
run: >
breeze ci-image build
@@ -290,23 +288,23 @@ jobs:
--run-in-parallel
--force-build
--platform ${{ matrix.platform }}
- if: needs.build-info.outputs.merge-run == 'true'
+ if: needs.build-info.outputs.canary-run == 'true'
- name: "Push CI latest image ${{ matrix.platform }}"
run: >
breeze ci-image build
--tag-as-latest --push --run-in-parallel --platform ${{ matrix.platform }}
# We only push "amd" image as it is really only needed for any kind of automated builds in CI
# and currently there is not an easy way to make multi-platform image from two separate builds
- if: matrix.platform == 'linux/amd64' && needs.build-info.outputs.merge-run == 'true'
+ if: matrix.platform == 'linux/amd64' && needs.build-info.outputs.canary-run == 'true'
- name: "Stop ARM instance"
run: ./scripts/ci/images/ci_stop_arm_instance.sh
- if: always() && matrix.platform == 'linux/arm64' && needs.build-info.outputs.merge-run == 'true'
+ if: always() && matrix.platform == 'linux/arm64' && needs.build-info.outputs.canary-run == 'true'
- name: "Clean docker cache for ${{ matrix.platform }}"
run: docker system prune --all --force
- if: matrix.platform == 'linux/amd64' && needs.build-info.outputs.merge-run == 'true'
+ if: matrix.platform == 'linux/amd64' && needs.build-info.outputs.canary-run == 'true'
- name: "Fix ownership"
run: breeze ci fix-ownership
- if: always() && needs.build-info.outputs.merge-run == 'true'
+ if: always() && needs.build-info.outputs.canary-run == 'true'
# Check that after earlier cache push, breeze command will build quickly
chcek-that-image-builds-quicklly:
@@ -323,30 +321,30 @@ jobs:
steps:
- name: Cleanup repo
run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
- if: needs.build-info.outputs.merge-run == 'true'
+ if: needs.build-info.outputs.canary-run == 'true'
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v3
with:
persist-credentials: false
- if: needs.build-info.outputs.merge-run == 'true'
+ if: needs.build-info.outputs.canary-run == 'true'
- name: "Setup python"
uses: actions/setup-python@v4
with:
python-version: ${{ needs.build-info.outputs.default-python-version }}
cache: 'pip'
cache-dependency-path: ./dev/breeze/setup*
- if: needs.build-info.outputs.merge-run == 'true'
+ if: needs.build-info.outputs.canary-run == 'true'
- run: ./scripts/ci/install_breeze.sh
- if: needs.build-info.outputs.merge-run == 'true'
+ if: needs.build-info.outputs.canary-run == 'true'
- name: "Free space"
run: breeze ci free-space
- if: needs.build-info.outputs.merge-run == 'true'
+ if: needs.build-info.outputs.canary-run == 'true'
- name: "Check that image builds quickly"
run: breeze shell --max-time 120
- if: matrix.platform == 'linux/amd64' && needs.build-info.outputs.merge-run == 'true'
+ if: matrix.platform == 'linux/amd64' && needs.build-info.outputs.canary-run == 'true'
- name: "Fix ownership"
run: breeze ci fix-ownership
- if: always() && needs.build-info.outputs.merge-run == 'true'
+ if: always() && needs.build-info.outputs.canary-run == 'true'
build-ci-images:
permissions:
@@ -1636,20 +1634,20 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- name: "Set constraints branch name"
id: constraints-branch
run: ./scripts/ci/constraints/ci_branch_constraints.sh
- if: needs.build-info.outputs.merge-run == 'true'
+ if: needs.build-info.outputs.canary-run == 'true'
- name: Checkout ${{ steps.constraints-branch.outputs.branch }}
uses: actions/checkout@v3
- if: needs.build-info.outputs.merge-run == 'true'
+ if: needs.build-info.outputs.canary-run == 'true'
with:
path: "repo"
ref: ${{ steps.constraints-branch.outputs.branch }}
persist-credentials: false
- name: "Commit changed constraint files for ${{needs.build-info.outputs.python-versions}}"
run: ./scripts/ci/constraints/ci_commit_constraints.sh
- if: needs.build-info.outputs.merge-run == 'true'
+ if: needs.build-info.outputs.canary-run == 'true'
- name: "Push changes"
uses: ./.github/actions/github-push-action
- if: needs.build-info.outputs.merge-run == 'true'
+ if: needs.build-info.outputs.canary-run == 'true'
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
branch: ${{ steps.constraints-branch.outputs.branch }}
@@ -1672,7 +1670,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- build-info
- constraints
- docs
- if: needs.build-info.outputs.merge-run == 'true'
+ if: needs.build-info.outputs.canary-run == 'true'
strategy:
fail-fast: false
matrix:
@@ -1776,7 +1774,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
if: >
needs.build-info.outputs.upgrade-to-newer-dependencies != 'false' &&
needs.build-info.outputs.in-workflow-build == 'true' &&
- needs.build-info.outputs.merge-run != 'true'
+ needs.build-info.outputs.canary-run != 'true'
steps:
- name: Cleanup repo
run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
diff --git a/CI.rst b/CI.rst
index 1b86a43b41..ccabd58463 100644
--- a/CI.rst
+++ b/CI.rst
@@ -21,7 +21,7 @@ CI Environment
==============
Continuous Integration is important component of making Apache Airflow robust and stable. We are running
-a lot of tests for every pull request, for main and v2-*-test branches and regularly as CRON jobs.
+a lot of tests for every pull request, for main and v2-*-test branches and regularly as scheduled jobs.
Our execution environment for CI is `GitHub Actions <https://github.com/features/actions>`_. GitHub Actions
(GA) are very well integrated with GitHub code and Workflow and it has evolved fast in 2019/202 to become
@@ -84,6 +84,13 @@ We use `GitHub Container Registry <https://docs.github.com/en/packages/guides/ab
``GITHUB_TOKEN`` is needed to push to the registry and we configured scopes of the tokens in our jobs
to be able to write to the registry.
+The latest cache is kept as ``:cache-amd64`` and ``:cache-arm64`` tagged cache (suitable for
+``--cache-from`` directive of buildx - it contains metadata and cache for all segments in the image,
+and cache is separately kept for different platform.
+
+The ``latest`` images of CI and PROD are ``amd64`` only images for CI, because there is no very easy way
+to push multiplatform images without merging the manifests and it is not really needed nor used for cache.
+
Locally replicating CI failures
-------------------------------
@@ -151,13 +158,6 @@ You can use those variables when you try to reproduce the build locally.
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| Force variables |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
-| ``FORCE_BUILD_IMAGES`` | false | false | false | Forces building images. This is generally not |
-| | | | | very useful in CI as in CI environment image |
-| | | | | is built or pulled only once, so there is no |
-| | | | | need to set the variable to true. For local |
-| | | | | builds it forces rebuild, regardless if it |
-| | | | | is determined to be needed. |
-+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| ``ANSWER`` | | yes | yes | This variable determines if answer to questions |
| | | | | during the build process should be |
| | | | | automatically given. For local development, |
@@ -167,17 +167,6 @@ You can use those variables when you try to reproduce the build locally.
| | | | | the user has to answer but in the CI |
| | | | | environment, we force "yes" answer. |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
-| ``SKIP_CHECK_REMOTE_IMAGE`` | false | true | true | Determines whether we check if remote image |
-| | | | | is "fresher" than the current image. |
-| | | | | When doing local breeze runs we try to |
-| | | | | determine if it will be faster to rebuild |
-| | | | | the image or whether the image should be |
-| | | | | pulled first from the cache because it has |
-| | | | | been rebuilt. This is slightly experimental |
-| | | | | feature and will be improved in the future |
-| | | | | as the current mechanism does not always |
-| | | | | work properly. |
-+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| Host variables |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| ``HOST_USER_ID`` | | | | User id of the host user. |
@@ -265,107 +254,38 @@ You can use those variables when you try to reproduce the build locally.
| | | | | \* true in case of direct pushes and |
| | | | | scheduled builds |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
-| ``CHECK_IMAGE_FOR_REBUILD`` | true | true | true\* | Determines whether attempt should be |
-| | | | | made to rebuild the CI image with latest |
-| | | | | sources. It is true by default for |
-| | | | | local builds, however it is set to |
-| | | | | true in case we know that the image |
-| | | | | we pulled or built already contains |
-| | | | | the right sources. In such case we |
-| | | | | should set it to false, especially |
-| | | | | in case our local sources are not the |
-| | | | | ones we intend to use (for example |
-| | | | | when ``--image-tag`` is used |
-| | | | | in Breeze. |
-| | | | | |
-| | | | | In CI jobs it is set to true |
-| | | | | in case of the ``Build Images`` |
-| | | | | workflow or when |
-| | | | | waiting for images is disabled |
-| | | | | in the "Tests" workflow. |
-| | | | | |
-| | | | | \* if waiting for images the variable is set |
-| | | | | to false automatically. |
-+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
-| ``SKIP_BUILDING_PROD_IMAGE`` | false | false | false\* | Determines whether we should skip building |
-| | | | | the PROD image with latest sources. |
-| | | | | It is set to false, but in deploy app for |
-| | | | | kubernetes step it is set to "true", because at |
-| | | | | this stage we know we have good image build or |
-| | | | | pulled. |
-| | | | | |
-| | | | | \* set to true in "Deploy App to Kubernetes" |
-| | | | | to false automatically. |
-+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
Running CI Jobs locally
=======================
-The scripts and configuration files for CI jobs are all in ``scripts/ci`` - so that in the
-``pull_request_target`` target workflow, we can copy those scripts from the ``main`` branch and use them
-regardless of the changes done in the PR. This way we are kept safe from PRs injecting code into the builds.
-
-* ``build_airflow`` - builds airflow packages
-* ``constraints`` - scripts to build and publish latest set of valid constraints
-* ``docs`` - scripts to build documentation
-* ``images`` - scripts to build and push CI and PROD images
-* ``kubernetes`` - scripts to setup kubernetes cluster, deploy airflow and run kubernetes tests with it
-* ``openapi`` - scripts to run openapi generation
-* ``pre_commit`` - scripts to run pre-commit checks
-* ``provider_packages`` - scripts to build and test provider packages
-* ``static_checks`` - scripts to run static checks manually
-* ``testing`` - scripts that run unit and integration tests
-* ``tools`` - scripts that can be used for various clean-up and preparation tasks
-
-Common libraries of functions for all the scripts can be found in ``libraries`` folder. The ``dockerfiles``,
-``mysql.d``, ``openldap``, ``spectral_rules`` folders contains DockerFiles and configuration of integrations
-needed to run tests.
-
-For detailed use of those scripts you can refer to ``.github/workflows/`` - those scripts are used
-by the CI workflows of ours. There are some variables that you can set to change the behaviour of the
-scripts.
-
-The default values are "sane" you can change them to interact with your own repositories or registries.
-Note that you need to set "CI" variable to true in order to get the same results as in CI.
-
-+------------------------------+----------------------+-----------------------------------------------------+
-| Variable | Default | Comment |
-+==============================+======================+=====================================================+
-| CI | ``false`` | If set to "true", we simulate behaviour of |
-| | | all scripts as if they are in CI environment |
-+------------------------------+----------------------+-----------------------------------------------------+
-| CI_TARGET_REPO | ``apache/airflow`` | Target repository for the CI job. Used to |
-| | | compare incoming changes from PR with the target. |
-+------------------------------+----------------------+-----------------------------------------------------+
-| CI_TARGET_BRANCH | ``main`` | Target branch where the PR should land. Used to |
-| | | compare incoming changes from PR with the target. |
-+------------------------------+----------------------+-----------------------------------------------------+
-| CI_BUILD_ID | ``0`` | Unique id of the build that is kept across re runs |
-| | | (for GitHub actions it is ``GITHUB_RUN_ID``) |
-+------------------------------+----------------------+-----------------------------------------------------+
-| CI_JOB_ID | ``0`` | Unique id of the job - used to produce unique |
-| | | artifact names. |
-+------------------------------+----------------------+-----------------------------------------------------+
-| CI_EVENT_TYPE | ``pull_request`` | Type of the event. It can be one of |
-| | | [``pull_request``, ``pull_request_target``, |
-| | | ``schedule``, ``push``] |
-+------------------------------+----------------------+-----------------------------------------------------+
-| CI_REF | ``refs/head/main`` | Branch in the source repository that is used to |
-| | | make the pull request. |
-+------------------------------+----------------------+-----------------------------------------------------+
+All our CI jobs are executed via ``breeze`` commands. You can replicate exactly what our CI is doing
+by running the sequence of corresponding ``breeze`` command. Make sure however that you look at both:
+* flags passed to ``breeze`` commands
+* environment variables used when ``breeze`` command is run - this is useful when we want
+ to set a common flag for all ``breeze`` commands in the same job or even the whole workflow. For
+ example ``VERBOSE`` variable is set to ``true`` for all our workflows so that more detailed information
+ about internal commands executed in CI is printed.
+
+In the output of the CI jobs, you will find both - the flags passed and environment variables set.
GitHub Registry Variables
=========================
-Our CI uses GitHub Registry to pull and push images to/from by default. You can use your own repo by changing
-``GITHUB_REPOSITORY`` and providing your own GitHub Username and Token.
+Our CI uses GitHub Registry to pull and push images to/from by default. Those variables are set automatically
+by GitHub Actions when you run Airflow workflows in your fork, so they should automatically use your
+own repository as GitHub Registry to build and keep the images as build image cache.
+
+The variables are automatically set in GitHub actions
+--------------------------------+---------------------------+----------------------------------------------+
| Variable | Default | Comment |
+================================+===========================+==============================================+
| GITHUB_REPOSITORY | ``apache/airflow`` | Prefix of the image. It indicates which. |
-| | | registry from GitHub to use |
+| | | registry from GitHub to use for image cache |
+| | | and to determine the name of the image. |
++--------------------------------+---------------------------+----------------------------------------------+
+| CONSTRAINTS_GITHUB_REPOSITORY | ``apache/airflow`` | Repository where constraints are stored |
+--------------------------------+---------------------------+----------------------------------------------+
| GITHUB_USERNAME | | Username to use to login to GitHub |
| | | |
@@ -373,37 +293,37 @@ Our CI uses GitHub Registry to pull and push images to/from by default. You can
| GITHUB_TOKEN | | Token to use to login to GitHub. |
| | | Only used when pushing images on CI. |
+--------------------------------+---------------------------+----------------------------------------------+
-| GITHUB_REGISTRY_PULL_IMAGE_TAG | ``latest`` | Pull this image tag. This is "latest" by |
-| | | default, can also be full-length commit SHA. |
-+--------------------------------+---------------------------+----------------------------------------------+
-| GITHUB_REGISTRY_PUSH_IMAGE_TAG | ``latest`` | Push this image tag. This is "latest" by |
-| | | default, can also be full-length commit SHA. |
-+--------------------------------+---------------------------+----------------------------------------------+
+
+The Variables beginning with ``GITHUB_`` cannot be overridden in GitHub Actions by the workflow.
+Those variables are set by GitHub Actions automatically and they are immutable. Therefore if
+you want to override them in your own CI workflow and use ``breeze``, you need to pass the
+values by corresponding ``breeze`` flags ``--github-repository``, ``--github-username``,
+``--github-token`` rather than by setting them as environment variables in your workflow.
+Unless you want to keep your own copy of constraints in orphaned ``constraints-*``
+branches, the ``CONSTRAINTS_GITHUB_REPOSITORY`` should remain ``apache/airflow``, regardless in which
+repository the CI job is run.
+
+One of the variables you might want to override in your own GitHub Actions workflow when using ``breeze`` is
+``--github-repository`` - you might want to force it to ``apache/airflow``, because then the cache from
+``apache/airflow`` repository will be used and your builds will be much faster.
+
+Example command to build your CI image efficiently in your own CI workflow:
+
+.. code-block:: bash
+
+ # GITHUB_REPOSITORY is set automatically in Github Actions so we need to override it with flag
+ #
+ breeze ci-image build --github-repository apache/airflow --python 3.10
+ docker tag ghcr.io/apache/airflow/main/ci/python3.10 your-image-name:tag
+
Authentication in GitHub Registry
=================================
We are using GitHub Container Registry as cache for our images. Authentication uses GITHUB_TOKEN mechanism.
Authentication is needed for pushing the images (WRITE) only in "push", "pull_request_target" workflows.
+When you are running the CI jobs in GitHub Actions, GITHUB_TOKEN is set automatically by the actions.
-CI Architecture
-===============
-
-The following components are part of the CI infrastructure
-
-* **Apache Airflow Code Repository** - our code repository at https://github.com/apache/airflow
-* **Apache Airflow Forks** - forks of the Apache Airflow Code Repository from which contributors make
- Pull Requests
-* **GitHub Actions** - (GA) UI + execution engine for our jobs
-* **GA CRON trigger** - GitHub Actions CRON triggering our jobs
-* **GA Workers** - virtual machines running our jobs at GitHub Actions (max 20 in parallel)
-* **GitHub Image Registry** - image registry used as build cache for CI jobs.
- It is at https://ghcr.io/apache/airflow
-* **DockerHub Image Registry** - image registry used to pull base Python images and (manually) publish
- the released Production Airflow images. It is at https://dockerhub.com/apache/airflow
-* **Official Images** (future) - these are official images that are prominently visible in DockerHub.
- We aim our images to become official images so that you will be able to pull them
- with ``docker pull apache-airflow``
CI run types
============
@@ -411,6 +331,12 @@ CI run types
The following CI Job run types are currently run for Apache Airflow (run by ci.yaml workflow)
and each of the run types has different purpose and context.
+Besides the regular "PR" runs we also have "Canary" runs that are able to detect most of the
+problems that might impact regular PRs early, without necessarily failing all PRs when those
+problems happen. This allows to provide much more stable environment for contributors, who
+contribute their PR, while giving a chance to maintainers to react early on problems that
+need reaction, when the "canary" builds fail.
+
Pull request run
----------------
@@ -426,27 +352,37 @@ CI, Production Images as well as base Python images that are also cached in the
Also for those builds we only execute Python tests if important files changed (so for example if it is
"no-code" change, no tests will be executed.
-Direct Push/Merge Run
----------------------
+Regular PR builds run in a "stable" environment:
+
+* fixed set of constraints (constraints that passed the tests) - except the PRs that change dependencies
+* limited matrix and set of tests (determined by selective checks based on what changed in the PR)
+* no ARM image builds are build in the regular PRs
+* lower probability of flaky tests for non-committer PRs (public runners and less parallelism)
-Those runs are results of direct pushes done by the committers or as result of merge of a Pull Request
+Canary run
+----------
+
+Those runs are results of direct pushes done by the committers - basically merging of a Pull Request
by the committers. Those runs execute in the context of the Apache Airflow Code Repository and have also
write permission for GitHub resources (container registry, code repository).
+
The main purpose for the run is to check if the code after merge still holds all the assertions - like
-whether it still builds, all tests are green.
+whether it still builds, all tests are green. This is a "Canary" build that helps us to detect early
+problems with dependencies, image building, full matrix of tests in case they passed through selective checks.
This is needed because some of the conflicting changes from multiple PRs might cause build and test failures
after merge even if they do not fail in isolation. Also those runs are already reviewed and confirmed by the
committers so they can be used to do some housekeeping:
-- pushing most recent image build in the PR to the GitHub Container Registry (for caching)
+
+- pushing most recent image build in the PR to the GitHub Container Registry (for caching) including recent
+ Dockerfile changes and setup.py/setup.cfg changes (Early Cache)
+- test that image in ``breeze`` command builds quickly
+- run full matrix of tests to detect any tests that will be mistakenly missed in ``selective checks``
- upgrading to latest constraints and pushing those constraints if all tests succeed
- refresh latest Python base images in case new patch-level is released
The housekeeping is important - Python base images are refreshed with varying frequency (once every few months
usually but sometimes several times per week) with the latest security and bug fixes.
-Those patch level images releases can occasionally break Airflow builds (specifically Docker image builds
-based on those images) therefore in PRs we only use latest "good" Python image that we store in the
-GitHub Container Registry and those push requests will refresh the latest images if they changed.
Scheduled runs
--------------
@@ -522,53 +458,55 @@ Tests Workflow
This workflow is a regular workflow that performs all checks of Airflow code.
-+---------------------------+----------------------------------------------+-------+-------+------+
-| Job | Description | PR | Push | CRON |
-| | | | Merge | (1) |
-+===========================+==============================================+=======+=======+======+
-| Build info | Prints detailed information about the build | Yes | Yes | Yes |
-+---------------------------+----------------------------------------------+-------+-------+------+
-| Push early cache & images | Pushes early cache/images to GitHub Registry | - | Yes | Yes |
-+---------------------------+----------------------------------------------+-------+-------+------+
-| Test OpenAPI client gen | Tests if OpenAPIClient continues to generate | Yes | Yes | Yes |
-+---------------------------+----------------------------------------------+-------+-------+------+
-| UI tests | React UI tests for new Airflow UI | Yes | Yes | Yes |
-+---------------------------+----------------------------------------------+-------+-------+------+
-| WWW tests | React tests for current Airflow UI | Yes | Yes | Yes |
-+---------------------------+----------------------------------------------+-------+-------+------+
-| Test image building | Tests if PROD image build examples work | Yes | Yes | Yes |
-+---------------------------+----------------------------------------------+-------+-------+------+
-| CI Images | Waits for and verify CI Images (3) | Yes | Yes | Yes |
-+---------------------------+----------------------------------------------+-------+-------+------+
-| (Basic) Static checks | Performs static checks (full or basic) | Yes | Yes | Yes |
-+---------------------------+----------------------------------------------+-------+-------+------+
-| Build docs | Builds documentation | Yes | Yes | Yes |
-+---------------------------+----------------------------------------------+-------+-------+------+
-| Tests | Run all the Pytest tests for Python code | Yes(2)| Yes | Yes |
-+---------------------------+----------------------------------------------+-------+-------+------+
-| Tests provider packages | Tests if provider packages work | Yes | Yes | Yes |
-+---------------------------+----------------------------------------------+-------+-------+------+
-| Upload coverage | Uploads test coverage from all the tests | - | Yes | - |
-+---------------------------+----------------------------------------------+-------+-------+------+
-| PROD Images | Waits for and verify PROD Images (3) | Yes | Yes | Yes |
-+---------------------------+----------------------------------------------+-------+-------+------+
-| Tests Kubernetes | Run Kubernetes test | Yes(2)| Yes | Yes |
-+---------------------------+----------------------------------------------+-------+-------+------+
-| Constraints | Upgrade constraints to latest ones (4) | - | Yes | Yes |
-+---------------------------+----------------------------------------------+-------+-------+------+
-| Push cache & images | Pushes cache/images to GitHub Registry (4) | - | Yes | Yes |
-+---------------------------+----------------------------------------------+-------+-------+------+
-
-
-Comments:
-
- (1) CRON jobs builds images from scratch - to test if everything works properly for clean builds
- (2) The tests are run when the Trigger Tests job determine that important files change (this allows
- for example "no-code" changes to build much faster)
- (3) The jobs wait for CI images to be available.
- (4) PROD and CI cache & images are pushed as "latest" to GitHub Container registry and constraints are
- upgrade only if all tests are successful. The images are rebuilt in this step using constraints pushed
- in the previous step.
++-----------------------------+----------------------------------------------------------+---------+----------+-----------+
+| Job | Description | PR | Canary | Scheduled |
++=============================+==========================================================+=========+==========+===========+
+| Build info | Prints detailed information about the build | Yes | Yes | Yes |
++-----------------------------+----------------------------------------------------------+---------+----------+-----------+
+| Build CI/PROD images | Builds images in-workflow (not in the build images one) | - | Yes | Yes (1) |
++-----------------------------+----------------------------------------------------------+---------+----------+-----------+
+| Push early cache & images | Pushes early cache/images to GitHub Registry and test | - | Yes | - |
+| | speed of building breeze images from scratch | | | |
++-----------------------------+----------------------------------------------------------+---------+----------+-----------+
+| Test OpenAPI client gen | Tests if OpenAPIClient continues to generate | Yes | Yes | Yes |
++-----------------------------+----------------------------------------------------------+---------+----------+-----------+
+| UI tests | React UI tests for new Airflow UI | Yes | Yes | Yes |
++-----------------------------+----------------------------------------------------------+---------+----------+-----------+
+| WWW tests | React tests for current Airflow UI | Yes | Yes | Yes |
++-----------------------------+----------------------------------------------------------+---------+----------+-----------+
+| Test image building | Tests if PROD image build examples work | Yes | Yes | Yes |
++-----------------------------+----------------------------------------------------------+---------+----------+-----------+
+| CI Images | Waits for and verify CI Images (3) | Yes | Yes | Yes |
++-----------------------------+----------------------------------------------------------+---------+----------+-----------+
+| (Basic) Static checks | Performs static checks (full or basic) | Yes | Yes | Yes |
++-----------------------------+----------------------------------------------------------+---------+----------+-----------+
+| Build docs | Builds documentation | Yes | Yes | Yes |
++-----------------------------+----------------------------------------------------------+---------+----------+-----------+
+| Tests | Run all the Pytest tests for Python code | Yes(2) | Yes | Yes |
++-----------------------------+----------------------------------------------------------+---------+----------+-----------+
+| Tests provider packages | Tests if provider packages work | Yes | Yes | Yes |
++-----------------------------+----------------------------------------------------------+---------+----------+-----------+
+| Upload coverage | Uploads test coverage from all the tests | - | Yes | - |
++-----------------------------+----------------------------------------------------------+---------+----------+-----------+
+| PROD Images | Waits for and verify PROD Images (3) | Yes | Yes | Yes |
++-----------------------------+----------------------------------------------------------+---------+----------+-----------+
+| Tests Kubernetes | Run Kubernetes test | Yes(2) | Yes | Yes |
++-----------------------------+----------------------------------------------------------+---------+----------+-----------+
+| Constraints | Upgrade constraints to latest ones (4) | - | Yes | Yes |
++-----------------------------+----------------------------------------------------------+---------+----------+-----------+
+| Push cache & images | Pushes cache/images to GitHub Registry (4) | - | Yes | Yes |
++-----------------------------+----------------------------------------------------------+---------+----------+-----------+
+
+``(1)`` Scheduled jobs builds images from scratch - to test if everything works properly for clean builds
+
+``(2)`` The tests are run when the Trigger Tests job determine that important files change (this allows
+for example "no-code" changes to build much faster)
+
+``(3)`` The jobs wait for CI images to be available.
+
+``(4)`` PROD and CI cache & images are pushed as "latest" to GitHub Container registry and constraints are
+upgraded only if all tests are successful. The images are rebuilt in this step using constraints pushed
+in the previous step.
CodeQL scan
-----------
diff --git a/CI_DIAGRAMS.md b/CI_DIAGRAMS.md
index 878990645f..e19228a024 100644
--- a/CI_DIAGRAMS.md
+++ b/CI_DIAGRAMS.md
@@ -178,11 +178,11 @@ sequenceDiagram
deactivate Tests
```
-## Direct Push/Merge flow
+## Merge "Canary" run
```mermaid
sequenceDiagram
- Note over Airflow Repo: pull request
+ Note over Airflow Repo: push/merge
Note over Tests: push<br>[Write Token]
activate Airflow Repo
Airflow Repo -->> Tests: Trigger 'push'
@@ -199,6 +199,7 @@ sequenceDiagram
and
Note over Tests: Build CI Images<br>Use original constraints
Tests ->> GitHub Registry: Push CI Image Early cache + latest
+ Note over Tests: Test 'breeze' image build quickly
end
Tests ->> GitHub Registry: Push CI Images<br>[COMMIT_SHA]
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
@@ -257,7 +258,7 @@ sequenceDiagram
deactivate Tests
```
-## Scheduled build flow
+## Scheduled run
```mermaid
sequenceDiagram
@@ -279,6 +280,7 @@ sequenceDiagram
and
Note over Tests: Build CI Images<br>Use original constraints
Tests ->> GitHub Registry: Push CI Image Early cache + latest
+ Note over Tests: Test 'breeze' image build quickly
end
Tests ->> GitHub Registry: Push CI Images<br>[COMMIT_SHA]
GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
diff --git a/dev/breeze/src/airflow_breeze/commands/ci_commands.py b/dev/breeze/src/airflow_breeze/commands/ci_commands.py
index bddb96b36e..c5cd05c8b4 100644
--- a/dev/breeze/src/airflow_breeze/commands/ci_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/ci_commands.py
@@ -299,7 +299,7 @@ class WorkflowInfo(NamedTuple):
print(get_ga_output(name="runs-on", value=self.get_runs_on()))
print(get_ga_output(name='in-workflow-build', value=self.in_workflow_build()))
print(get_ga_output(name="build-job-description", value=self.get_build_job_description()))
- print(get_ga_output(name="merge-run", value=self.is_merge_run()))
+ print(get_ga_output(name="canary-run", value=self.is_canary_run()))
print(get_ga_output(name="run-coverage", value=self.run_coverage()))
def get_runs_on(self) -> str:
@@ -321,7 +321,7 @@ class WorkflowInfo(NamedTuple):
return "Build"
return "Skip Build (look in pull_request_target)"
- def is_merge_run(self) -> str:
+ def is_canary_run(self) -> str:
if (
self.event_name == 'push'
and self.head_repo == "apache/airflow"
diff --git a/dev/breeze/src/airflow_breeze/global_constants.py b/dev/breeze/src/airflow_breeze/global_constants.py
index 07676e6d2c..a5a7f9126a 100644
--- a/dev/breeze/src/airflow_breeze/global_constants.py
+++ b/dev/breeze/src/airflow_breeze/global_constants.py
@@ -29,13 +29,7 @@ from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT
RUNS_ON_PUBLIC_RUNNER = "ubuntu-20.04"
RUNS_ON_SELF_HOSTED_RUNNER = "self-hosted"
-# Commented this out as we are using buildkit and this vars became irrelevant
-# FORCE_PULL_IMAGES = False
-# CHECK_IF_BASE_PYTHON_IMAGE_UPDATED = False
-FORCE_BUILD_IMAGES = False
ANSWER = ""
-SKIP_CHECK_REMOTE_IMAGE = False
-# PUSH_PYTHON_BASE_IMAGE = False
APACHE_AIRFLOW_GITHUB_REPOSITORY = "apache/airflow"
diff --git a/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py b/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
index e3aa376c77..f276051008 100644
--- a/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
+++ b/dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
@@ -579,7 +579,6 @@ def update_expected_environment_variables(env: dict[str, str]) -> None:
set_value_to_default_if_not_set(env, 'DEFAULT_BRANCH', AIRFLOW_BRANCH)
set_value_to_default_if_not_set(env, 'ENABLED_SYSTEMS', "")
set_value_to_default_if_not_set(env, 'ENABLE_TEST_COVERAGE', "false")
- set_value_to_default_if_not_set(env, 'GITHUB_REGISTRY_PULL_IMAGE_TAG', "latest")
set_value_to_default_if_not_set(env, 'HOST_GROUP_ID', get_host_group_id())
set_value_to_default_if_not_set(env, 'HOST_OS', get_host_os())
set_value_to_default_if_not_set(env, 'HOST_USER_ID', get_host_user_id())
diff --git a/dev/breeze/tests/test_pr_info.py b/dev/breeze/tests/test_pr_info.py
index 031ae752cd..9cdc1b1bb7 100644
--- a/dev/breeze/tests/test_pr_info.py
+++ b/dev/breeze/tests/test_pr_info.py
@@ -40,7 +40,7 @@ def test_pr_info():
assert wi.event_name == 'pull_request'
assert wi.pr_number == 26004
assert wi.get_runs_on() == "ubuntu-20.04"
- assert wi.is_merge_run() == "false"
+ assert wi.is_canary_run() == "false"
assert wi.run_coverage() == "false"
@@ -54,7 +54,7 @@ def test_push_info():
assert wi.event_name == 'push'
assert wi.pr_number is None
assert wi.get_runs_on() == "ubuntu-20.04"
- assert wi.is_merge_run() == "true"
+ assert wi.is_canary_run() == "true"
assert wi.run_coverage() == "true"
@@ -68,7 +68,7 @@ def test_schedule():
assert wi.event_name == 'schedule'
assert wi.pr_number is None
assert wi.get_runs_on() == "ubuntu-20.04"
- assert wi.is_merge_run() == "false"
+ assert wi.is_canary_run() == "false"
assert wi.run_coverage() == "false"
@@ -82,7 +82,7 @@ def test_runs_on_self_hosted():
assert wi.event_name == 'pull_request'
assert wi.pr_number == 1234
assert wi.get_runs_on() == "self-hosted"
- assert wi.is_merge_run() == "false"
+ assert wi.is_canary_run() == "false"
assert wi.run_coverage() == "false"
@@ -96,7 +96,7 @@ def test_runs_on_forced_public_runner():
assert wi.event_name == 'pull_request'
assert wi.pr_number == 1234
assert wi.get_runs_on() == "ubuntu-20.04"
- assert wi.is_merge_run() == "false"
+ assert wi.is_canary_run() == "false"
assert wi.run_coverage() == "false"
@@ -110,7 +110,7 @@ def test_runs_on_simple_pr_other_repo():
assert wi.event_name == 'pull_request'
assert wi.pr_number == 1234
assert wi.get_runs_on() == "ubuntu-20.04"
- assert wi.is_merge_run() == "false"
+ assert wi.is_canary_run() == "false"
assert wi.run_coverage() == "false"
@@ -124,7 +124,7 @@ def test_runs_on_push_other_branch():
assert wi.event_name == 'push'
assert wi.pr_number is None
assert wi.get_runs_on() == "self-hosted"
- assert wi.is_merge_run() == "false"
+ assert wi.is_canary_run() == "false"
assert wi.run_coverage() == "false"
@@ -138,5 +138,5 @@ def test_runs_on_push_v_test_branch():
assert wi.event_name == 'push'
assert wi.pr_number is None
assert wi.get_runs_on() == "self-hosted"
- assert wi.is_merge_run() == "true"
+ assert wi.is_canary_run() == "true"
assert wi.run_coverage() == "false"
diff --git a/scripts/ci/docker-compose/_docker.env b/scripts/ci/docker-compose/_docker.env
index 8b90912eaa..3dafb73f68 100644
--- a/scripts/ci/docker-compose/_docker.env
+++ b/scripts/ci/docker-compose/_docker.env
@@ -35,7 +35,6 @@ ENABLED_INTEGRATIONS
ENABLED_SYSTEMS
ENABLE_TEST_COVERAGE
GITHUB_ACTIONS
-GITHUB_REGISTRY_PULL_IMAGE_TAG
HOST_USER_ID
HOST_GROUP_ID
HOST_OS
diff --git a/scripts/ci/docker-compose/base.yml b/scripts/ci/docker-compose/base.yml
index 7904bc74fb..b5d0f895b5 100644
--- a/scripts/ci/docker-compose/base.yml
+++ b/scripts/ci/docker-compose/base.yml
@@ -48,7 +48,6 @@ services:
- ENABLED_SYSTEMS=${ENABLED_SYSTEMS}
- ENABLE_TEST_COVERAGE=${ENABLE_TEST_COVERAGE}
- GITHUB_ACTIONS=${GITHUB_ACTIONS}
- - GITHUB_REGISTRY_PULL_IMAGE_TAG=${GITHUB_REGISTRY_PULL_IMAGE_TAG}
- HOST_USER_ID=${HOST_USER_ID}
- HOST_GROUP_ID=${HOST_GROUP_ID}
- HOST_OS=${HOST_OS}
diff --git a/scripts/ci/docker-compose/devcontainer.env b/scripts/ci/docker-compose/devcontainer.env
index 3f40808d98..4f10d5beb6 100644
--- a/scripts/ci/docker-compose/devcontainer.env
+++ b/scripts/ci/docker-compose/devcontainer.env
@@ -35,7 +35,6 @@ ENABLED_INTEGRATIONS=
ENABLED_SYSTEMS=
ENABLE_TEST_COVERAGE="false"
GITHUB_ACTIONS="false"
-GITHUB_REGISTRY_PULL_IMAGE_TAG=""
HOST_USER_ID=
HOST_GROUP_ID=
HOST_OS="linux"
diff --git a/scripts/ci/libraries/_all_libs.sh b/scripts/ci/libraries/_all_libs.sh
index 87de95bfa0..da893d5dcb 100755
--- a/scripts/ci/libraries/_all_libs.sh
+++ b/scripts/ci/libraries/_all_libs.sh
@@ -36,8 +36,6 @@ readonly SCRIPTS_CI_DIR
. "${LIBRARIES_DIR}"/_sanity_checks.sh
# shellcheck source=scripts/ci/libraries/_local_mounts.sh
. "${LIBRARIES_DIR}"/_local_mounts.sh
-# shellcheck source=scripts/ci/libraries/_md5sum.sh
-. "${LIBRARIES_DIR}"/_md5sum.sh
# shellcheck source=scripts/ci/libraries/_start_end.sh
. "${LIBRARIES_DIR}"/_start_end.sh
# shellcheck source=scripts/ci/libraries/_testing.sh
diff --git a/scripts/ci/libraries/_initialization.sh b/scripts/ci/libraries/_initialization.sh
index d16f817290..82edac3491 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -274,16 +274,10 @@ function initialization::initialize_mount_variables() {
# Determine values of force settings
function initialization::initialize_force_variables() {
- # Determines whether to force build without checking if it is needed
- # Can be overridden by '--force-build-images' flag.
- export FORCE_BUILD_IMAGES=${FORCE_BUILD_IMAGES:="false"}
# Can be set to "yes/no/quit" in order to force specified answer to all questions asked to the user.
export ANSWER=${ANSWER:=""}
- # Can be set to true to skip if the image is newer in registry
- export SKIP_CHECK_REMOTE_IMAGE=${SKIP_CHECK_REMOTE_IMAGE:="false"}
-
# integrations are disabled by default
export ENABLED_INTEGRATIONS=${ENABLED_INTEGRATIONS:=""}
@@ -461,8 +455,6 @@ function initialization::initialize_git_variables() {
}
function initialization::initialize_github_variables() {
- export GITHUB_REGISTRY_PULL_IMAGE_TAG=${GITHUB_REGISTRY_PULL_IMAGE_TAG:="latest"}
- export GITHUB_REGISTRY_PUSH_IMAGE_TAG=${GITHUB_REGISTRY_PUSH_IMAGE_TAG:="latest"}
export GITHUB_REPOSITORY=${GITHUB_REPOSITORY:="apache/airflow"}
# Allows to override the repository which is used as source of constraints during the build
@@ -557,7 +549,7 @@ function initialization::get_docker_cache_image_names() {
# Example:
# ghcr.io/apache/airflow/main/ci/python3.8:latest
# ghcr.io/apache/airflow/main/ci/python3.8:<COMMIT_SHA>
- export AIRFLOW_CI_IMAGE_WITH_TAG="${image_name}/${BRANCH_NAME}/ci/python${PYTHON_MAJOR_MINOR_VERSION}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
+ export AIRFLOW_CI_IMAGE_WITH_TAG="${image_name}/${BRANCH_NAME}/ci/python${PYTHON_MAJOR_MINOR_VERSION}:latest"
# File that is touched when the CI image is built for the first time locally
export BUILT_CI_IMAGE_FLAG_FILE="${BUILD_CACHE_DIR}/${BRANCH_NAME}/.built_${PYTHON_MAJOR_MINOR_VERSION}"
@@ -585,9 +577,7 @@ Mount variables:
Force variables:
- FORCE_BUILD_IMAGES: ${FORCE_BUILD_IMAGES}
ANSWER: ${ANSWER}
- SKIP_CHECK_REMOTE_IMAGE: ${SKIP_CHECK_REMOTE_IMAGE}
Host variables:
@@ -643,7 +633,6 @@ Detected GitHub environment:
GITHUB_REPOSITORY: '${GITHUB_REPOSITORY}'
GITHUB_USERNAME: '${GITHUB_USERNAME}'
GITHUB_REGISTRY_PULL_IMAGE_TAG: '${GITHUB_REGISTRY_PULL_IMAGE_TAG}'
- GITHUB_REGISTRY_PUSH_IMAGE_TAG: '${GITHUB_REGISTRY_PUSH_IMAGE_TAG}'
GITHUB_ACTIONS: '${GITHUB_ACTIONS=}'
Initialization variables:
@@ -772,7 +761,6 @@ function initialization::make_constants_read_only() {
readonly ADDITIONAL_RUNTIME_APT_ENV
readonly GITHUB_REGISTRY_PULL_IMAGE_TAG
- readonly GITHUB_REGISTRY_PUSH_IMAGE_TAG
readonly GITHUB_REPOSITORY
readonly GITHUB_TOKEN
diff --git a/scripts/ci/libraries/_md5sum.sh b/scripts/ci/libraries/_md5sum.sh
deleted file mode 100644
index 1838936fc9..0000000000
--- a/scripts/ci/libraries/_md5sum.sh
+++ /dev/null
@@ -1,154 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-declare -a MODIFIED_FILES
-#
-# Verifies if stored md5sum of the file changed since the last time it was checked
-# The md5sum files are stored in .build directory - you can delete this directory
-# If you want to rebuild everything from the scratch
-#
-function md5sum::calculate_file_md5sum {
- local file="${1}"
- local md5sum
- local md5sum_cache_dir="${BUILD_CACHE_DIR}/${BRANCH_NAME}/${PYTHON_MAJOR_MINOR_VERSION}/${THE_IMAGE_TYPE}"
- mkdir -pv "${md5sum_cache_dir}"
- md5sum=$(md5sum "${file}")
- local md5sum_file
- md5sum_file="${md5sum_cache_dir}"/$(basename "$(dirname "${file}")")-$(basename "${file}").md5sum
- local md5sum_file_new
- md5sum_file_new=${CACHE_TMP_FILE_DIR}/$(basename "$(dirname "${file}")")-$(basename "${file}").md5sum.new
- echo "${md5sum}" > "${md5sum_file_new}"
- local ret_code=0
- if [[ ! -f "${md5sum_file}" ]]; then
- verbosity::print_info "Missing md5sum for ${file#"${AIRFLOW_SOURCES}"} (${md5sum_file#"${AIRFLOW_SOURCES}"})"
- ret_code=1
- else
- diff "${md5sum_file_new}" "${md5sum_file}" >/dev/null
- local res=$?
- if [[ "${res}" != "0" ]]; then
- verbosity::print_info "The md5sum changed for ${file}: was $(cat "${md5sum_file}") now it is $(cat "${md5sum_file_new}")"
- if [[ ${CI} == "true" ]]; then
- echo "${COLOR_RED}The file has changed: ${file}${COLOR_RESET}"
- echo "${COLOR_BLUE}==============================${COLOR_RESET}"
- cat "${file}"
- echo "${COLOR_BLUE}==============================${COLOR_RESET}"
- fi
- ret_code=1
- fi
- fi
- return ${ret_code}
-}
-
-#
-# Moves md5sum file from it's temporary location in CACHE_TMP_FILE_DIR to
-# BUILD_CACHE_DIR - thus updating stored MD5 sum for the file
-#
-function md5sum::move_file_md5sum {
- local file="${1}"
- local md5sum_file
- local md5sum_cache_dir="${BUILD_CACHE_DIR}/${BRANCH_NAME}/${PYTHON_MAJOR_MINOR_VERSION}/${THE_IMAGE_TYPE}"
- mkdir -pv "${md5sum_cache_dir}"
- md5sum_file="${md5sum_cache_dir}"/$(basename "$(dirname "${file}")")-$(basename "${file}").md5sum
- local md5sum_file_new
- md5sum_file_new=${CACHE_TMP_FILE_DIR}/$(basename "$(dirname "${file}")")-$(basename "${file}").md5sum.new
- if [[ -f "${md5sum_file_new}" ]]; then
- mv "${md5sum_file_new}" "${md5sum_file}"
- verbosity::print_info "Updated md5sum file ${md5sum_file} for ${file}: $(cat "${md5sum_file}")"
- fi
-}
-
-#
-# Stores md5sum files for all important files and
-# records that we built the images locally so that next time we use
-# it from the local docker cache rather than pull (unless forced)
-#
-function md5sum::update_all_md5() {
- verbosity::print_info
- verbosity::print_info "Updating md5sum files"
- verbosity::print_info
- for file in "${FILES_FOR_REBUILD_CHECK[@]}"
- do
- md5sum::move_file_md5sum "${AIRFLOW_SOURCES}/${file}"
- done
- mkdir -pv "${BUILD_CACHE_DIR}/${BRANCH_NAME}"
- touch "${BUILT_CI_IMAGE_FLAG_FILE}"
-}
-
-function md5sum::update_all_md5_with_group() {
- start_end::group_start "Update MD5 hashes for pulled images"
- md5sum::update_all_md5
- start_end::group_end
-}
-
-function md5sum::calculate_md5sum_for_all_files() {
- FILES_MODIFIED="false"
- set +e
- for file in "${FILES_FOR_REBUILD_CHECK[@]}"
- do
- if ! md5sum::calculate_file_md5sum "${AIRFLOW_SOURCES}/${file}"; then
- FILES_MODIFIED="true"
- MODIFIED_FILES+=( "${file}" )
- fi
- done
- set -e
-}
-
-#
-# Checks md5sum of all important files in order to optimise speed of running various operations
-# That mount sources of Airflow to container and require docker image built with latest dependencies.
-# the Docker image will only be marked for rebuilding only in case any of the important files change:
-# * setup.py
-# * setup.cfg
-# * Dockerfile.ci
-#
-# This is needed because we want to skip rebuilding of the image when only airflow sources change but
-# Trigger rebuild in case we need to change dependencies (setup.py, setup.cfg, change version of Airflow
-# or the Dockerfile.ci itself changes.
-#
-# Another reason to skip rebuilding Docker is thar currently it takes a bit longer time than simple Docker
-# We need to fix group permissions of files in Docker because different linux build services have
-# different default umask and Docker uses group permissions in checking for cache invalidation.
-#
-# As result of this check - most of the static checks will start pretty much immediately.
-#
-function md5sum::check_if_docker_build_is_needed() {
- verbosity::print_info
- verbosity::print_info "Checking if image build is needed for ${THE_IMAGE_TYPE} image."
- verbosity::print_info
- if [[ ${FORCE_BUILD_IMAGES:=""} == "true" ]]; then
- verbosity::print_info
- verbosity::print_info "${COLOR_YELLOW}Docker image build is forced for ${THE_IMAGE_TYPE} image${COLOR_RESET}"
- verbosity::print_info
- md5sum::calculate_md5sum_for_all_files
- needs_docker_build="true"
- else
- md5sum::calculate_md5sum_for_all_files
- if [[ ${FILES_MODIFIED} == "true" ]]; then
- needs_docker_build="true"
- fi
- if [[ ${needs_docker_build} == "true" ]]; then
- verbosity::print_info
- verbosity::print_info "${COLOR_YELLOW}The files were modified and likely the ${THE_IMAGE_TYPE} image needs rebuild: ${MODIFIED_FILES[*]}${COLOR_RESET}"
- verbosity::print_info
- else
- verbosity::print_info
- verbosity::print_info "${COLOR_GREEN}Docker image build is not needed for ${THE_IMAGE_TYPE} image!${COLOR_RESET}"
- verbosity::print_info
- fi
- fi
-}
[airflow] 01/13: Respect "common" options value in breeze sub-commands. (#26264)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-4-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 5c073b52f63c900ab2429892695827ab4b1a57bc
Author: Ash Berlin-Taylor <as...@apache.org>
AuthorDate: Fri Sep 9 14:51:01 2022 +0100
Respect "common" options value in breeze sub-commands. (#26264)
This lets us run `breeze -v static-checks` and have the sub-command pick
up the value from the parent context. If you specify a value for the
subcommand, that is used in place of the global one.
(The fact that `-v` was allowed before the sub-command but had no effect
caused me lots of confusion)
(cherry picked from commit 6da571f705bbfd92f8641f2604ef6f4c61bfa889)
---
.../src/airflow_breeze/utils/common_options.py | 30 +++++++++++++++++++++-
1 file changed, 29 insertions(+), 1 deletion(-)
diff --git a/dev/breeze/src/airflow_breeze/utils/common_options.py b/dev/breeze/src/airflow_breeze/utils/common_options.py
index 79b21d906e..d6f4699a2f 100644
--- a/dev/breeze/src/airflow_breeze/utils/common_options.py
+++ b/dev/breeze/src/airflow_breeze/utils/common_options.py
@@ -49,8 +49,32 @@ from airflow_breeze.utils.custom_param_types import (
)
from airflow_breeze.utils.recording import generating_command_images
+
+def _set_default_from_parent(ctx: "click.core.Context", option: "click.core.Option", value):
+ from click.core import ParameterSource
+
+ if (
+ ctx.parent
+ and option.name in ctx.parent.params
+ and ctx.get_parameter_source(option.name)
+ in (
+ ParameterSource.DEFAULT,
+ ParameterSource.DEFAULT_MAP,
+ )
+ ):
+ # Current value is the default, use the parent's value (i.e. for `breeze
+ # # -v static-checks` respect the "global" option)
+ value = ctx.parent.params[option.name]
+ return value
+
+
option_verbose = click.option(
- "-v", "--verbose", is_flag=True, help="Print verbose information about performed steps.", envvar='VERBOSE'
+ "-v",
+ "--verbose",
+ is_flag=True,
+ help="Print verbose information about performed steps.",
+ envvar='VERBOSE',
+ callback=_set_default_from_parent,
)
option_dry_run = click.option(
"-D",
@@ -58,6 +82,7 @@ option_dry_run = click.option(
is_flag=True,
help="If dry-run is set, commands are only printed, not executed.",
envvar='DRY_RUN',
+ callback=_set_default_from_parent,
)
option_answer = click.option(
"-a",
@@ -65,6 +90,7 @@ option_answer = click.option(
type=AnswerChoice(['y', 'n', 'q', 'yes', 'no', 'quit']),
help="Force answer to questions.",
envvar='ANSWER',
+ callback=_set_default_from_parent,
)
option_github_repository = click.option(
'-g',
@@ -73,6 +99,7 @@ option_github_repository = click.option(
default=APACHE_AIRFLOW_GITHUB_REPOSITORY,
show_default=True,
envvar='GITHUB_REPOSITORY',
+ callback=_set_default_from_parent,
)
option_python = click.option(
'-p',
@@ -501,4 +528,5 @@ option_max_time = click.option(
help="Maximum time that the command should take - if it takes longer, the command will fail.",
type=click.IntRange(min=1),
envvar='MAX_TIME',
+ callback=_set_default_from_parent,
)
[airflow] 03/13: Create a workflow to sync the list of authors (#26275)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-4-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e574aa78f9ef13149daa9c3096ce09d708a8a189
Author: Mark Norman Francis <no...@201created.com>
AuthorDate: Tue Sep 13 13:30:47 2022 +0100
Create a workflow to sync the list of authors (#26275)
The CI jobs caused by Pull Requests run on Airflow infrastructure only
when an author is trusted, which currently means a core committer. The
list is currently duplicated in the ci.yml workflow.
As part of being able to open up the infra to other trusted authors,
this list needs to be updated based on a list of trusted authors kept
in the `airflow-ci-infra` repository.
(cherry picked from commit 73257628219e09c6cfd14b38d0aeed8b6e46a83c)
---
.github/workflows/ci.yml | 12 +++---
.github/workflows/sync_authors.yml | 64 ++++++++++++++++++++++++++++++++
scripts/ci/runners/sync_authors.py | 76 ++++++++++++++++++++++++++++++++++++++
3 files changed, 145 insertions(+), 7 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index ece19a4c8f..742583271c 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -60,13 +60,11 @@ jobs:
# is checked again by the runner using it's own list, so a PR author cannot
# change this and get access to our self-hosted runners
#
- # When changing this list, ensure that it is kept in sync with the
- # /runners/apache/airflow/configOverlay
- # parameter in AWS SSM ParameterStore (which is what the runner uses)
- # and restart the self-hosted runners.
- #
- # This list of committers can be generated with:
- # https://github.com/apache/airflow-ci-infra/blob/main/scripts/list_committers
+ # This list is kept up-to-date from the list of authors found in the
+ # 'airflow-ci-infra' by the 'sync_authors' Github workflow. It uses a regexp
+ # to find the list of authors and replace them, so any changes to the
+ # formatting of the contains(fromJSON()) structure below will need to be
+ # reflected in that workflow too.
runs-on: >-
${{ (
(
diff --git a/.github/workflows/sync_authors.yml b/.github/workflows/sync_authors.yml
new file mode 100644
index 0000000000..8405292859
--- /dev/null
+++ b/.github/workflows/sync_authors.yml
@@ -0,0 +1,64 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: Sync authors list
+
+on: # yamllint disable-line rule:truthy
+ schedule:
+ # min hr dom mon dow
+ - cron: '11 01 * * *' # daily at 1.11am
+
+permissions:
+ contents: write
+ pull-requests: write
+
+jobs:
+ sync:
+ name: Sync
+ runs-on: ubuntu-latest
+
+ steps:
+ - name: Checkout
+ uses: actions/checkout@v3
+ with:
+ persist-credentials: false
+
+ - name: Python
+ uses: actions/setup-python@v4
+ with:
+ python-version: '3.10'
+
+ - name: Requests
+ run: |
+ pip install requests toml
+
+ - name: Sync the authors list
+ env:
+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ run: |
+ python scripts/ci/runners/sync_authors.py
+ git config user.name "GitHub Actions"
+ git config user.email "actions@users.noreply.github.com"
+ if [ -n "$(git status --porcelain)" ]; then
+ branch=update-$(date +%s)
+ git add -A
+ git checkout -b $branch
+ git commit --message "Authors list automatic update"
+ git push origin $branch
+ gh pr create --title "Authors list automatic update" --body ''
+ fi
diff --git a/scripts/ci/runners/sync_authors.py b/scripts/ci/runners/sync_authors.py
new file mode 100644
index 0000000000..6df09305a5
--- /dev/null
+++ b/scripts/ci/runners/sync_authors.py
@@ -0,0 +1,76 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import re
+
+import requests
+import toml
+
+
+# The list of users in the 'build-info' job looks like:
+#
+# contains(fromJSON('[
+# "BasPH",
+# ...
+# ]'), github.event.pull_request.user.login)
+#
+# This script should replace the contents of the array with a new list of
+# identically formatted names, such that changes to the source of truth:
+
+AUTHORS = 'https://raw.githubusercontent.com/apache/airflow-ci-infra/main/authors.toml'
+
+# end up being reflected in the ci.yml.
+
+
+req = requests.get(AUTHORS)
+author_list = toml.loads(req.text)
+
+author_set = set()
+for membership in author_list:
+ author_set.update([author for author in author_list[membership]])
+
+authors = ''
+for author in sorted(author_set):
+ authors += f' "{author}",\n'
+
+authors = authors[:-2]
+
+with open('ci.yml') as handle:
+
+
+ new_ci = re.sub(
+ r'''
+ ^
+ # matches the entire file up to contains(fromJSON('[
+ ( .*? contains.fromJSON \( ' \[ \n )
+
+ # the list of authors (which is replaced)
+ .*?
+
+ # the remainder of the file, from the end of the list onwards
+ ( \s+ \] ' \), . github\.event\.pull_request\.user\.login .*? )
+ $
+ ''',
+ f'\\1{authors}\\2',
+ handle.read(),
+ flags=re.DOTALL | re.VERBOSE,
+ )
+
+with open('ci.yml', 'w') as handle:
+ handle.write(new_ci)
[airflow] 10/13: Manual triggering of author sync workflow (#26414)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-4-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 79e017f5589694698b7d98428731db6315b30ebc
Author: Mark Norman Francis <no...@201created.com>
AuthorDate: Thu Sep 15 11:45:52 2022 +0100
Manual triggering of author sync workflow (#26414)
* Allow committers to run author sync manually
* Fix test failures
(cherry picked from commit 4261210aecbac43fba355d348beb3301f4f79ac5)
---
.github/workflows/sync_authors.yml | 3 +++
.pre-commit-config.yaml | 1 +
2 files changed, 4 insertions(+)
diff --git a/.github/workflows/sync_authors.yml b/.github/workflows/sync_authors.yml
index 52cb6db782..5ef30ba9c4 100644
--- a/.github/workflows/sync_authors.yml
+++ b/.github/workflows/sync_authors.yml
@@ -22,6 +22,9 @@ on: # yamllint disable-line rule:truthy
schedule:
# min hr dom mon dow
- cron: '11 01 * * *' # daily at 1.11am
+ workflow_dispatch:
+ # only users with write access to apache/airflow can run manually
+ # https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow
permissions:
contents: write
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 8b6cd16b8e..2509776664 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -780,6 +780,7 @@ repos:
language: python
pass_filenames: true
files: ^\.github/workflows/.*\.yml$
+ exclude: ^\.github/workflows/sync_authors\.yml$
additional_dependencies: ['PyYAML', 'rich>=12.4.4']
- id: check-docstring-param-types
name: Check that docstrings do not specify param types
[airflow] 06/13: Use the full path to the workflow being edited (#26386)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-4-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 7c7ce3c7bf6f190a4611cf96747c6d3b93398ab7
Author: Mark Norman Francis <no...@201created.com>
AuthorDate: Wed Sep 14 13:39:58 2022 +0100
Use the full path to the workflow being edited (#26386)
(cherry picked from commit 3e917f1c02c4654a135c9a740ee264b766bbc58a)
---
scripts/ci/runners/sync_authors.py | 9 ++++-----
1 file changed, 4 insertions(+), 5 deletions(-)
diff --git a/scripts/ci/runners/sync_authors.py b/scripts/ci/runners/sync_authors.py
index 8fc9717e5b..6e68352711 100644
--- a/scripts/ci/runners/sync_authors.py
+++ b/scripts/ci/runners/sync_authors.py
@@ -31,10 +31,10 @@ import toml
#
# This script should replace the contents of the array with a new list of
# identically formatted names, such that changes to the source of truth:
-
AUTHORS = 'https://raw.githubusercontent.com/apache/airflow-ci-infra/main/authors.toml'
-# end up being reflected in the ci.yml.
+# ...end up being reflected in the ci.yml:
+WORKFLOW = '.github/workflows/ci.yml'
req = requests.get(AUTHORS)
@@ -50,8 +50,7 @@ for author in sorted(author_set):
authors = authors[:-2]
-with open('ci.yml') as handle:
-
+with open(WORKFLOW) as handle:
new_ci = re.sub(
r'''
^
@@ -70,5 +69,5 @@ with open('ci.yml') as handle:
flags=re.DOTALL | re.VERBOSE,
)
-with open('ci.yml', 'w') as handle:
+with open(WORKFLOW, 'w') as handle:
handle.write(new_ci)