You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2024/07/26 13:19:43 UTC
(airflow) branch bring-back-arm-image-building created (now 0bbf1829e2)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch bring-back-arm-image-building
in repository https://gitbox.apache.org/repos/asf/airflow.git
at 0bbf1829e2 Bring back ARM image building using ASF self-hosted ARM runners
This branch includes the following new commits:
new 0bbf1829e2 Bring back ARM image building using ASF self-hosted ARM runners
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(airflow) 01/01: Bring back ARM image building using ASF self-hosted ARM runners
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch bring-back-arm-image-building
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 0bbf1829e275a401b56a3e7f4c91eb5c4f449fa9
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Fri Jul 26 14:59:50 2024 +0200
Bring back ARM image building using ASF self-hosted ARM runners
We can now use self-hosted ARM runners to run ARM builds instead
of spinning our own instances. This PR brings back those builds to
use the ASF runners.
This still does not solve the problem of building multi-platform
images because with buildkit we need to have two interconnected
machines with two different platforms to build such an image and
this is not easy/possible with GitHub Actions. We might need to
eventually manually generate manifest and put together such
images later, but for now we build those images manually.
Related: #40925
---
.github/workflows/additional-ci-image-checks.yml | 52 ++++++++++++----------
.github/workflows/build-images.yml | 3 +-
.github/workflows/ci-image-build.yml | 18 ++------
.github/workflows/ci.yml | 5 ++-
.github/workflows/finalize-tests.yml | 48 ++++++++++----------
.github/workflows/push-image-cache.yml | 23 ++--------
dev/breeze/src/airflow_breeze/global_constants.py | 5 ++-
.../src/airflow_breeze/utils/selective_checks.py | 4 ++
8 files changed, 71 insertions(+), 87 deletions(-)
diff --git a/.github/workflows/additional-ci-image-checks.yml b/.github/workflows/additional-ci-image-checks.yml
index ae9efdb6b0..17e2753c54 100644
--- a/.github/workflows/additional-ci-image-checks.yml
+++ b/.github/workflows/additional-ci-image-checks.yml
@@ -32,12 +32,16 @@ on: # yamllint disable-line rule:truthy
description: "The array of labels (in json form) determining self-hosted runners."
required: true
type: string
+ runs-on-as-json-self-hosted-arm:
+ description: "The array of labels (in json form) determining self-hosted ARM runners."
+ required: true
+ type: string
image-tag:
description: "Tag to set for the image"
required: true
type: string
python-versions:
- description: "The list of python versions (stringified JSON array) to run the tests on."
+ description: "The list of python versions (stringifies JSON array) to run the tests on."
required: true
type: string
branch:
@@ -145,26 +149,26 @@ jobs:
- name: "Check that image builds quickly"
run: breeze shell --max-time 600 --platform "linux/amd64"
-# # This is only a check if ARM images are successfully building when committer runs PR from
-# # Apache repository. This is needed in case you want to fix failing cache job in "canary" run
-# # There is no point in running this one in "canary" run, because the above step is doing the
-# # same build anyway.
-# build-ci-arm-images:
-# name: Build CI ARM images (in-workflow)
-# uses: ./.github/workflows/ci-image-build.yml
-# permissions:
-# contents: read
-# packages: write
-# secrets: inherit
-# with:
-# push-image: "false"
-# runs-on-as-json-public: ${{ inputs.runs-on-as-json-public }}
-# runs-on-as-json-self-hosted: ${{ inputs.runs-on-as-json-self-hosted }}
-# image-tag: ${{ inputs.image-tag }}
-# python-versions: ${{ inputs.python-versions }}
-# platform: "linux/arm64"
-# branch: ${{ inputs.branch }}
-# constraints-branch: ${{ inputs.constraints-branch }}
-# use-uv: "true"
-# upgrade-to-newer-dependencies: ${{ inputs.upgrade-to-newer-dependencies }}
-# docker-cache: ${{ inputs.docker-cache }}
+ # This is only a check if ARM images are successfully building when committer runs PR from
+ # Apache repository. This is needed in case you want to fix failing cache job in "canary" run
+ # There is no point in running this one in "canary" run, because the above step is doing the
+ # same build anyway.
+ build-ci-arm-images:
+ name: Build CI ARM images (in-workflow)
+ uses: ./.github/workflows/ci-image-build.yml
+ permissions:
+ contents: read
+ packages: write
+ secrets: inherit
+ with:
+ push-image: "false"
+ runs-on-as-json-public: ${{ inputs.runs-on-as-json-self-hosted-arm }}
+ runs-on-as-json-self-hosted-arm: ${{ inputs.runs-on-as-json-self-hosted-arm }}
+ image-tag: ${{ inputs.image-tag }}
+ python-versions: ${{ inputs.python-versions }}
+ platform: "linux/arm64"
+ branch: ${{ inputs.branch }}
+ constraints-branch: ${{ inputs.constraints-branch }}
+ use-uv: "true"
+ upgrade-to-newer-dependencies: ${{ inputs.upgrade-to-newer-dependencies }}
+ docker-cache: ${{ inputs.docker-cache }}
diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml
index bd10e73aac..e3de47a28d 100644
--- a/.github/workflows/build-images.yml
+++ b/.github/workflows/build-images.yml
@@ -71,6 +71,7 @@ jobs:
runs-on-as-json-default: ${{ steps.selective-checks.outputs.runs-on-as-json-default }}
runs-on-as-json-public: ${{ steps.selective-checks.outputs.runs-on-as-json-public }}
runs-on-as-json-self-hosted: ${{ steps.selective-checks.outputs.runs-on-as-json-self-hosted }}
+ runs-on-as-json-self-hosted-arm: ${{ steps.selective-checks.outputs.runs-on-as-json-self-hosted-arm }}
is-self-hosted-runner: ${{ steps.selective-checks.outputs.is-self-hosted-runner }}
is-committer-build: ${{ steps.selective-checks.outputs.is-committer-build }}
is-airflow-runner: ${{ steps.selective-checks.outputs.is-airflow-runner }}
@@ -192,7 +193,7 @@ jobs:
github.event.pull_request.head.repo.full_name != 'apache/airflow'
with:
runs-on-as-json-public: ${{ needs.build-info.outputs.runs-on-as-json-public }}
- runs-on-as-json-self-hosted: ${{ needs.build-info.outputs.runs-on-as-json-self-hosted }}
+ runs-on-as-json-self-hosted-arm: ${{ needs.build-info.outputs.runs-on-as-json-self-hosted-arm }}
do-build: ${{ needs.build-info.outputs.ci-image-build }}
target-commit-sha: ${{ needs.build-info.outputs.target-commit-sha }}
pull-request-target: "true"
diff --git a/.github/workflows/ci-image-build.yml b/.github/workflows/ci-image-build.yml
index 07ba028cf7..3bc63f3625 100644
--- a/.github/workflows/ci-image-build.yml
+++ b/.github/workflows/ci-image-build.yml
@@ -24,8 +24,8 @@ on: # yamllint disable-line rule:truthy
description: "The array of labels (in json form) determining public runners."
required: true
type: string
- runs-on-as-json-self-hosted:
- description: "The array of labels (in json form) determining self-hosted runners."
+ runs-on-as-json-self-hosted-arm:
+ description: "The array of labels (in json form) determining self-hosted ARM runners."
required: true
type: string
do-build:
@@ -108,17 +108,11 @@ ${{ inputs.do-build == 'true' && 'Build' || 'Skip building' }} \
CI ${{ inputs.platform }} image\
${{ matrix.python-version }}${{ inputs.do-build == 'true' && ':' || '' }}\
${{ inputs.do-build == 'true' && inputs.image-tag || '' }}"
- # The ARM images need to be built using self-hosted runners as ARM macos public runners
- # do not yet allow us to run docker effectively and fast.
- # https://github.com/actions/runner-images/issues/9254#issuecomment-1917916016
- # https://github.com/abiosoft/colima/issues/970
- # https://github.com/actions/runner/issues/1456
- # See https://github.com/apache/airflow/pull/38640
# NOTE!!!!! This has to be put in one line for runs-on to recognize the "fromJSON" properly !!!!
# adding space before (with >) apparently turns the `runs-on` processed line into a string "Array"
# instead of an array of strings.
# yamllint disable-line rule:line-length
- runs-on: ${{ (inputs.platform == 'linux/amd64') && fromJSON(inputs.runs-on-as-json-public) || fromJSON(inputs.runs-on-as-json-self-hosted) }}
+ runs-on: ${{ (inputs.platform == 'linux/amd64') && fromJSON(inputs.runs-on-as-json-public) || fromJSON(inputs.runs-on-as-json-self-hosted-arm) }}
env:
BACKEND: sqlite
DEFAULT_BRANCH: ${{ inputs.branch }}
@@ -157,9 +151,6 @@ ${{ inputs.do-build == 'true' && inputs.image-tag || '' }}"
pip install rich>=12.4.4 pyyaml
python scripts/ci/pre_commit/update_providers_dependencies.py
if: inputs.do-build == 'true' && inputs.upgrade-to-newer-dependencies != 'false'
- - name: "Start ARM instance"
- run: ./scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh
- if: inputs.do-build == 'true' && inputs.platform == 'linux/arm64'
- name: Login to ghcr.io
run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
if: inputs.do-build == 'true'
@@ -185,6 +176,3 @@ ${{ inputs.do-build == 'true' && inputs.image-tag || '' }}"
PUSH: ${{ inputs.push-image }}
VERBOSE: "true"
if: inputs.do-build == 'true'
- - name: "Stop ARM instance"
- run: ./scripts/ci/images/ci_stop_arm_instance.sh
- if: always() && inputs.do-build == 'true' && inputs.platform == 'linux/arm64'
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index babb1da265..a77235380f 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -110,6 +110,7 @@ jobs:
runs-on-as-json-default: ${{ steps.selective-checks.outputs.runs-on-as-json-default }}
runs-on-as-json-public: ${{ steps.selective-checks.outputs.runs-on-as-json-public }}
runs-on-as-json-self-hosted: ${{ steps.selective-checks.outputs.runs-on-as-json-self-hosted }}
+ runs-on-as-json-self-hosted-arm: ${{ steps.selective-checks.outputs.runs-on-as-json-self-hosted-arm }}
is-self-hosted-runner: ${{ steps.selective-checks.outputs.is-self-hosted-runner }}
is-airflow-runner: ${{ steps.selective-checks.outputs.is-airflow-runner }}
is-amd-runner: ${{ steps.selective-checks.outputs.is-amd-runner }}
@@ -191,7 +192,7 @@ jobs:
secrets: inherit
with:
runs-on-as-json-public: ${{ needs.build-info.outputs.runs-on-as-json-public }}
- runs-on-as-json-self-hosted: ${{ needs.build-info.outputs.runs-on-as-json-self-hosted }}
+ runs-on-as-json-self-hosted-arm: ${{ needs.build-info.outputs.runs-on-as-json-self-hosted-arm }}
do-build: ${{ needs.build-info.outputs.in-workflow-build }}
image-tag: ${{ needs.build-info.outputs.image-tag }}
platform: "linux/amd64"
@@ -644,7 +645,7 @@ jobs:
uses: ./.github/workflows/finalize-tests.yml
with:
runs-on-as-json-public: ${{ needs.build-info.outputs.runs-on-as-json-public }}
- runs-on-as-json-self-hosted: ${{ needs.build-info.outputs.runs-on-as-json-self-hosted }}
+ runs-on-as-json-self-hosted-arm: ${{ needs.build-info.outputs.runs-on-as-json-self-hosted-arm }}
image-tag: ${{ needs.build-info.outputs.image-tag }}
python-versions: ${{ needs.build-info.outputs.python-versions }}
python-versions-list-as-string: ${{ needs.build-info.outputs.python-versions-list-as-string }}
diff --git a/.github/workflows/finalize-tests.yml b/.github/workflows/finalize-tests.yml
index 8b392ba204..512614afc3 100644
--- a/.github/workflows/finalize-tests.yml
+++ b/.github/workflows/finalize-tests.yml
@@ -24,8 +24,8 @@ on: # yamllint disable-line rule:truthy
description: "The array of labels (in json form) determining public runners."
required: true
type: string
- runs-on-as-json-self-hosted:
- description: "The array of labels (in json form) determining self-hosted runners."
+ runs-on-as-json-self-hosted-arm:
+ description: "The array of labels (in json form) determining self-hosted ARM runners."
required: true
type: string
image-tag:
@@ -150,28 +150,28 @@ jobs:
docker-cache: ${{ inputs.docker-cache }}
if: inputs.canary-run == 'true'
- # push-buildx-cache-to-github-registry-arm:
- # name: Push Regular ARM Image Cache
- # needs: [update-constraints]
- # uses: ./.github/workflows/push-image-cache.yml
- # permissions:
- # contents: read
- # packages: write
- # secrets: inherit
- # with:
- # runs-on-as-json-public: ${{ inputs.runs-on-as-json-public }}
- # runs-on-as-json-self-hosted: ${{ inputs.runs-on-as-json-self-hosted }}
- # cache-type: "Regular ARM"
- # include-prod-images: "true"
- # push-latest-images: "true"
- # platform: "linux/arm64"
- # python-versions: ${{ inputs.python-versions }}
- # branch: ${{ inputs.branch }}
- # constraints-branch: ${{ inputs.constraints-branch }}
- # use-uv: "true"
- # include-success-outputs: ${{ inputs.include-success-outputs }}
- # docker-cache: ${{ inputs.docker-cache }}
- # if: inputs.canary-run == 'true'
+ push-buildx-cache-to-github-registry-arm:
+ name: Push Regular ARM Image Cache
+ needs: [update-constraints]
+ uses: ./.github/workflows/push-image-cache.yml
+ permissions:
+ contents: read
+ packages: write
+ secrets: inherit
+ with:
+ runs-on-as-json-public: ${{ inputs.runs-on-as-json-public }}
+ runs-on-as-json-self-hosted-arm: ${{ inputs.runs-on-as-json-self-hosted-arm }}
+ cache-type: "Regular ARM"
+ include-prod-images: "true"
+ push-latest-images: "true"
+ platform: "linux/arm64"
+ python-versions: ${{ inputs.python-versions }}
+ branch: ${{ inputs.branch }}
+ constraints-branch: ${{ inputs.constraints-branch }}
+ use-uv: "true"
+ include-success-outputs: ${{ inputs.include-success-outputs }}
+ docker-cache: ${{ inputs.docker-cache }}
+ if: inputs.canary-run == 'true'
summarize-warnings:
timeout-minutes: 15
diff --git a/.github/workflows/push-image-cache.yml b/.github/workflows/push-image-cache.yml
index 1cdb5861e4..5312726461 100644
--- a/.github/workflows/push-image-cache.yml
+++ b/.github/workflows/push-image-cache.yml
@@ -24,8 +24,8 @@ on: # yamllint disable-line rule:truthy
description: "The array of labels (in json form) determining public runners."
required: true
type: string
- runs-on-as-json-self-hosted:
- description: "The array of labels (in json form) determining self-hosted runners."
+ runs-on-as-json-self-hosted-arm:
+ description: "The array of labels (in json form) determining self-hosted ARM runners."
required: true
type: string
cache-type:
@@ -83,7 +83,7 @@ jobs:
# adding space before (with >) apparently turns the `runs-on` processed line into a string "Array"
# instead of an array of strings.
# yamllint disable-line rule:line-length
- runs-on: ${{ (inputs.platform == 'linux/amd64') && fromJSON(inputs.runs-on-as-json-public) || fromJSON(inputs.runs-on-as-json-self-hosted) }}
+ runs-on: ${{ (inputs.platform == 'linux/amd64') && fromJSON(inputs.runs-on-as-json-public) || fromJSON(inputs.runs-on-as-json-self-hosted-arm) }}
strategy:
fail-fast: false
matrix:
@@ -121,18 +121,12 @@ jobs:
run: ./scripts/ci/cleanup_docker.sh
- name: "Install Breeze"
uses: ./.github/actions/breeze
- - name: "Start ARM instance"
- run: ./scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh
- if: inputs.platform == 'linux/arm64'
- name: Login to ghcr.io
run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
- name: "Push CI ${{ inputs.cache-type }} cache: ${{ matrix.python }} ${{ inputs.platform }}"
run: >
breeze ci-image build --builder airflow_cache --prepare-buildx-cache
--platform "${{ inputs.platform }}" --python ${{ matrix.python }}
- - name: "Stop ARM instance"
- run: ./scripts/ci/images/ci_stop_arm_instance.sh
- if: always() && inputs.platform == 'linux/arm64'
- name: "Push CI latest images: ${{ matrix.python }} (linux/amd64 only)"
run: >
breeze ci-image build --builder airflow_cache --push
@@ -145,7 +139,7 @@ jobs:
# adding space before (with >) apparently turns the `runs-on` processed line into a string "Array"
# instead of an array of strings.
# yamllint disable-line rule:line-length
- runs-on: ${{ (inputs.platform == 'linux/amd64') && fromJSON(inputs.runs-on-as-json-public) || fromJSON(inputs.runs-on-as-json-self-hosted) }}
+ runs-on: ${{ (inputs.platform == 'linux/amd64') && fromJSON(inputs.runs-on-as-json-public) || fromJSON(inputs.runs-on-as-json-self-hosted-arm) }}
strategy:
fail-fast: false
matrix:
@@ -190,9 +184,6 @@ jobs:
with:
name: prod-packages
path: ./docker-context-files
- - name: "Start ARM instance"
- run: ./scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh
- if: inputs.platform == 'linux/arm64'
- name: Login to ghcr.io
run: echo "${{ env.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
- name: "Push PROD ${{ inputs.cache-type }} cache: ${{ matrix.python-version }} ${{ inputs.platform }}"
@@ -201,12 +192,6 @@ jobs:
--prepare-buildx-cache --platform "${{ inputs.platform }}"
--install-packages-from-context --airflow-constraints-mode constraints-source-providers
--python ${{ matrix.python }}
- - name: "Stop ARM instance"
- run: ./scripts/ci/images/ci_stop_arm_instance.sh
- if: always() && inputs.platform == 'linux/arm64'
- # We only push "AMD" images as it is really only needed for any kind of automated builds in CI
- # and currently there is not an easy way to make multi-platform image from two separate builds
- # and we can do it after we stopped the ARM instance as it is not needed anymore
- name: "Push PROD latest image: ${{ matrix.python }} (linux/amd64 ONLY)"
run: >
breeze prod-image build --builder airflow_cache --install-packages-from-context
diff --git a/dev/breeze/src/airflow_breeze/global_constants.py b/dev/breeze/src/airflow_breeze/global_constants.py
index 94b64d621c..62aca0a7a8 100644
--- a/dev/breeze/src/airflow_breeze/global_constants.py
+++ b/dev/breeze/src/airflow_breeze/global_constants.py
@@ -30,8 +30,8 @@ from airflow_breeze.utils.host_info_utils import Architecture
from airflow_breeze.utils.path_utils import AIRFLOW_SOURCES_ROOT
RUNS_ON_PUBLIC_RUNNER = '["ubuntu-22.04"]'
-# we should get more sophisticated logic here in the future, but for now we just check if
-# we use self airflow, vm-based, amd hosted runner as a default
+RUNS_ON_SELF_HOSTED_ARM_RUNNER = '["self-hosted", "asf-arm"] '
+
# TODO: temporarily we need to switch to public runners to avoid issues with self-hosted runners
RUNS_ON_SELF_HOSTED_RUNNER = '["ubuntu-22.04"]'
# TODO: when we have it properly set-up with labels we should change it to
@@ -39,6 +39,7 @@ RUNS_ON_SELF_HOSTED_RUNNER = '["ubuntu-22.04"]'
# RUNS_ON_SELF_HOSTED_RUNNER = '["self-hosted", "Linux", "X64"]'
SELF_HOSTED_RUNNERS_CPU_COUNT = 8
+
ANSWER = ""
APACHE_AIRFLOW_GITHUB_REPOSITORY = "apache/airflow"
diff --git a/dev/breeze/src/airflow_breeze/utils/selective_checks.py b/dev/breeze/src/airflow_breeze/utils/selective_checks.py
index 070f669bd4..fe37a726e9 100644
--- a/dev/breeze/src/airflow_breeze/utils/selective_checks.py
+++ b/dev/breeze/src/airflow_breeze/utils/selective_checks.py
@@ -1142,6 +1142,10 @@ class SelectiveChecks:
def runs_on_as_json_self_hosted(self) -> str:
return RUNS_ON_SELF_HOSTED_RUNNER
+ @cached_property
+ def runs_on_as_json_self_hosted_arm(self) -> str:
+ return RUNS_ON_SELF_HOSTED_RUNNER
+
@cached_property
def runs_on_as_json_public(self) -> str:
return RUNS_ON_PUBLIC_RUNNER