You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2022/03/26 21:23:53 UTC
[airflow] branch main updated: Optimize direct push workflows in GitHub Actions (#22542)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new bf0a11c Optimize direct push workflows in GitHub Actions (#22542)
bf0a11c is described below
commit bf0a11c63dffe3953d1490cfc04b3a5e0366e710
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sat Mar 26 22:23:05 2022 +0100
Optimize direct push workflows in GitHub Actions (#22542)
When the build is run via direct push to apache airflow repo
we do not need to run two separate workflows. The "push" workflow
is never a "pull request from fork" so it should have
the capability to build and push images to registry.
This allows the committers to make direct push requests to run PRs
that are actually running the build without having to merge
build-image.yml first.
This is cool because committers can simply push a branch to apache
and test if it works with some build image changes that otherwise
would require to push to `main` of an apache-airflow fork.
Another advantage is that merge and schedule builds do not run two
separate workflows - both building the image and running tests is done
in the same workflow (and the build-image workflow is not started)
This saves some build time on "wait for CI images" and
"wait for PROD images" jobs - because in merge builds and
schedule builds they only start after the images are actually
built.
---
.github/workflows/build-images.yml | 12 +-
.github/workflows/ci.yml | 222 ++++++++++--
CI.rst | 14 +-
CI_DIAGRAMS.md | 376 +++++++++++----------
PULL_REQUEST_WORKFLOW.rst | 2 +-
docs/apache-airflow/installation/prerequisites.rst | 2 +-
6 files changed, 414 insertions(+), 214 deletions(-)
diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml
index e23b363..855e83c 100644
--- a/.github/workflows/build-images.yml
+++ b/.github/workflows/build-images.yml
@@ -18,11 +18,7 @@
---
name: "Build Images"
on: # yamllint disable-line rule:truthy
- schedule:
- - cron: '28 0 * * *'
pull_request_target:
- push:
- branches: ['main', 'v[0-9]+-[0-9]+-test']
permissions:
# all other permissions are set to none
contents: read
@@ -168,7 +164,9 @@ jobs:
matrix:
python-version: ${{ fromJson(needs.build-info.outputs.allPythonVersions) }}
fail-fast: true
- if: needs.build-info.outputs.image-build == 'true'
+ if: |
+ needs.build-info.outputs.image-build == 'true' &&
+ github.event.pull_request.repo.head.full_name != 'apache/airflow'
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn)[0] }}
BACKEND: sqlite
@@ -254,7 +252,9 @@ jobs:
matrix:
python-version: ${{ fromJson(needs.build-info.outputs.allPythonVersions) }}
fail-fast: true
- if: needs.build-info.outputs.image-build == 'true'
+ if: |
+ needs.build-info.outputs.image-build == 'true' &&
+ github.event.pull_request.repo.head.full_name != 'apache/airflow'
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn)[0] }}
BACKEND: sqlite
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 2a6bde2..49c0c0b 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -21,7 +21,6 @@ on: # yamllint disable-line rule:truthy
schedule:
- cron: '28 0 * * *'
push:
- branches: ['main', 'v[0-9]+-[0-9]+-test']
pull_request:
branches: ['main', 'v[0-9]+-[0-9]+-test', 'v[0-9]+-[0-9]+-stable']
permissions:
@@ -45,10 +44,6 @@ env:
secrets.CONSTRAINTS_GITHUB_REPOSITORY || 'apache/airflow' }}
# In builds from forks, this token is read-only. For scheduler/direct push it is WRITE one
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- # In builds from forks, this token is empty, and this is good because such builds do not even try
- # to push images to the registry.
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ github.event.pull_request.head.sha || github.sha }}"
- GITHUB_REGISTRY_PUSH_IMAGE_TAG: "latest"
INSTALL_PROVIDERS_FROM_SOURCES: "true"
AIRFLOW_LOGIN_TO_GITHUB_REGISTRY: "true"
ENABLE_TEST_COVERAGE: "${{ github.event_name == 'push' }}"
@@ -127,6 +122,10 @@ jobs:
env:
GITHUB_CONTEXT: ${{ toJson(github) }}
outputs:
+ targetBranch: ${{ steps.dynamic-outputs.outputs.targetBranch }}
+ defaultBranch: ${{ steps.selective-checks.outputs.default-branch }}
+ waitForImage: ${{ steps.wait-for-image.outputs.wait-for-image }}
+ allPythonVersions: ${{ steps.selective-checks.outputs.all-python-versions }}
upgradeToNewerDependencies: ${{ steps.selective-checks.outputs.upgrade-to-newer-dependencies }}
pythonVersions: ${{ steps.selective-checks.outputs.python-versions }}
pythonVersionsListAsString: ${{ steps.selective-checks.outputs.python-versions-list-as-string }}
@@ -161,10 +160,14 @@ jobs:
needs-api-tests: ${{ steps.selective-checks.outputs.needs-api-tests }}
needs-api-codegen: ${{ steps.selective-checks.outputs.needs-api-codegen }}
default-branch: ${{ steps.selective-checks.outputs.default-branch }}
+ sourceHeadRepo: ${{ steps.source-run-info.outputs.sourceHeadRepo }}
pullRequestNumber: ${{ steps.source-run-info.outputs.pullRequestNumber }}
pullRequestLabels: ${{ steps.source-run-info.outputs.pullRequestLabels }}
runsOn: ${{ steps.set-runs-on.outputs.runsOn }}
runCoverage: ${{ steps.set-run-coverage.outputs.runCoverage }}
+ inWorkflowBuild: ${{ steps.set-in-workflow-build.outputs.inWorkflowBuild }}
+ buildJobDescription: ${{ steps.set-in-workflow-build.outputs.buildJobDescription }}
+ mergeRun: ${{ steps.set-merge-run.outputs.merge-run }}
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
@@ -216,12 +219,174 @@ jobs:
# Avoid having to specify the coverage logic every time.
- name: Set run coverage
id: set-run-coverage
- run: |
- echo "::set-output name=runCoverage::true"
+ run: echo "::set-output name=runCoverage::true"
if: >
github.ref == 'refs/heads/main' && github.repository == 'apache/airflow' &&
github.event_name == 'push' &&
steps.selective-checks.outputs.default-branch == 'main'
+ - name: Determine where to run image builds
+ id: set-in-workflow-build
+ # Run in-workflow build image when:
+ # * direct push is run
+ # * schedule build is run
+ # * pull request is run not from fork
+ run: |
+ set -x
+ if [[ ${GITHUB_EVENT_NAME} == "push" || ${GITHUB_EVENT_NAME} == "push" || \
+ ${{steps.source-run-info.outputs.sourceHeadRepo}} == "apache/airflow" ]]; then
+ echo "Images will be built in current workflow"
+ echo "::set-output name=inWorkflowBuild::true"
+ echo "::set-output name=buildJobDescription::Build"
+ else
+ echo "Images will be built in pull_request_target workflow"
+ echo "::set-output name=inWorkflowBuild::false"
+ echo "::set-output name=buildJobDescription::Skip Build (pull_request_target)"
+ fi
+ - name: Determine if this is merge run (or schedule)
+ id: set-merge-run
+ run: echo "::set-output name=merge-run::true"
+ # Only in Apache Airflow repo, when either schedule or push to main or any of v2*test branches
+ if: |
+ github.repository == 'apache/airflow' &&
+ (github.event_name == 'push' || github.event_name == 'schedule') &&
+ (
+ github.ref_name == 'main' ||
+ startsWith(github.ref_name, 'v2') && endsWith(github.ref_name, 'test')
+ )
+
+ build-ci-images:
+ permissions:
+ packages: write
+ timeout-minutes: 80
+ name: "${{needs.build-info.outputs.buildJobDescription}} CI image ${{matrix.python-version}}"
+ runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
+ needs: [build-info]
+ strategy:
+ matrix:
+ python-version: ${{ fromJson(needs.build-info.outputs.allPythonVersions) }}
+ fail-fast: true
+ env:
+ RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn)[0] }}
+ BACKEND: sqlite
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
+ UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgradeToNewerDependencies }}
+ DOCKER_CACHE: ${{ needs.build-info.outputs.cacheDirective }}
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
+ GITHUB_REGISTRY_PUSH_IMAGE_TAG: ${{ github.event.pull_request.head.sha || github.sha }}
+ GITHUB_REGISTRY_WAIT_FOR_IMAGE: "false"
+ outputs: ${{toJSON(needs.build-info.outputs) }}
+ steps:
+ - uses: actions/checkout@v2
+ with:
+ ref: ${{ needs.build-info.outputs.targetCommitSha }}
+ persist-credentials: false
+ submodules: recursive
+ if: needs.build-info.outputs.inWorkflowBuild == 'true'
+ - name: "Setup python"
+ uses: actions/setup-python@v2
+ with:
+ python-version: ${{ needs.build-info.outputs.defaultPythonVersion }}
+ if: needs.build-info.outputs.inWorkflowBuild == 'true'
+ - run: python -m pip install --editable ./dev/breeze/
+ if: needs.build-info.outputs.inWorkflowBuild == 'true'
+ - name: "Retrieve DEFAULTS from the _initialization.sh"
+ # We cannot "source" the script here because that would be a security problem (we cannot run
+ # any code that comes from the sources coming from the PR. Therefore we extract the
+ # DEFAULT_BRANCH and DEFAULT_CONSTRAINTS_BRANCH and DEBIAN_VERSION via custom grep/awk/sed commands
+ id: defaults
+ run: |
+ DEFAULT_BRANCH=$(grep "export DEFAULT_BRANCH" scripts/ci/libraries/_initialization.sh | \
+ awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
+ echo "DEFAULT_BRANCH=${DEFAULT_BRANCH}" >> $GITHUB_ENV
+ DEFAULT_CONSTRAINTS_BRANCH=$(grep "export DEFAULT_CONSTRAINTS_BRANCH" \
+ scripts/ci/libraries/_initialization.sh | \
+ awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
+ echo "DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH}" >> $GITHUB_ENV
+ DEBIAN_VERSION=$(grep "export DEBIAN_VERSION" scripts/ci/libraries/_initialization.sh | \
+ awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
+ echo "DEBIAN_VERSION=${DEBIAN_VERSION}" >> $GITHUB_ENV
+ if: needs.build-info.outputs.inWorkflowBuild == 'true'
+ - name: "Free space"
+ run: airflow-freespace
+ if: needs.build-info.outputs.inWorkflowBuild == 'true'
+ - name: "Build CI image ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
+ run: Breeze2 build-ci-image
+ if: needs.build-info.outputs.inWorkflowBuild == 'true'
+ - name: "Push CI image ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
+ run: ./scripts/ci/images/ci_push_ci_images.sh
+ if: needs.build-info.outputs.inWorkflowBuild == 'true'
+ - name: "Candidates for pip resolver backtrack triggers: ${{ matrix.python-version }}"
+ if: failure() || cancelled()
+ run: airflow-find-newer-dependencies --max-age 1 --python "${{ matrix.python-version }}"
+
+ build-prod-images:
+ permissions:
+ packages: write
+ timeout-minutes: 80
+ name: "${{needs.build-info.outputs.buildJobDescription}} PROD image ${{matrix.python-version}}"
+ runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
+ needs: [build-info, build-ci-images]
+ strategy:
+ matrix:
+ python-version: ${{ fromJson(needs.build-info.outputs.allPythonVersions) }}
+ fail-fast: true
+ env:
+ RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn)[0] }}
+ BACKEND: sqlite
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
+ UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgradeToNewerDependencies }}
+ DOCKER_CACHE: ${{ needs.build-info.outputs.cacheDirective }}
+ VERSION_SUFFIX_FOR_PYPI: ".dev0"
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
+ GITHUB_REGISTRY_PUSH_IMAGE_TAG: ${{ github.event.pull_request.head.sha || github.sha }}
+ GITHUB_REGISTRY_WAIT_FOR_IMAGE: "false"
+ INSTALL_PROVIDERS_FROM_SOURCES: >
+ ${{ needs.build-info.outputs.defaultBranch == 'main' && 'true' || 'false' }}
+ steps:
+ - uses: actions/checkout@v2
+ with:
+ ref: ${{ needs.build-info.outputs.targetCommitSha }}
+ persist-credentials: false
+ submodules: recursive
+ if: needs.build-info.outputs.inWorkflowBuild == 'true'
+ - name: "Setup python"
+ uses: actions/setup-python@v2
+ with:
+ python-version: ${{ needs.build-info.outputs.defaultPythonVersion }}
+ if: needs.build-info.outputs.inWorkflowBuild == 'true'
+ - name: "Retrieve DEFAULTS from the _initialization.sh"
+ # We cannot "source" the script here because that would be a security problem (we cannot run
+ # any code that comes from the sources coming from the PR. Therefore we extract the
+ # DEFAULT_BRANCH and DEFAULT_CONSTRAINTS_BRANCH and DEBIAN_VERSION via custom grep/awk/sed commands
+ id: defaults
+ run: |
+ DEFAULT_BRANCH=$(grep "export DEFAULT_BRANCH" scripts/ci/libraries/_initialization.sh | \
+ awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
+ echo "DEFAULT_BRANCH=${DEFAULT_BRANCH}" >> $GITHUB_ENV
+ DEFAULT_CONSTRAINTS_BRANCH=$(grep "export DEFAULT_CONSTRAINTS_BRANCH" \
+ scripts/ci/libraries/_initialization.sh | \
+ awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
+ echo "DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH}" >> $GITHUB_ENV
+ DEBIAN_VERSION=$(grep "export DEBIAN_VERSION" scripts/ci/libraries/_initialization.sh | \
+ cut -d "=" -f 3 | sed s'/["}]//g')
+ echo "DEBIAN_VERSION=${DEBIAN_VERSION}" >> $GITHUB_ENV
+ if: needs.build-info.outputs.inWorkflowBuild == 'true'
+ - run: python -m pip install --editable ./dev/breeze/
+ if: needs.build-info.outputs.inWorkflowBuild == 'true'
+ - name: "Free space"
+ run: airflow-freespace
+ if: needs.build-info.outputs.inWorkflowBuild == 'true'
+ - name: "Pull CI image for PROD ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
+ run: ./scripts/ci/images/ci_pull_ci_image_on_ci.sh
+ env:
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: ${{ github.event.pull_request.head.sha || github.sha }}
+ if: needs.build-info.outputs.inWorkflowBuild == 'true'
+ - name: "Build PROD image ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
+ run: ./scripts/ci/images/ci_build_prod_image_on_ci.sh
+ if: needs.build-info.outputs.inWorkflowBuild == 'true'
+ - name: "Push PROD image ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
+ run: ./scripts/ci/images/ci_push_production_images.sh
+ if: needs.build-info.outputs.inWorkflowBuild == 'true'
run-new-breeze-tests:
timeout-minutes: 10
@@ -367,12 +532,13 @@ jobs:
timeout-minutes: 120
name: "Wait for CI images"
runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
- needs: [build-info]
+ needs: [build-info, build-ci-images]
if: needs.build-info.outputs.image-build == 'true'
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
BACKEND: sqlite
UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgradeToNewerDependencies }}
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ github.event.pull_request.head.sha || github.sha }}"
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
@@ -415,6 +581,7 @@ jobs:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
MOUNT_SELECTED_LOCAL_SOURCES: "true"
PYTHON_MAJOR_MINOR_VERSION: ${{needs.build-info.outputs.defaultPythonVersion}}
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ github.event.pull_request.head.sha || github.sha }}"
if: needs.build-info.outputs.basic-checks-only == 'false'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
@@ -580,6 +747,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
VERSION_SUFFIX_FOR_PYPI: ".dev0"
NON_INTERACTIVE: "true"
GENERATE_PROVIDERS_ISSUE: "true"
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ github.event.pull_request.head.sha || github.sha }}"
if: needs.build-info.outputs.image-build == 'true' && needs.build-info.outputs.default-branch == 'main'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
@@ -632,6 +800,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
VERSION_SUFFIX_FOR_PYPI: ".dev0"
NON_INTERACTIVE: "true"
GENERATE_PROVIDERS_ISSUE: "true"
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ github.event.pull_request.head.sha || github.sha }}"
if: needs.build-info.outputs.image-build == 'true' && needs.build-info.outputs.default-branch == 'main'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
@@ -682,6 +851,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
BACKEND: ""
DB_RESET: "false"
PYTHON_MAJOR_MINOR_VERSION: ${{needs.build-info.outputs.defaultPythonVersion}}
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ github.event.pull_request.head.sha || github.sha }}"
if: >
needs.build-info.outputs.needs-helm-tests == 'true' &&
(github.repository == 'apache/airflow' || github.event_name != 'schedule') &&
@@ -749,6 +919,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
POSTGRES_VERSION: ${{ matrix.postgres-version }}
TEST_TYPES: "${{needs.build-info.outputs.testTypes}}"
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ github.event.pull_request.head.sha || github.sha }}"
if: needs.build-info.outputs.run-tests == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
@@ -816,6 +987,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
MYSQL_VERSION: ${{ matrix.mysql-version }}
TEST_TYPES: "${{needs.build-info.outputs.testTypes}}"
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ github.event.pull_request.head.sha || github.sha }}"
if: needs.build-info.outputs.run-tests == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
@@ -882,6 +1054,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
MSSQL_VERSION: ${{ matrix.mssql-version }}
TEST_TYPES: "${{needs.build-info.outputs.testTypes}}"
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ github.event.pull_request.head.sha || github.sha }}"
if: needs.build-info.outputs.run-tests == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
@@ -944,6 +1117,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgradeToNewerDependencies }}
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
TEST_TYPES: "${{needs.build-info.outputs.testTypes}}"
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ github.event.pull_request.head.sha || github.sha }}"
if: needs.build-info.outputs.run-tests == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
@@ -1003,6 +1177,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
TEST_TYPES: "Quarantined"
NUM_RUNS: 10
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ github.event.pull_request.head.sha || github.sha }}"
if: needs.build-info.outputs.run-tests == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
@@ -1102,13 +1277,14 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
timeout-minutes: 120
name: "Wait for PROD images"
runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
- needs: [build-info, ci-images]
+ needs: [build-info, ci-images, build-prod-images]
if: needs.build-info.outputs.image-build == 'true'
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
BACKEND: sqlite
PYTHON_MAJOR_MINOR_VERSION: ${{ needs.build-info.outputs.defaultPythonVersion }}
UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgradeToNewerDependencies }}
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ github.event.pull_request.head.sha || github.sha }}"
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
@@ -1165,6 +1341,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
${{needs.build-info.outputs.pythonVersionsListAsString}}
CURRENT_KUBERNETES_VERSIONS_AS_STRING: >
${{needs.build-info.outputs.kubernetesVersionsListAsString}}
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ github.event.pull_request.head.sha || github.sha }}"
if: >
( needs.build-info.outputs.run-kubernetes-tests == 'true' ||
needs.build-info.outputs.needs-helm-tests == 'true' ) &&
@@ -1226,6 +1403,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
${{needs.build-info.outputs.pythonVersionsListAsString}}
CURRENT_KUBERNETES_VERSIONS_AS_STRING: >
${{needs.build-info.outputs.kubernetesVersionsListAsString}}
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ github.event.pull_request.head.sha || github.sha }}"
if: >
needs.build-info.outputs.run-kubernetes-tests == 'true' &&
needs.build-info.outputs.default-branch == 'main'
@@ -1298,6 +1476,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
CURRENT_PYTHON_MAJOR_MINOR_VERSIONS_AS_STRING: ${{needs.build-info.outputs.pythonVersionsListAsString}}
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "${{ github.event.pull_request.head.sha || github.sha }}"
if: needs.build-info.outputs.upgradeToNewerDependencies != 'false'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
@@ -1338,26 +1517,20 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- name: "Set constraints branch name"
id: constraints-branch
run: ./scripts/ci/constraints/ci_branch_constraints.sh
- # only actually checkout and push it when we are in apache/airflow repository
+ if: needs.build-info.outputs.mergeRun == 'true'
- name: Checkout ${{ steps.constraints-branch.outputs.branch }}
uses: actions/checkout@v2
- if: >
- github.repository == 'apache/airflow' &&
- (github.event_name == 'push' || github.event_name == 'schedule')
+ if: needs.build-info.outputs.mergeRun == 'true'
with:
path: "repo"
ref: ${{ steps.constraints-branch.outputs.branch }}
persist-credentials: false
- name: "Commit changed constraint files for ${{needs.build-info.outputs.pythonVersions}}"
run: ./scripts/ci/constraints/ci_commit_constraints.sh
- if: >
- github.repository == 'apache/airflow' &&
- (github.event_name == 'push' || github.event_name == 'schedule')
+ if: needs.build-info.outputs.mergeRun == 'true'
- name: "Push changes"
uses: ./.github/actions/github-push-action
- if: >
- github.repository == 'apache/airflow' &&
- (github.event_name == 'push' || github.event_name == 'schedule')
+ if: needs.build-info.outputs.mergeRun == 'true'
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
branch: ${{ steps.constraints-branch.outputs.branch }}
@@ -1377,10 +1550,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- build-info
- constraints
- docs
- # Only run it for direct pushes and scheduled builds
- if: >
- (github.event_name == 'push' || github.event_name == 'schedule')
- && github.repository == 'apache/airflow'
+ if: needs.build-info.outputs.mergeRun == 'true'
strategy:
matrix:
python-version: ${{ fromJson(needs.build-info.outputs.pythonVersions) }}
@@ -1393,6 +1563,8 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
# Rebuild images before push using the latest constraints (just pushed) without
# eager upgrade. Do not wait for images, but rebuild them
UPGRADE_TO_NEWER_DEPENDENCIES: "false"
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
+ GITHUB_REGISTRY_PUSH_IMAGE_TAG: "latest"
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
@@ -1411,12 +1583,8 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
run: ./scripts/ci/images/ci_build_ci_image_on_ci.sh
env:
PREPARE_BUILDX_CACHE: "true"
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
- GITHUB_REGISTRY_PUSH_IMAGE_TAG: "latest"
- name: "Build CI image cache and push ${{env.PYTHON_MAJOR_MINOR_VERSION}}"
run: ./scripts/ci/images/ci_build_prod_image_on_ci.sh
env:
VERSION_SUFFIX_FOR_PYPI: ".dev0"
PREPARE_BUILDX_CACHE: "true"
- GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
- GITHUB_REGISTRY_PUSH_IMAGE_TAG: "latest"
diff --git a/CI.rst b/CI.rst
index 43ea218..da16486 100644
--- a/CI.rst
+++ b/CI.rst
@@ -59,7 +59,7 @@ Container Registry used as cache
We are using GitHub Container Registry to store the results of the ``Build Images``
workflow which is used in the ``Tests`` workflow.
-Currently in main version of Airflow we run tests in 4 different versions of Python (3.6, 3.7, 3.8, 3.9)
+Currently in main version of Airflow we run tests in 4 different versions of Python (3.7, 3.8, 3.9, 3.10)
which means that we have to build 8 images (4 CI ones and 4 PROD ones). Yet we run around 12 jobs
with each of the CI images. That is a lot of time to just build the environment to run. Therefore
we are utilising ``pull_request_target`` feature of GitHub Actions.
@@ -491,7 +491,7 @@ running, GitHub Actions will cancel the old workflow run automatically.
Build Images Workflow
---------------------
-This workflow builds images for the CI Workflow.
+This workflow builds images for the CI Workflow for Pull Requests coming from forks.
It's a special type of workflow: ``pull_request_target`` which means that it is triggered when a pull request
is opened. This also means that the workflow has Write permission to push to the GitHub registry the images
@@ -499,8 +499,14 @@ used by CI jobs which means that the images can be built only once and reused by
(including the matrix jobs). We've implemented it so that the ``Tests`` workflow waits
until the images are built by the ``Build Images`` workflow before running.
-This workflow is also triggered on normal pushes to our "main" branches, i.e. after a
-pull request is merged and whenever ``scheduled`` run is triggered.
+Those "Build Image" steps are skipped in case Pull Requests do not come from "forks" (i.e. those
+are internal PRs for Apache Airflow repository. This is because in case of PRs coming from
+Apache Airflow (only committers can create those) the "pull_request" workflows have enough
+permission to push images to Github Registry.
+
+This workflow is not triggered on normal pushes to our "main" branches, i.e. after a
+pull request is merged and whenever ``scheduled`` run is triggered. Again in this case the "CI" workflow
+has enough permissions to push the images. In this case we simply do not run this workflow.
The workflow has the following jobs:
diff --git a/CI_DIAGRAMS.md b/CI_DIAGRAMS.md
index 1bbee13..3b228b1 100644
--- a/CI_DIAGRAMS.md
+++ b/CI_DIAGRAMS.md
@@ -33,15 +33,13 @@ sequenceDiagram
activate Tests
Tests -->> Build Images: Trigger 'pull_request_target'
activate Build Images
- Note over Build Images: Build info
- par 3.6, [3.7, 3.8, 3.9]
- activate GitHub Registry
- GitHub Registry ->> Build Images: Pull CI Images from Cache
- deactivate GitHub Registry
- Note over Build Images: Build CI Images<br>[COMMIT_SHA]
- end
- par No CI image
- Note over Tests: Build info<br>Which tests?<br>Which Python?
+ Note over Build Images: Build info<br>Decide which Python
+ Note over Tests: Build info<br>Decide on tests<br>Decide on Matrix (selective)
+ Note over Tests: Skip Build<br>(Runs in 'Build Images')<br>CI Images
+ Note over Tests: Skip Build<br>(Runs in 'Build Images')<br>PROD Images
+ par
+ GitHub Registry ->> Build Images: Pull CI Images<br>[latest]
+ Note over Build Images: Build CI Images<br>[COMMIT_SHA]<br>Use latest constraints<br>or upgrade if setup changed
and
Note over Tests: OpenAPI client gen
and
@@ -49,77 +47,135 @@ sequenceDiagram
and
Note over Tests: Test examples<br>PROD image building
end
- par 3.6, [3.7, 3.8, 3.9]
- activate GitHub Registry
- Build Images ->> GitHub Registry: Push CI Images
- Note over GitHub Registry: Tagged CI Images<br>[COMMIT_SHA]
- end
- par 3.6, [3.7, 3.8, 3.9]
- GitHub Registry ->> Build Images: Pull PROD Images from Cache
- Note over Build Images: Build PROD Images<br>[COMMIT_SHA]
- end
+ Build Images ->> GitHub Registry: Push CI Images<br>[COMMIT_SHA]
loop Wait for CI images
- par 3.6, [3.7, 3.8, 3.9]
- Tests ->> Tests: Check CI Images
- Note over Tests: Wait for<br>[COMMIT_SHA]
- end
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
end
- par 3.6, [3.7, 3.8, 3.9]
- GitHub Registry ->> Tests: Pull CI Image
- Note over Tests: Verify CI Image
- end
- deactivate GitHub Registry
- par 3.6, [3.7, 3.8, 3.9]
- opt Needed?
+ Note over Tests: Verify CI Images<br>[COMMIT_SHA]
+ par
+ GitHub Registry ->> Build Images: Pull PROD Images<br>[latest]
+ Note over Build Images: Build PROD Images<br>[COMMIT_SHA]
+ and
+ opt
Note over Tests: Run static checks
end
and
- opt Needed?
+ opt
Note over Tests: Run basic <br>static checks
end
and
- opt Needed?
+ opt
Note over Tests: Build docs
end
and
- opt Needed?
+ opt
Note over Tests: Tests
end
and
- opt Needed?
+ opt
Note over Tests: Test provider <br>packages build
end
and
- opt Needed?
+ opt
Note over Tests: Helm tests
end
end
- par 3.6, [3.7, 3.8, 3.9]
- Build Images ->> GitHub Registry: Push PROD Images
- activate GitHub Registry
- end
+ Build Images ->> GitHub Registry: Push PROD Images<br>[COMMIT_SHA]
deactivate Build Images
- Note over GitHub Registry: Tagged PROD Images<br>[COMMIT_SHA]
loop Wait for PROD images
- par 3.6, [3.7, 3.8, 3.9]
- Tests ->> Tests: Check PROD Images
- Note over Tests: Wait for<br>[COMMIT_SHA]
+ GitHub Registry ->> Tests: Pull PROD Images<br>[COMMIT_SHA]
+ end
+ Note over Tests: Verify PROD Image<br>[COMMIT_SHA]
+ par
+ opt
+ Note over Tests: Run Kubernetes<br>tests
+ end
+ and
+ opt
+ Note over Tests: Run Kubernetes<br>upgrade tests
end
end
- par 3.6, [3.7, 3.8, 3.9]
- GitHub Registry ->> Tests: Pull PROD Image
- Note over Tests: Verify PROD Image
+ opt
+ Note over Tests: Generate constraints
end
- deactivate GitHub Registry
- par 3.6, [3.7, 3.8, 3.9]
- opt Needed?
+ Tests -->> Airflow Repo: Status update
+ deactivate Airflow Repo
+ deactivate Tests
+```
+
+## Pull request flow from "apache/airflow" repo
+
+```mermaid
+sequenceDiagram
+ Note over Airflow Repo: pull request
+ Note over Tests: pull_request<br>[Read Token]
+ Note over Build Images: pull_request_target<br>[Write Token]
+ activate Airflow Repo
+ Airflow Repo -->> Tests: Trigger 'pull_request'
+ activate Tests
+ Tests -->> Build Images: Trigger 'pull_request_target'
+ activate Build Images
+ Note over Build Images: Build info
+ Note over Build Images: Skip Build<br>(Runs in 'Tests')<br>CI Images
+ Note over Build Images: Skip Build<br>(Runs in 'Tests')<br>PROD Images
+ deactivate Build Images
+ Note over Tests: Build info<br>Decide on tests<br>Decide on Matrix (selective)
+ par
+ GitHub Registry ->> Tests: Pull CI Images<br>[latest]
+ Note over Tests: Build CI Images<br>[COMMIT_SHA]<br>Use latest constraints<br>or upgrade if setup changed
+ and
+ Note over Tests: OpenAPI client gen
+ and
+ Note over Tests: Test UI
+ and
+ Note over Tests: Test examples<br>PROD image building
+ end
+ Tests ->> GitHub Registry: Push CI Images<br>[COMMIT_SHA]
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Verify CI Image<br>[COMMIT_SHA]
+ par
+ GitHub Registry ->> Tests: Pull PROD Images<br>[latest]
+ Note over Tests: Build PROD Images<br>[COMMIT_SHA]
+ and
+ opt
+ Note over Tests: Run static checks
+ end
+ and
+ opt
+ Note over Tests: Run basic <br>static checks
+ end
+ and
+ opt
+ Note over Tests: Build docs
+ end
+ and
+ opt
+ Note over Tests: Tests
+ end
+ and
+ opt
+ Note over Tests: Test provider <br>packages build
+ end
+ and
+ opt
+ Note over Tests: Helm tests
+ end
+ end
+ Tests ->> GitHub Registry: Push PROD Images<br>[COMMIT_SHA]
+ GitHub Registry ->> Tests: Pull PROD Image<br>[COMIT_SHA]
+ Note over Tests: Verify PROD Image<br>[COMMIT_SHA]
+ par
+ opt
Note over Tests: Run Kubernetes <br>tests
end
and
- opt Needed?
+ opt
Note over Tests: Run Kubernetes <br>upgrade tests
end
end
+ opt
+ Note over Tests: Generate constraints
+ end
Tests -->> Airflow Repo: Status update
deactivate Airflow Repo
deactivate Tests
@@ -129,23 +185,15 @@ sequenceDiagram
```mermaid
sequenceDiagram
- Note over Airflow Repo: merge
+ Note over Airflow Repo: pull request
Note over Tests: push<br>[Write Token]
- Note over Build Images: push<br>[Write Token]
activate Airflow Repo
Airflow Repo -->> Tests: Trigger 'push'
activate Tests
- Airflow Repo -->> Build Images: Trigger 'push'
- activate Build Images
- Note over Build Images: Build info
- par 3.6, 3.7, 3.8, 3.9
- activate GitHub Registry
- GitHub Registry ->> Build Images: Pull CI Images from Cache
- deactivate GitHub Registry
- Note over Build Images: Build CI Images<br>[COMMIT_SHA]
- end
- par No CI image
- Note over Tests: Build info<br>All tests<br>All python
+ Note over Tests: Build info<br>All tests<br>Full matrix
+ par
+ GitHub Registry ->> Tests: Pull CI Images<br>[latest]
+ Note over Tests: Build CI Images<br>[COMMIT_SHA]<br>Always upgrade deps
and
Note over Tests: OpenAPI client gen
and
@@ -153,72 +201,60 @@ sequenceDiagram
and
Note over Tests: Test examples<br>PROD image building
end
- par 3.6, 3.7, 3.8, 3.9
- Build Images ->> GitHub Registry: Push CI Images
- activate GitHub Registry
- Note over GitHub Registry: Tagged CI Images<br>[COMMIT_SHA]
- end
- par 3.6, 3.7, 3.8, 3.9
- GitHub Registry ->> Build Images: Pull PROD Images from Cache
- Note over Build Images: Build PROD Images<br>[COMMIT_SHA]
- end
- loop Wait for CI images
- par 3.6, 3.7, 3.8, 3.9
- Tests ->> Tests: Check CI Images
- Note over Tests: Wait for<br>[COMMIT_SHA]
+ Tests ->> GitHub Registry: Push CI Images<br>[COMMIT_SHA]
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Verify CI Image<br>[COMMIT_SHA]
+ par
+ GitHub Registry ->> Tests: Pull PROD Images<br>[latest]
+ Note over Tests: Build PROD Images<br>[COMMIT_SHA]
+ and
+ opt
+ Note over Tests: Run static checks
end
- end
- par 3.6, 3.7, 3.8, 3.9
- GitHub Registry ->> Tests: Pull CI Image [COMMIT_SHA]
- Note over Tests: Verify CI Image
- end
- deactivate GitHub Registry
- par 3.6, 3.7, 3.8, 3.9
- Note over Tests: Run static checks
and
- Note over Tests: Build docs
+ opt
+ Note over Tests: Run basic <br>static checks
+ end
and
- Note over Tests: Tests
+ opt
+ Note over Tests: Build docs
+ end
and
- Note over Tests: Test provider <br>packages build
+ opt
+ Note over Tests: Tests
+ end
and
- Note over Tests: Helm tests
- end
- par 3.6, 3.7, 3.8, 3.9
- Build Images ->> GitHub Registry: Push PROD Images
- Note over GitHub Registry: Tagged PROD Images<br>[COMMIT_SHA]
- activate GitHub Registry
- end
- deactivate Build Images
- loop Wait for PROD images
- par 3.6, 3.7, 3.8, 3.9
- Tests ->> Tests: Check PROD Images
- Note over Tests: Wait for<br>[COMMIT_SHA]
+ opt
+ Note over Tests: Test provider <br>packages build
end
- end
- par 3.6, 3.7, 3.8, 3.9
- GitHub Registry ->> Tests: Pull PROD Image [COMMIT_SHA]
- Note over Tests: Verify PROD Image
- end
- deactivate GitHub Registry
- par 3.6, 3.7, 3.8, 3.9
- Note over Tests: Run Kubernetes <br>tests
and
- Note over Tests: Run Kubernetes <br>upgrade tests
+ opt
+ Note over Tests: Helm tests
+ end
end
- Note over Tests: Merge Coverage
- Tests -->> Coverage.io: Upload Coverage
- par 3.6, 3.7, 3.8, 3.9
- Tests ->> GitHub Registry: Push CI Images to Cache
- activate GitHub Registry
+ Tests ->> GitHub Registry: Push PROD Images<br>[COMMIT_SHA]
+ GitHub Registry ->> Tests: Pull PROD Image<br>[COMMIT_SHA]
+ Note over Tests: Verify PROD Image<br>[COMMIT_SHA]
+ par
+ opt
+ Note over Tests: Run Kubernetes <br>tests
+ end
and
- Tests ->> GitHub Registry: Push PROD Images to Cache
+ opt
+ Note over Tests: Run Kubernetes <br>upgrade tests
+ end
end
- Note over GitHub Registry: Tagged Images<br>[latest]
- deactivate GitHub Registry
- par 3.6, 3.7, 3.8, 3.9
- Note over Tests: Generate constraints
- Tests ->> Airflow Repo: Push constraints
+ Note over Tests: Generate constraints
+ opt In merge run?
+ Tests ->> Airflow Repo: Push constraints if changed
+ end
+ opt In merge run?
+ GitHub Registry ->> Tests: Pull CI Image<br>[latest]
+ Note over Tests: Build CI Images<br>[latest]<br>Use latest constraints
+ Tests ->> GitHub Registry: Push CI Image<br>[latest]
+ GitHub Registry ->> Tests: Pull PROD Image<br>[latest]
+ Note over Tests: Build PROD Images<br>[latest]
+ Tests ->> GitHub Registry: Push PROD Image<br>[latest]
end
Tests -->> Airflow Repo: Status update
deactivate Airflow Repo
@@ -230,19 +266,14 @@ sequenceDiagram
```mermaid
sequenceDiagram
Note over Airflow Repo: scheduled
- Note over Tests: schedule<br>[Write Token]
- Note over Build Images: schedule<br>[Write Token]
+ Note over Tests: push<br>[Write Token]
activate Airflow Repo
Airflow Repo -->> Tests: Trigger 'schedule'
activate Tests
- Airflow Repo -->> Build Images: Trigger 'schedule'
- activate Build Images
- Note over Build Images: Build info
- par 3.6, 3.7, 3.8, 3.9
- Note over Build Images: Build CI Images<br>Cache disabled<br>[COMMIT_SHA]
- end
- par No CI image
- Note over Tests: Build info<br>All tests<br>All python
+ Note over Tests: Build info<br>All tests<br>Full matrix
+ par
+ GitHub Registry ->> Tests: Pull CI Images<br>[latest]
+ Note over Tests: Build CI Images<br>[COMMIT_SHA]<br>Always upgrade deps
and
Note over Tests: OpenAPI client gen
and
@@ -250,62 +281,57 @@ sequenceDiagram
and
Note over Tests: Test examples<br>PROD image building
end
- par 3.6, 3.7, 3.8, 3.9
- Build Images ->> GitHub Registry: Push CI Images
- activate GitHub Registry
- Note over GitHub Registry: Tagged CI Images<br>[COMMIT_SHA]
- end
- par 3.6, 3.7, 3.8, 3.9
- Note over Build Images: Build PROD Images<br>Cache disabled<br>[COMMIT_SHA]
- end
- loop Wait for CI images
- par 3.6, 3.7, 3.8, 3.9
- Tests ->> Tests: Check CI Images
- Note over Tests: Wait for<br>[COMMIT_SHA]
+ Tests ->> GitHub Registry: Push CI Images<br>[COMMIT_SHA]
+ GitHub Registry ->> Tests: Pull CI Images<br>[COMMIT_SHA]
+ Note over Tests: Verify CI Image<br>[COMMIT_SHA]
+ par
+ GitHub Registry ->> Tests: Pull PROD Images<br>[latest]
+ Note over Tests: Build PROD Images<br>[COMMIT_SHA]
+ and
+ opt
+ Note over Tests: Run static checks
end
- end
- par 3.6, 3.7, 3.8, 3.9
- GitHub Registry ->> Tests: Pull CI Image [COMMIT_SHA]
- Note over Tests: Verify CI Image
- end
- deactivate GitHub Registry
- par 3.6, 3.7, 3.8, 3.9
- Note over Tests: Run static checks
and
- Note over Tests: Build docs
+ opt
+ Note over Tests: Run basic <br>static checks
+ end
and
- Note over Tests: Tests
+ opt
+ Note over Tests: Build docs
+ end
and
- Note over Tests: Test provider <br>packages build
+ opt
+ Note over Tests: Tests
+ end
and
- Note over Tests: Helm tests
- end
- par 3.6, 3.7, 3.8, 3.9
- Build Images ->> GitHub Registry: Push PROD Images
- activate GitHub Registry
- Note over GitHub Registry: Tagged PROD Images<br>[COMMIT_SHA]
- end
- deactivate Build Images
- loop Wait for PROD images
- par 3.6, 3.7, 3.8, 3.9
- Tests ->> Tests: Check PROD Images
- Note over Tests: Wait for<br>[COMMIT_SHA]
+ opt
+ Note over Tests: Test provider <br>packages build
end
- end
- par 3.6, 3.7, 3.8, 3.9
- GitHub Registry ->> Tests: Pull PROD Image [COMMIT_SHA]
- Note over Tests: Verify PROD Image
- end
- deactivate GitHub Registry
- par 3.6, 3.7, 3.8, 3.9
- Note over Tests: Run Kubernetes <br>tests
and
- Note over Tests: Run Kubernetes <br>upgrade tests
+ opt
+ Note over Tests: Helm tests
+ end
end
- par 3.6, 3.7, 3.8, 3.9
- Note over Tests: Generate constraints
- Tests ->> Airflow Repo: Push constraints
+ Tests ->> GitHub Registry: Push PROD Images<br>[COMMIT_SHA]
+ GitHub Registry ->> Tests: Pull PROD Image<br>[COMMIT_SHA]
+ Note over Tests: Verify PROD Image<br>[COMMIT_SHA]
+ par
+ opt
+ Note over Tests: Run Kubernetes <br>tests
+ end
+ and
+ opt
+ Note over Tests: Run Kubernetes <br>upgrade tests
+ end
end
+ Note over Tests: Generate constraints
+ Tests ->> Airflow Repo: Push constraints if changed
+ GitHub Registry ->> Tests: Pull CI Image<br>[latest]
+ Note over Tests: Build CI Images<br>[latest]<br>Use latest constraints
+ Tests ->> GitHub Registry: Push CI Image<br>[latest]
+ GitHub Registry ->> Tests: Pull PROD Image<br>[latest]
+ Note over Tests: Build PROD Images<br>[latest]
+ Tests ->> GitHub Registry: Push PROD Image<br>[latest]
Tests -->> Airflow Repo: Status update
deactivate Airflow Repo
deactivate Tests
diff --git a/PULL_REQUEST_WORKFLOW.rst b/PULL_REQUEST_WORKFLOW.rst
index 7676586..5c3905c 100644
--- a/PULL_REQUEST_WORKFLOW.rst
+++ b/PULL_REQUEST_WORKFLOW.rst
@@ -57,7 +57,7 @@ We approached the problem by:
3) Even more optimisation came from limiting the scope of tests to only "default" matrix parameters. So far
in Airflow we always run all tests for all matrix combinations. The primary matrix components are:
- * Python versions (currently 3.6, 3.7, 3.8, 3.9, 3.10)
+ * Python versions (currently 3.7, 3.8, 3.9, 3.10)
* Backend types (currently MySQL/Postgres)
* Backed version (currently MySQL 5.7, MySQL 8, Postgres 13
diff --git a/docs/apache-airflow/installation/prerequisites.rst b/docs/apache-airflow/installation/prerequisites.rst
index 8416f02..84b8535 100644
--- a/docs/apache-airflow/installation/prerequisites.rst
+++ b/docs/apache-airflow/installation/prerequisites.rst
@@ -20,7 +20,7 @@ Prerequisites
Airflow is tested with:
-* Python: 3.6, 3.7, 3.8, 3.9
+* Python: 3.7, 3.8, 3.9, 3.10
* Databases: