You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2022/03/26 11:47:18 UTC
[airflow] branch optimize-direct-push-workflows updated (3d733db -> e9869f3)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch optimize-direct-push-workflows
in repository https://gitbox.apache.org/repos/asf/airflow.git.
discard 3d733db Optimize direct push workflows in GitHub Actions
new e9869f3 Optimize direct push workflows in GitHub Actions
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (3d733db)
\
N -- N -- N refs/heads/optimize-direct-push-workflows (e9869f3)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
[airflow] 01/01: Optimize direct push workflows in GitHub Actions
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch optimize-direct-push-workflows
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e9869f34c59508a34b1be8b106ce4bf7ff401bd4
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Fri Mar 25 20:36:50 2022 +0100
Optimize direct push workflows in GitHub Actions
When the build is run via direct push to apache airflow repo
we do not need to run two separate workflows. The "push" workflow
is never a "pull request from fork" so it should have
the capability to build and push images to registry.
This allows the committers to make direct push requests to run PRs
that are actually running the build without having to merge
build-image.yml first.
This is cool because committers can simply push a branch to apache
and test if it works with some build image changes that otherwise
would require to push to `main` of an apache-airflow fork.
Another advantage is that merge and schedule builds do not run two
separate workflows - both building the image and running tests is done
in the same workflow (and the build-image workflow is not started)
This saves some build time on "wait for CI images" and
"wait for PROD images" jobs - because in merge builds and
schedule builds they only start after the images are actually
built.
---
.github/workflows/build-images.yml | 4 -
.github/workflows/ci.yml | 156 ++++++++++++++++++++++++++++++++++++-
2 files changed, 153 insertions(+), 7 deletions(-)
diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml
index e23b363..800466d 100644
--- a/.github/workflows/build-images.yml
+++ b/.github/workflows/build-images.yml
@@ -18,11 +18,7 @@
---
name: "Build Images"
on: # yamllint disable-line rule:truthy
- schedule:
- - cron: '28 0 * * *'
pull_request_target:
- push:
- branches: ['main', 'v[0-9]+-[0-9]+-test']
permissions:
# all other permissions are set to none
contents: read
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 2a6bde2..051753c 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -21,7 +21,6 @@ on: # yamllint disable-line rule:truthy
schedule:
- cron: '28 0 * * *'
push:
- branches: ['main', 'v[0-9]+-[0-9]+-test']
pull_request:
branches: ['main', 'v[0-9]+-[0-9]+-test', 'v[0-9]+-[0-9]+-stable']
permissions:
@@ -127,6 +126,10 @@ jobs:
env:
GITHUB_CONTEXT: ${{ toJson(github) }}
outputs:
+ targetBranch: ${{ steps.dynamic-outputs.outputs.targetBranch }}
+ defaultBranch: ${{ steps.selective-checks.outputs.default-branch }}
+ waitForImage: ${{ steps.wait-for-image.outputs.wait-for-image }}
+ allPythonVersions: ${{ steps.selective-checks.outputs.all-python-versions }}
upgradeToNewerDependencies: ${{ steps.selective-checks.outputs.upgrade-to-newer-dependencies }}
pythonVersions: ${{ steps.selective-checks.outputs.python-versions }}
pythonVersionsListAsString: ${{ steps.selective-checks.outputs.python-versions-list-as-string }}
@@ -165,6 +168,8 @@ jobs:
pullRequestLabels: ${{ steps.source-run-info.outputs.pullRequestLabels }}
runsOn: ${{ steps.set-runs-on.outputs.runsOn }}
runCoverage: ${{ steps.set-run-coverage.outputs.runCoverage }}
+ localBuild: ${{ steps.set-local-build.outputs.localBuild }}
+ buildJobDescription: ${{ steps.set-local-build.outputs.buildJobDescription }}
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
@@ -222,6 +227,151 @@ jobs:
github.ref == 'refs/heads/main' && github.repository == 'apache/airflow' &&
github.event_name == 'push' &&
steps.selective-checks.outputs.default-branch == 'main'
+ # Avoid having to specify the local build logic every time.
+ - name: Set local build
+ id: set-local-build
+ run: |
+ echo "::set-output name=localBuild::true"
+ echo "::set-output name=buildJobDescription::Build"
+ if: github.event_name == 'push' || github.event_name == 'schedule'
+ - name: Set remote build
+ id: set-remote-build
+ run: |
+ echo "::set-output name=localBuild::false"
+ echo "::set-output name=buildJobDescription::Skip (separate workflow)"
+ if: github.event_name != 'push' && github.event_name != 'schedule'
+
+ build-ci-images:
+ permissions:
+ packages: write
+ timeout-minutes: 80
+ name: "${{needs.build-info.outputs.buildJobDescription}} CI image ${{matrix.python-version}}"
+ runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
+ needs: [build-info]
+ strategy:
+ matrix:
+ python-version: ${{ fromJson(needs.build-info.outputs.allPythonVersions) }}
+ fail-fast: true
+ env:
+ RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn)[0] }}
+ BACKEND: sqlite
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
+ UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgradeToNewerDependencies }}
+ DOCKER_CACHE: ${{ needs.build-info.outputs.cacheDirective }}
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
+ GITHUB_REGISTRY_WAIT_FOR_IMAGE: "false"
+ outputs: ${{toJSON(needs.build-info.outputs) }}
+ steps:
+ - uses: actions/checkout@v2
+ with:
+ ref: ${{ needs.build-info.outputs.targetCommitSha }}
+ persist-credentials: false
+ submodules: recursive
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Setup python"
+ uses: actions/setup-python@v2
+ with:
+ python-version: ${{ needs.build-info.outputs.defaultPythonVersion }}
+ if: needs.build-info.outputs.localBuild == 'true'
+ - run: python -m pip install --editable ./dev/breeze/
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Retrieve DEFAULTS from the _initialization.sh"
+ # We cannot "source" the script here because that would be a security problem (we cannot run
+ # any code that comes from the sources coming from the PR. Therefore we extract the
+ # DEFAULT_BRANCH and DEFAULT_CONSTRAINTS_BRANCH and DEBIAN_VERSION via custom grep/awk/sed commands
+ id: defaults
+ run: |
+ DEFAULT_BRANCH=$(grep "export DEFAULT_BRANCH" scripts/ci/libraries/_initialization.sh | \
+ awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
+ echo "DEFAULT_BRANCH=${DEFAULT_BRANCH}" >> $GITHUB_ENV
+ DEFAULT_CONSTRAINTS_BRANCH=$(grep "export DEFAULT_CONSTRAINTS_BRANCH" \
+ scripts/ci/libraries/_initialization.sh | \
+ awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
+ echo "DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH}" >> $GITHUB_ENV
+ DEBIAN_VERSION=$(grep "export DEBIAN_VERSION" scripts/ci/libraries/_initialization.sh | \
+ awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
+ echo "DEBIAN_VERSION=${DEBIAN_VERSION}" >> $GITHUB_ENV
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Free space"
+ run: airflow-freespace
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Build CI image ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
+ run: Breeze2 build-ci-image
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Push CI image ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
+ run: ./scripts/ci/images/ci_push_ci_images.sh
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Candidates for pip resolver backtrack triggers: ${{ matrix.python-version }}"
+ if: failure() || cancelled()
+ run: airflow-find-newer-dependencies --max-age 1 --python "${{ matrix.python-version }}"
+
+ build-prod-images:
+ permissions:
+ packages: write
+ timeout-minutes: 80
+ name: "${{needs.build-info.outputs.buildJobDescription}} PROD image ${{matrix.python-version}}"
+ runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
+ needs: [build-info, build-ci-images]
+ strategy:
+ matrix:
+ python-version: ${{ fromJson(needs.build-info.outputs.allPythonVersions) }}
+ fail-fast: true
+ env:
+ RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn)[0] }}
+ BACKEND: sqlite
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
+ UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgradeToNewerDependencies }}
+ DOCKER_CACHE: ${{ needs.build-info.outputs.cacheDirective }}
+ VERSION_SUFFIX_FOR_PYPI: ".dev0"
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
+ GITHUB_REGISTRY_WAIT_FOR_IMAGE: "false"
+ INSTALL_PROVIDERS_FROM_SOURCES: >
+ ${{ needs.build-info.outputs.defaultBranch == 'main' && 'true' || 'false' }}
+ steps:
+ - uses: actions/checkout@v2
+ with:
+ ref: ${{ needs.build-info.outputs.targetCommitSha }}
+ persist-credentials: false
+ submodules: recursive
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Setup python"
+ uses: actions/setup-python@v2
+ with:
+ python-version: ${{ needs.build-info.outputs.defaultPythonVersion }}
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Retrieve DEFAULTS from the _initialization.sh"
+ # We cannot "source" the script here because that would be a security problem (we cannot run
+ # any code that comes from the sources coming from the PR. Therefore we extract the
+ # DEFAULT_BRANCH and DEFAULT_CONSTRAINTS_BRANCH and DEBIAN_VERSION via custom grep/awk/sed commands
+ id: defaults
+ run: |
+ DEFAULT_BRANCH=$(grep "export DEFAULT_BRANCH" scripts/ci/libraries/_initialization.sh | \
+ awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
+ echo "DEFAULT_BRANCH=${DEFAULT_BRANCH}" >> $GITHUB_ENV
+ DEFAULT_CONSTRAINTS_BRANCH=$(grep "export DEFAULT_CONSTRAINTS_BRANCH" \
+ scripts/ci/libraries/_initialization.sh | \
+ awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
+ echo "DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH}" >> $GITHUB_ENV
+ DEBIAN_VERSION=$(grep "export DEBIAN_VERSION" scripts/ci/libraries/_initialization.sh | \
+ cut -d "=" -f 3 | sed s'/["}]//g')
+ echo "DEBIAN_VERSION=${DEBIAN_VERSION}" >> $GITHUB_ENV
+ if: needs.build-info.outputs.localBuild == 'true'
+ - run: python -m pip install --editable ./dev/breeze/
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Free space"
+ run: airflow-freespace
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Pull CI image for PROD ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
+ run: ./scripts/ci/images/ci_pull_ci_image_on_ci.sh
+ env:
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: ${{ github.event.pull_request.head.sha || github.sha }}
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Build PROD image ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
+ run: ./scripts/ci/images/ci_build_prod_image_on_ci.sh
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Push PROD image ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
+ run: ./scripts/ci/images/ci_push_production_images.sh
+ if: needs.build-info.outputs.localBuild == 'true'
run-new-breeze-tests:
timeout-minutes: 10
@@ -367,7 +517,7 @@ jobs:
timeout-minutes: 120
name: "Wait for CI images"
runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
- needs: [build-info]
+ needs: [build-info, build-ci-images]
if: needs.build-info.outputs.image-build == 'true'
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
@@ -1102,7 +1252,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
timeout-minutes: 120
name: "Wait for PROD images"
runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
- needs: [build-info, ci-images]
+ needs: [build-info, ci-images, build-prod-images]
if: needs.build-info.outputs.image-build == 'true'
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}