You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2022/03/26 11:09:04 UTC
[airflow] branch optimize-direct-push-workflows updated (191be93 -> 0b39b82)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch optimize-direct-push-workflows
in repository https://gitbox.apache.org/repos/asf/airflow.git.
discard 191be93 Optimize direct push workflows in GitHub Actions
new 0b39b82 Optimize direct push workflows in GitHub Actions
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (191be93)
\
N -- N -- N refs/heads/optimize-direct-push-workflows (0b39b82)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/ci.yml | 36 ++++++++++++++++++------------------
1 file changed, 18 insertions(+), 18 deletions(-)
[airflow] 01/01: Optimize direct push workflows in GitHub Actions
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch optimize-direct-push-workflows
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 0b39b82305733d99c63e885efeb6a680e968a4a9
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Fri Mar 25 20:36:50 2022 +0100
Optimize direct push workflows in GitHub Actions
When the build is run via direct push to apache airflow repo
we do not need to run two separate workflows. The "push" workflow
is never a "pull request from fork" so it should have
the capability to build and push images to registry.
This allows the committers to make direct push requests to run PRs
that are actually running the build without having to merge
build-image.yml first.
This is cool because committers can simply push a branch to apache
and test if it works with some build image changes that otherwise
would require to push to `main` of an apache-airflow fork.
Another advantage is that merge builds do not run two separate
workflows - both building the image and running tests is done in
the same workflow (and the build-image workflow is not started)
---
.github/workflows/build-images.yml | 4 -
.github/workflows/ci.yml | 175 ++++++++++++++++++++++++++++++++++++-
2 files changed, 172 insertions(+), 7 deletions(-)
diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml
index e23b363..800466d 100644
--- a/.github/workflows/build-images.yml
+++ b/.github/workflows/build-images.yml
@@ -18,11 +18,7 @@
---
name: "Build Images"
on: # yamllint disable-line rule:truthy
- schedule:
- - cron: '28 0 * * *'
pull_request_target:
- push:
- branches: ['main', 'v[0-9]+-[0-9]+-test']
permissions:
# all other permissions are set to none
contents: read
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 2a6bde2..7181110 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -21,7 +21,6 @@ on: # yamllint disable-line rule:truthy
schedule:
- cron: '28 0 * * *'
push:
- branches: ['main', 'v[0-9]+-[0-9]+-test']
pull_request:
branches: ['main', 'v[0-9]+-[0-9]+-test', 'v[0-9]+-[0-9]+-stable']
permissions:
@@ -127,6 +126,10 @@ jobs:
env:
GITHUB_CONTEXT: ${{ toJson(github) }}
outputs:
+ targetBranch: ${{ steps.dynamic-outputs.outputs.targetBranch }}
+ defaultBranch: ${{ steps.selective-checks.outputs.default-branch }}
+ waitForImage: ${{ steps.wait-for-image.outputs.wait-for-image }}
+ allPythonVersions: ${{ steps.selective-checks.outputs.all-python-versions }}
upgradeToNewerDependencies: ${{ steps.selective-checks.outputs.upgrade-to-newer-dependencies }}
pythonVersions: ${{ steps.selective-checks.outputs.python-versions }}
pythonVersionsListAsString: ${{ steps.selective-checks.outputs.python-versions-list-as-string }}
@@ -165,6 +168,8 @@ jobs:
pullRequestLabels: ${{ steps.source-run-info.outputs.pullRequestLabels }}
runsOn: ${{ steps.set-runs-on.outputs.runsOn }}
runCoverage: ${{ steps.set-run-coverage.outputs.runCoverage }}
+ localBuild: ${{ steps.set-local-build.outputs.localBuild }}
+ buildJobDescription: ${{ steps.set-local-build.outputs.buildJobDescription }}
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
@@ -222,6 +227,170 @@ jobs:
github.ref == 'refs/heads/main' && github.repository == 'apache/airflow' &&
github.event_name == 'push' &&
steps.selective-checks.outputs.default-branch == 'main'
+ # Avoid having to specify the local build logic every time.
+ - name: Set local build
+ id: set-local-build
+ run: |
+ echo "::set-output name=localBuild::true"
+ echo "::set-output name=buildJobDescription::Build"
+ if: github.event_name == 'push' || github.event_name == 'schedule'
+ - name: Set remote build
+ id: set-remote-build
+ run: |
+ echo "::set-output name=localBuild::false"
+ echo "::set-output name=buildJobDescription::Skip (separate workflow)"
+ if: github.event_name != 'push' && github.event_name != 'schedule'
+
+ build-ci-images:
+ permissions:
+ packages: write
+ timeout-minutes: 80
+ name: "${{needs.build-info.outputs.buildJobDescription}} CI image ${{matrix.python-version}}"
+ runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
+ needs: [build-info]
+ strategy:
+ matrix:
+ python-version: ${{ fromJson(needs.build-info.outputs.allPythonVersions) }}
+ fail-fast: true
+ env:
+ RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn)[0] }}
+ BACKEND: sqlite
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
+ UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgradeToNewerDependencies }}
+ DOCKER_CACHE: ${{ needs.build-info.outputs.cacheDirective }}
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
+ GITHUB_REGISTRY_WAIT_FOR_IMAGE: "false"
+ outputs: ${{toJSON(needs.build-info.outputs) }}
+ steps:
+ - uses: actions/checkout@v2
+ with:
+ ref: ${{ needs.build-info.outputs.targetCommitSha }}
+ persist-credentials: false
+ submodules: recursive
+ if: needs.build-info.outputs.localBuild == 'true'
+ - run: python -m pip install --editable ./dev/breeze/
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Retrieve DEFAULTS from the _initialization.sh"
+ # We cannot "source" the script here because that would be a security problem (we cannot run
+ # any code that comes from the sources coming from the PR. Therefore we extract the
+ # DEFAULT_BRANCH and DEFAULT_CONSTRAINTS_BRANCH and DEBIAN_VERSION via custom grep/awk/sed commands
+ id: defaults
+ run: |
+ DEFAULT_BRANCH=$(grep "export DEFAULT_BRANCH" scripts/ci/libraries/_initialization.sh | \
+ awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
+ echo "DEFAULT_BRANCH=${DEFAULT_BRANCH}" >> $GITHUB_ENV
+ DEFAULT_CONSTRAINTS_BRANCH=$(grep "export DEFAULT_CONSTRAINTS_BRANCH" \
+ scripts/ci/libraries/_initialization.sh | \
+ awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
+ echo "DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH}" >> $GITHUB_ENV
+ DEBIAN_VERSION=$(grep "export DEBIAN_VERSION" scripts/ci/libraries/_initialization.sh | \
+ awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
+ echo "DEBIAN_VERSION=${DEBIAN_VERSION}" >> $GITHUB_ENV
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: >
+ Checkout "${{ needs.build-info.outputs.targetBranch }}" branch to 'main-airflow' folder
+ to use ci/scripts from there.
+ uses: actions/checkout@v2
+ with:
+ path: "main-airflow"
+ ref: "${{ needs.build-info.outputs.targetBranch }}"
+ persist-credentials: false
+ submodules: recursive
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Setup python"
+ uses: actions/setup-python@v2
+ with:
+ python-version: ${{ needs.build-info.outputs.defaultPythonVersion }}
+ - name: "Free space"
+ run: airflow-freespace
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Build CI image ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
+ run: Breeze2 build-ci-image
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Push CI image ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
+ run: ./scripts/ci/images/ci_push_ci_images.sh
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Candidates for pip resolver backtrack triggers: ${{ matrix.python-version }}"
+ if: failure() || cancelled()
+ run: airflow-find-newer-dependencies --max-age 1 --python "${{ matrix.python-version }}"
+
+ build-prod-images:
+ permissions:
+ packages: write
+ timeout-minutes: 80
+ name: "${{needs.build-info.outputs.buildJobDescription}} PROD image ${{matrix.python-version}}"
+ runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
+ needs: [build-info, build-ci-images]
+ strategy:
+ matrix:
+ python-version: ${{ fromJson(needs.build-info.outputs.allPythonVersions) }}
+ fail-fast: true
+ env:
+ RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn)[0] }}
+ BACKEND: sqlite
+ PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
+ UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgradeToNewerDependencies }}
+ DOCKER_CACHE: ${{ needs.build-info.outputs.cacheDirective }}
+ VERSION_SUFFIX_FOR_PYPI: ".dev0"
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
+ GITHUB_REGISTRY_WAIT_FOR_IMAGE: "false"
+ INSTALL_PROVIDERS_FROM_SOURCES: >
+ ${{ needs.build-info.outputs.defaultBranch == 'main' && 'true' || 'false' }}
+ steps:
+ - uses: actions/checkout@v2
+ with:
+ ref: ${{ needs.build-info.outputs.targetCommitSha }}
+ persist-credentials: false
+ submodules: recursive
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Retrieve DEFAULTS from the _initialization.sh"
+ # We cannot "source" the script here because that would be a security problem (we cannot run
+ # any code that comes from the sources coming from the PR. Therefore we extract the
+ # DEFAULT_BRANCH and DEFAULT_CONSTRAINTS_BRANCH and DEBIAN_VERSION via custom grep/awk/sed commands
+ id: defaults
+ run: |
+ DEFAULT_BRANCH=$(grep "export DEFAULT_BRANCH" scripts/ci/libraries/_initialization.sh | \
+ awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
+ echo "DEFAULT_BRANCH=${DEFAULT_BRANCH}" >> $GITHUB_ENV
+ DEFAULT_CONSTRAINTS_BRANCH=$(grep "export DEFAULT_CONSTRAINTS_BRANCH" \
+ scripts/ci/libraries/_initialization.sh | \
+ awk 'BEGIN{FS="="} {print $3}' | sed s'/["}]//g')
+ echo "DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH}" >> $GITHUB_ENV
+ DEBIAN_VERSION=$(grep "export DEBIAN_VERSION" scripts/ci/libraries/_initialization.sh | \
+ cut -d "=" -f 3 | sed s'/["}]//g')
+ echo "DEBIAN_VERSION=${DEBIAN_VERSION}" >> $GITHUB_ENV
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: >
+ Checkout "${{ needs.build-info.outputs.targetBranch }}" branch to 'main-airflow' folder
+ to use ci/scripts from there.
+ uses: actions/checkout@v2
+ with:
+ path: "main-airflow"
+ ref: "${{ needs.build-info.outputs.targetBranch }}"
+ persist-credentials: false
+ submodules: recursive
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Setup python"
+ uses: actions/setup-python@v2
+ with:
+ python-version: ${{ needs.build-info.outputs.defaultPythonVersion }}
+ if: needs.build-info.outputs.localBuild == 'true'
+ - run: python -m pip install --editable ./dev/breeze/
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Free space"
+ run: airflow-freespace
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Pull CI image for PROD ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
+ run: ./scripts/ci/images/ci_pull_ci_image_on_ci.sh
+ env:
+ GITHUB_REGISTRY_PULL_IMAGE_TAG: ${{ github.event.pull_request.head.sha || github.sha }}
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Build PROD image ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
+ run: ./scripts/ci/images/ci_build_prod_image_on_ci.sh
+ if: needs.build-info.outputs.localBuild == 'true'
+ - name: "Push PROD image ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
+ run: ./scripts/ci/images/ci_push_production_images.sh
+ if: needs.build-info.outputs.localBuild == 'true'
run-new-breeze-tests:
timeout-minutes: 10
@@ -367,7 +536,7 @@ jobs:
timeout-minutes: 120
name: "Wait for CI images"
runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
- needs: [build-info]
+ needs: [build-info, build-ci-images]
if: needs.build-info.outputs.image-build == 'true'
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
@@ -1102,7 +1271,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
timeout-minutes: 120
name: "Wait for PROD images"
runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
- needs: [build-info, ci-images]
+ needs: [build-info, ci-images, build-prod-images]
if: needs.build-info.outputs.image-build == 'true'
env:
RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}