You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2023/11/18 23:26:09 UTC
(airflow) branch optimize-prod-image-elapsed-time created (now 7966e4dd2d)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch optimize-prod-image-elapsed-time
in repository https://gitbox.apache.org/repos/asf/airflow.git
at 7966e4dd2d Decrease delay between CI and PROD image build jobs
This branch includes the following new commits:
new 7966e4dd2d Decrease delay between CI and PROD image build jobs
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(airflow) 01/01: Decrease delay between CI and PROD image build jobs
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch optimize-prod-image-elapsed-time
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 7966e4dd2d68d2835ed04bfe842fe7415d7cce13
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Nov 19 00:09:11 2023 +0100
Decrease delay between CI and PROD image build jobs
The PROD image is built based on source constraints that are constraints
based on depdencies available in the CI image. We have a separate job
generating all constraints (source, PyPI, No providers).
Generation of Source constraints is very fast (seconds) but generation
of PyPI and No Providers constraints is a lot longer because it requires
to remove (and reinstalling) of provider packages from PyPI and it takes
~15 minutes - but we could start it way faster if we do not wait for the
non-source constraints and let them run in parallel.
This PR optimizes it:
* source constraints are also generated right after images are built and
separately uploaded as source-constraints artifacts
* PROD image build uses the source constraints and the PROD image job
does not have to wait for the "preview-costraints" job.
---
.github/actions/build-ci-images/action.yml | 11 +++++++++++
.github/actions/build-prod-images/action.yml | 2 +-
.github/workflows/build-images.yml | 14 --------------
.github/workflows/ci.yml | 4 ++--
4 files changed, 14 insertions(+), 17 deletions(-)
diff --git a/.github/actions/build-ci-images/action.yml b/.github/actions/build-ci-images/action.yml
index e93b2f4e71..d8c0216bb4 100644
--- a/.github/actions/build-ci-images/action.yml
+++ b/.github/actions/build-ci-images/action.yml
@@ -34,6 +34,17 @@ runs:
- name: "Build & Push AMD64 CI images ${{ env.IMAGE_TAG }} ${{ env.PYTHON_VERSIONS }}"
shell: bash
run: breeze ci-image build --push --tag-as-latest --run-in-parallel --upgrade-on-failure
+ - name: "Source constraints"
+ shell: bash
+ run: >
+ breeze release-management generate-constraints --run-in-parallel
+ --airflow-constraints-mode constraints-source-providers
+ - name: "Upload constraint artifacts"
+ uses: actions/upload-artifact@v3
+ with:
+ name: source-constraints
+ path: ./files/constraints-*/constraints-*.txt
+ retention-days: 7
- name: "Fix ownership"
shell: bash
run: breeze ci fix-ownership
diff --git a/.github/actions/build-prod-images/action.yml b/.github/actions/build-prod-images/action.yml
index 74667f0de1..4f1530b56a 100644
--- a/.github/actions/build-prod-images/action.yml
+++ b/.github/actions/build-prod-images/action.yml
@@ -52,7 +52,7 @@ runs:
- name: "Download constraints from the CI build"
uses: actions/download-artifact@v3
with:
- name: constraints
+ name: source-constraints
path: ./docker-context-files
- name: "Build & Push PROD images with source providers ${{ env.IMAGE_TAG }}:${{ env.PYTHON_VERSIONS }}"
shell: bash
diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml
index 952ece5fe8..5872978158 100644
--- a/.github/workflows/build-images.yml
+++ b/.github/workflows/build-images.yml
@@ -237,20 +237,6 @@ jobs:
PYTHON_VERSIONS: ${{needs.build-info.outputs.all-python-versions-list-as-string}}
DEBUG_RESOURCES: ${{ needs.build-info.outputs.debug-resources }}
BUILD_TIMEOUT_MINUTES: 70
- - name: "Source constraints"
- shell: bash
- run: >
- breeze release-management generate-constraints --run-in-parallel
- --airflow-constraints-mode constraints-source-providers
- env:
- PYTHON_VERSIONS: ${{needs.build-info.outputs.all-python-versions-list-as-string}}
- DEBUG_RESOURCES: ${{ needs.build-info.outputs.debug-resources }}
- - name: "Upload constraint artifacts"
- uses: actions/upload-artifact@v3
- with:
- name: constraints
- path: ./files/constraints-*/constraints-*.txt
- retention-days: 7
build-prod-images:
permissions:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 7a9ffb2251..1f0240eddc 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -395,7 +395,7 @@ jobs:
${{needs.build-info.outputs.build-job-description}} PROD images
${{needs.build-info.outputs.all-python-versions-list-as-string}}
runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
- needs: [build-info, build-ci-images, generate-constraints]
+ needs: [build-info, build-ci-images]
env:
DEFAULT_BRANCH: ${{ needs.build-info.outputs.default-branch }}
DEFAULT_CONSTRAINTS_BRANCH: ${{ needs.build-info.outputs.default-constraints-branch }}
@@ -445,7 +445,7 @@ jobs:
${{needs.build-info.outputs.build-job-description}} Bullseye PROD images
${{needs.build-info.outputs.all-python-versions-list-as-string}}
runs-on: ${{fromJSON(needs.build-info.outputs.runs-on)}}
- needs: [build-info, build-ci-images, generate-constraints]
+ needs: [build-info, build-ci-images]
if: needs.build-info.outputs.canary-run == 'true'
env:
DEFAULT_BRANCH: ${{ needs.build-info.outputs.default-branch }}