You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by js...@apache.org on 2025/03/15 19:06:39 UTC
(airflow) branch v2-10-test updated: Bugfix/fix disk space issues in GitHub (#47816)
This is an automated email from the ASF dual-hosted git repository.
jscheffl pushed a commit to branch v2-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/v2-10-test by this push:
new 6d42ccec754 Bugfix/fix disk space issues in GitHub (#47816)
6d42ccec754 is described below
commit 6d42ccec754e84a96e016bbc20e29c014003b1f4
Author: Jens Scheffler <95...@users.noreply.github.com>
AuthorDate: Sat Mar 15 20:06:32 2025 +0100
Bugfix/fix disk space issues in GitHub (#47816)
* Fix disk space issues on github runners (#46358)
* Triage disk space issues, DO NOT MERGE
* Triage disk space issues, DO NOT MERGE
* Triage disk space issues, DO NOT MERGE
* Triage disk space issues, DO NOT MERGE
* Triage disk space issues, DO NOT MERGE
* Triage disk space issues, DO NOT MERGE
* Triage disk space issues, DO NOT MERGE
* Move docker storage to second drive in general
* Cleanup debug and triaging stuff
* Exception of docker volume for constraints building
* Fix ownership of files that docker uses in mounted directories (#46428)
The #46358 moved docker to another mounted directory - but this
directory and all files in it are owned by host user. The directory
and all files inside should be owned by root in order to properly
reflect permissions of the files when building docker images.
The change is now simplified. Rather than passing mount directory
by variable and passing it through GitHub Actions, we hard-code
the location of docker in cleanup_docker.sh script - we also
incorporate changing ownership and showing disk space in the same
cleanup_docker.sh script and make sure that script is only called
in the "real" (not composite) actions at the beginning - right
after the repository is checked out - previously that script
was also called in composite actions and changing the repo to be
writeable was done AFTER cleanup_docker.sh - which would not
work as we want the /mnt directory to be still owned by the
host user, but the docker storage should be still owned by root.
---------
Co-authored-by: Jarek Potiuk <ja...@potiuk.com>
---
.github/actions/prepare_all_ci_images/action.yml | 6 +++---
.github/actions/prepare_breeze_and_image/action.yml | 6 ------
.github/actions/prepare_single_ci_image/action.yml | 6 ------
.github/workflows/ci-image-build.yml | 6 ------
.github/workflows/generate-constraints.yml | 4 ++++
.github/workflows/prod-image-build.yml | 6 ------
scripts/ci/cleanup_docker.sh | 12 ++++++++++++
7 files changed, 19 insertions(+), 27 deletions(-)
diff --git a/.github/actions/prepare_all_ci_images/action.yml b/.github/actions/prepare_all_ci_images/action.yml
index d156818b9b2..76c00a72a39 100644
--- a/.github/actions/prepare_all_ci_images/action.yml
+++ b/.github/actions/prepare_all_ci_images/action.yml
@@ -22,15 +22,15 @@ inputs:
python-versions-list-as-string:
description: 'Stringified array of all Python versions to test - separated by spaces.'
required: true
+ docker-volume-location:
+ description: File system location where to move docker space to
+ default: /mnt/var-lib-docker
platform:
description: 'Platform for the build - linux/amd64 or linux/arm64'
required: true
runs:
using: "composite"
steps:
- - name: "Cleanup docker"
- run: ./scripts/ci/cleanup_docker.sh
- shell: bash
# TODO: Currently we cannot loop through the list of python versions and have dynamic list of
# tasks. Instead we hardcode all possible python versions and they - but
# this should be implemented in stash action as list of keys to download.
diff --git a/.github/actions/prepare_breeze_and_image/action.yml b/.github/actions/prepare_breeze_and_image/action.yml
index 26be0b76315..7332afc7eb2 100644
--- a/.github/actions/prepare_breeze_and_image/action.yml
+++ b/.github/actions/prepare_breeze_and_image/action.yml
@@ -46,12 +46,6 @@ runs:
with:
use-uv: ${{ inputs.use-uv }}
id: breeze
- - name: Check free space
- run: df -H
- shell: bash
- - name: Make /mnt/ directory writeable
- run: sudo chown -R ${USER} /mnt
- shell: bash
- name: "Restore ${{ inputs.image-type }} docker image ${{ inputs.platform }}:${{ inputs.python }}"
uses: apache/infrastructure-actions/stash/restore@c94b890bbedc2fc61466d28e6bd9966bc6c6643c
with:
diff --git a/.github/actions/prepare_single_ci_image/action.yml b/.github/actions/prepare_single_ci_image/action.yml
index ecae9f802c9..7250dbea4b8 100644
--- a/.github/actions/prepare_single_ci_image/action.yml
+++ b/.github/actions/prepare_single_ci_image/action.yml
@@ -35,12 +35,6 @@ inputs:
runs:
using: "composite"
steps:
- - name: Check free space
- run: df -H
- shell: bash
- - name: Make /mnt/ directory writeable
- run: sudo chown -R ${USER} /mnt
- shell: bash
- name: "Restore CI docker images ${{ inputs.platform }}:${{ inputs.python }}"
uses: apache/infrastructure-actions/stash/restore@c94b890bbedc2fc61466d28e6bd9966bc6c6643c
with:
diff --git a/.github/workflows/ci-image-build.yml b/.github/workflows/ci-image-build.yml
index 9283dc06b93..81ac6737fed 100644
--- a/.github/workflows/ci-image-build.yml
+++ b/.github/workflows/ci-image-build.yml
@@ -175,12 +175,6 @@ jobs:
PUSH: ${{ inputs.push-image }}
VERBOSE: "true"
PLATFORM: ${{ inputs.platform }}
- - name: Check free space
- run: df -H
- shell: bash
- - name: Make /mnt/ directory writeable
- run: sudo chown -R ${USER} /mnt
- shell: bash
- name: "Export CI docker image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}"
env:
PLATFORM: ${{ inputs.platform }}
diff --git a/.github/workflows/generate-constraints.yml b/.github/workflows/generate-constraints.yml
index 740310e1cc0..3c502d17bc6 100644
--- a/.github/workflows/generate-constraints.yml
+++ b/.github/workflows/generate-constraints.yml
@@ -68,6 +68,9 @@ jobs:
uses: actions/checkout@v4
with:
persist-credentials: false
+ - name: "Cleanup docker"
+ run: ./scripts/ci/cleanup_docker.sh
+ shell: bash
- name: "Install Breeze"
uses: ./.github/actions/breeze
with:
@@ -78,6 +81,7 @@ jobs:
with:
platform: "linux/amd64"
python-versions-list-as-string: ${{ inputs.python-versions-list-as-string }}
+ docker-volume-location: "" # TODO(jscheffl): Understand why it fails here and fix it
- name: "Verify all CI images ${{ inputs.python-versions-list-as-string }}"
run: breeze ci-image verify --run-in-parallel
- name: "Source constraints"
diff --git a/.github/workflows/prod-image-build.yml b/.github/workflows/prod-image-build.yml
index 470c149d26e..cdab2b4f0c0 100644
--- a/.github/workflows/prod-image-build.yml
+++ b/.github/workflows/prod-image-build.yml
@@ -278,12 +278,6 @@ jobs:
if: inputs.build-provider-packages != 'true'
- name: "Verify PROD image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}"
run: breeze prod-image verify
- - name: Check free space
- run: df -H
- shell: bash
- - name: Make /mnt/ directory writeable
- run: sudo chown -R ${USER} /mnt
- shell: bash
- name: "Export PROD docker image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}"
env:
PLATFORM: ${{ inputs.platform }}
diff --git a/scripts/ci/cleanup_docker.sh b/scripts/ci/cleanup_docker.sh
index 400cc501541..3327145c8f0 100755
--- a/scripts/ci/cleanup_docker.sh
+++ b/scripts/ci/cleanup_docker.sh
@@ -15,10 +15,22 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+
+
function cleanup_docker {
+ local target_docker_volume_location="/mnt/var-lib-docker"
+ echo "Checking free space!"
+ df -H
+ echo "Making sure that /mnt is writeable"
+ sudo chown -R "${USER}" /mnt
# This is faster than docker prune
+ echo "Stopping docker"
sudo systemctl stop docker
sudo rm -rf /var/lib/docker
+ echo "Mounting ${target_docker_volume_location} to /var/lib/docker"
+ sudo mkdir -p "${target_docker_volume_location}" /var/lib/docker
+ sudo mount --bind "${target_docker_volume_location}" /var/lib/docker
+ sudo chown -R 0:0 "${target_docker_volume_location}"
sudo systemctl start docker
}