You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@superset.apache.org by ru...@apache.org on 2026/06/03 05:14:09 UTC
(superset) branch ci/docker-registry-flake-hardening created (now 738563bc8ab)
This is an automated email from the ASF dual-hosted git repository.
rusackas pushed a change to branch ci/docker-registry-flake-hardening
in repository https://gitbox.apache.org/repos/asf/superset.git
at 738563bc8ab ci: harden CI against Docker Hub registry flakes (retries + auth)
This branch includes the following new commits:
new 738563bc8ab ci: harden CI against Docker Hub registry flakes (retries + auth)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(superset) 01/01: ci: harden CI against Docker Hub registry flakes (retries + auth)
Posted by ru...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rusackas pushed a commit to branch ci/docker-registry-flake-hardening
in repository https://gitbox.apache.org/repos/asf/superset.git
commit 738563bc8ab2fddda02fa61115a9ec943a56aa5c
Author: Evan <ev...@preset.io>
AuthorDate: Tue Jun 2 22:13:52 2026 -0700
ci: harden CI against Docker Hub registry flakes (retries + auth)
Transient Docker Hub pull failures ("context deadline exceeded", anonymous
rate-limit pressure on shared GitHub runner IPs) have been intermittently
failing several jobs on infrastructure hiccups rather than real problems —
check-python-deps, and the test-sqlite/test-postgres/test-mysql integration jobs.
Two complementary mitigations:
- Retry: pre-pull the python:slim image in scripts/uv-pip-compile.sh with a
small backoff loop before running, so a flaky pull no longer fails the whole
check-python-deps job. Works everywhere, including fork PRs.
- Auth: authenticate to Docker Hub so pulls use our higher authenticated rate
limit instead of the shared anonymous one. Adds a best-effort
(continue-on-error) docker/login-action to check-python-deps, and
`credentials:` to the mysql/postgres/redis service containers in the
integration-test workflow. On fork PRs the DOCKERHUB_* secrets are
unavailable, so login no-ops and the runner falls back to anonymous pulls
(covered by the retry loop) — no regression for external contributors.
Co-Authored-By: Claude Opus 4.8 <no...@anthropic.com>
---
.github/workflows/check-python-deps.yml | 13 ++++++++++++
.../workflows/superset-python-integrationtest.yml | 17 ++++++++++++++++
scripts/uv-pip-compile.sh | 23 +++++++++++++++++++++-
3 files changed, 52 insertions(+), 1 deletion(-)
diff --git a/.github/workflows/check-python-deps.yml b/.github/workflows/check-python-deps.yml
index 0201ea817cd..ca1d3eb20b6 100644
--- a/.github/workflows/check-python-deps.yml
+++ b/.github/workflows/check-python-deps.yml
@@ -38,6 +38,19 @@ jobs:
if: steps.check.outputs.python
uses: ./.github/actions/setup-backend/
+ # Authenticate the Docker daemon so the python:slim pull in
+ # uv-pip-compile.sh uses our (much higher) authenticated rate limit
+ # instead of the shared-runner anonymous one. Best-effort: on fork PRs the
+ # secrets are unavailable, so this no-ops and the pull falls back to
+ # anonymous (covered by the retry loop in the script).
+ - name: Login to Docker Hub
+ if: steps.check.outputs.python
+ continue-on-error: true
+ uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
+ with:
+ username: ${{ secrets.DOCKERHUB_USER }}
+ password: ${{ secrets.DOCKERHUB_TOKEN }}
+
- name: Run uv
if: steps.check.outputs.python
run: ./scripts/uv-pip-compile.sh
diff --git a/.github/workflows/superset-python-integrationtest.yml b/.github/workflows/superset-python-integrationtest.yml
index 580d87fcaaf..5cfcc056fb6 100644
--- a/.github/workflows/superset-python-integrationtest.yml
+++ b/.github/workflows/superset-python-integrationtest.yml
@@ -27,6 +27,11 @@ jobs:
services:
mysql:
image: mysql:8.0
+ # Authenticated pulls use our higher Docker Hub rate limit. Empty on
+ # fork PRs (secrets unavailable) -> runner falls back to anonymous.
+ credentials:
+ username: ${{ secrets.DOCKERHUB_USER }}
+ password: ${{ secrets.DOCKERHUB_TOKEN }}
env:
MYSQL_ROOT_PASSWORD: root
ports:
@@ -38,6 +43,9 @@ jobs:
--health-retries=5
redis:
image: redis:7-alpine
+ credentials:
+ username: ${{ secrets.DOCKERHUB_USER }}
+ password: ${{ secrets.DOCKERHUB_TOKEN }}
options: --entrypoint redis-server
ports:
- 16379:6379
@@ -121,6 +129,9 @@ jobs:
services:
postgres:
image: postgres:17-alpine
+ credentials:
+ username: ${{ secrets.DOCKERHUB_USER }}
+ password: ${{ secrets.DOCKERHUB_TOKEN }}
env:
POSTGRES_USER: superset
POSTGRES_PASSWORD: superset
@@ -130,6 +141,9 @@ jobs:
- 15432:5432
redis:
image: redis:7-alpine
+ credentials:
+ username: ${{ secrets.DOCKERHUB_USER }}
+ password: ${{ secrets.DOCKERHUB_TOKEN }}
ports:
- 16379:6379
steps:
@@ -186,6 +200,9 @@ jobs:
services:
redis:
image: redis:7-alpine
+ credentials:
+ username: ${{ secrets.DOCKERHUB_USER }}
+ password: ${{ secrets.DOCKERHUB_TOKEN }}
ports:
- 16379:6379
steps:
diff --git a/scripts/uv-pip-compile.sh b/scripts/uv-pip-compile.sh
index 4f715fe3fd4..a3ba7870adc 100755
--- a/scripts/uv-pip-compile.sh
+++ b/scripts/uv-pip-compile.sh
@@ -31,11 +31,32 @@ if [ -z "$RUNNING_IN_DOCKER" ]; then
echo "Running in Docker (Python ${PYTHON_VERSION} on Linux)..."
+ IMAGE="python:${PYTHON_VERSION}-slim"
+
+ # Pre-pull the image with a few retries to absorb transient Docker Hub
+ # registry failures ("context deadline exceeded" / anonymous rate-limit blips
+ # on shared CI runners). Without this a flaky pull fails the whole
+ # check-python-deps job on an infrastructure hiccup rather than a real
+ # dependency drift. The pull is in the `until` condition so `set -e` does not
+ # abort on an individual failed attempt.
+ attempt=1
+ max_attempts=4
+ until docker pull "$IMAGE"; do
+ if [ "$attempt" -ge "$max_attempts" ]; then
+ echo "docker pull $IMAGE failed after ${max_attempts} attempts" >&2
+ exit 1
+ fi
+ delay=$((attempt * 10))
+ echo "docker pull $IMAGE failed (attempt ${attempt}/${max_attempts}); retrying in ${delay}s..." >&2
+ sleep "$delay"
+ attempt=$((attempt + 1))
+ done
+
docker run --rm \
-v "$(pwd)":/app \
-w /app \
-e RUNNING_IN_DOCKER=1 \
- python:${PYTHON_VERSION}-slim \
+ "$IMAGE" \
bash -c "pip install uv && ./scripts/uv-pip-compile.sh $*"
exit $?