You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2026/05/05 14:35:15 UTC
[PR] Pin Docker Hub test images against K8s system-test rate-limit flakes [airflow]
potiuk opened a new pull request, #66423:
URL: https://github.com/apache/airflow/pull/66423
## Summary
The scheduled K8s system-test job intermittently fails because multiple
test pods pull `alpine:latest` (xcom sidecar default), `busybox:latest`,
and `ubuntu:latest` from Docker Hub anonymously and trip the
100-pulls-per-6h limit. Without a tag, kubelet defaults `imagePullPolicy`
to `Always`, so even nodes with the image cached re-pull every run.
Recent example: https://github.com/apache/airflow/actions/runs/25365187430/job/74380551079
## What changed
1. **Production default** — `xcom_sidecar.PodDefaults.SIDECAR_CONTAINER`
now uses `alpine:3.23` via a new `XCOM_SIDECAR_IMAGE` constant.
Tagged → kubelet defaults to `imagePullPolicy: IfNotPresent`.
Documented in the cncf-kubernetes provider changelog.
2. **Test pin** — every bare `image="ubuntu"` / `"busybox"` / `"alpine"`
in `kubernetes-tests/` and in `providers/cncf/kubernetes/tests/...`
is pinned (`ubuntu:24.04`, `busybox:1.37`, `alpine:3.23`).
3. **Pre-load into kind** — new `_preload_test_images_to_kind()` in
`breeze k8s`, called from `_run_complete_tests` after
`_upload_k8s_image`. Pulls each image on the runner with
exponential-backoff retries on Docker Hub 429s, then `kind load
docker-image` puts it on every node.
4. **Auto-tracker** — `scripts/ci/prek/upgrade_important_versions.py`
gains `UPGRADE_ALPINE` / `UPGRADE_BUSYBOX` flags, regex patterns for
`alpine:` / `busybox:` / chart `ALPINE_VERSION` ARGs, plus the
relevant call-sites added to `FILES_TO_UPDATE`. The next "Upgrade
important CI environment" run keeps these pins fresh. Ubuntu is
intentionally manual (interim releases beat LTS in semver-sort).
Drive-by: `# type: ignore[no-redef]` on the `tomli as tomllib` fallback
in `dev/registry/extract_{metadata,versions}.py` so `mypy-dev` passes on
edits to anything under `dev/`. The same fix is in PR #66314 — whichever
lands first, the other becomes a no-op rebase.
## Test plan
- [ ] Unit: `uv --project providers/cncf/kubernetes run pytest providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod.py -k xcom_sidecar_container_image_default -xvs` — passes locally.
- [ ] CI: K8s scheduled job (the one currently flaking) passes once this lands.
- [ ] CI: provider unit tests still green.
- [ ] Manual: `prek run --files <changed files>` — clean.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 4.7)
Generated-by: Claude Code (Opus 4.7) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pin Docker Hub test images against K8s system-test rate-limit flakes [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #66423:
URL: https://github.com/apache/airflow/pull/66423#issuecomment-4393668054
### Backport failed to create: v3-2-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/25472079279'> Run details </a>
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>❌</td>
<td>v3-2-test</td>
<td><a href="https://github.com/apache/airflow/commit/95bbf6ace25c013a67962c078ea7be75f7ac2190"><img src='https://img.shields.io/badge/Commit-95bbf6a-red' alt='Commit Link'></a></td>
</tr>
</table>
You can attempt to backport this manually by running:
```bash
cherry_picker 95bbf6a v3-2-test
```
This should apply the commit to the v3-2-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.
After you have resolved the conflicts, you can continue the backport process by running:
```bash
cherry_picker --continue
```
If you don't have cherry-picker installed, see the [installation guide](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#how-to-backport-pr-with-cherry-picker-cli).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pin Docker Hub test images against K8s system-test rate-limit flakes [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #66423:
URL: https://github.com/apache/airflow/pull/66423
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pin Docker Hub test images against K8s system-test rate-limit flakes [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #66423:
URL: https://github.com/apache/airflow/pull/66423#discussion_r3194467285
##########
kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:
##########
@@ -218,7 +218,7 @@ def test_config_path_move(self, kubeconfig_path, tmp_path):
def test_working_pod(self):
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
Review Comment:
I think using global variables is overrated in the age of AI - especially in tests - and especially that we have "prek" hook that will update them for us :).
In tests DAMP is often better than DRY - this has been widely recognized fact even before AI agents - and there are even less arguments now to extract constrants in tests, when your agent will automatically keep it consistent (And in our case we even have determinitic hook that will do it for you).
This article has good explanation why:
https://enterprisecraftsmanship.com/posts/dry-damp-unit-tests/
My simple translation of DAMP vs. DRY is "You should not need to look elsewhere to understand what the test does" - and this also means that tests should not really use constants (unless this is some complex calculation) - declared elsewhere especially if the constant names do not describe what it is.
In this case - if I apply that rule - I would have to declare that constant as
```python
UBUNITU_24_04 = "ubuntu:24.04"
```
In order to know what I am looking at.
If I do that instead:
```python
IMAGE="ubuntu:24.04"
```
The test will loos some information - which image we run it on, and you have to look elsewhere to find out - this is not good for human and also it's not good for `AGENTS` - because agents cannot determine the whole context of what the test does by just grepping the test and looking at it's definition - it will also have to look at the constant definition - so basically it will have to parse the whole test in order to find out what image we are running the test with. Which is more tokens and time.
Performance-wise - all those strings are interned anyway (there is one string in Python memory "ubuntu:24.04" that is reused because strings in Python are immutable and interned by default. So we are not loosing performance/memory here as well.
So ... I think this is is pretty good idea to leave the explicit image names as strings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pin Docker Hub test images against K8s system-test rate-limit flakes [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #66423:
URL: https://github.com/apache/airflow/pull/66423#discussion_r3194467285
##########
kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:
##########
@@ -218,7 +218,7 @@ def test_config_path_move(self, kubeconfig_path, tmp_path):
def test_working_pod(self):
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
Review Comment:
I think using global variables is overrated in the age of AI - especially in tests - and especially that we have "prek" hook that will update them for us :).
In tests DAMP is often better than DRY - this has been widely recognized fact even before AI agents - and there are even less arguments now to extract constrants in tests, when your agent will automatically keep it consistent (And in our case we even have determinitic hook that will do it for you).
This article has good explanation why:
https://enterprisecraftsmanship.com/posts/dry-damp-unit-tests/
My simple translation of DAMP vs. DRY is "You should not need to look elsewhere to understand what the test does" - and this also means that tests should not really use constants (unless this is some complex calculation) - declared elsewhere especially if the constant names do not describe what it is.
In this case - if I apply that rule - I would have to declare that constant as
```python
UBUNITU_24_04 = "ubuntu:24.04"
```
(and to be honest - that makes very little sense and no saving)
In order to know what I am looking at.
If I do that instead:
```python
IMAGE="ubuntu:24.04"
```
The test will loos some information - which image we run it on, and you have to look elsewhere to find out - this is not good for human and also it's not good for `AGENTS` - because agents cannot determine the whole context of what the test does by just grepping the test and looking at it's definition - it will also have to look at the constant definition - so basically it will have to parse the whole test in order to find out what image we are running the test with. Which is more tokens and time.
Performance-wise - all those strings are interned anyway (there is one string in Python memory "ubuntu:24.04" that is reused because strings in Python are immutable and interned by default. So we are not loosing performance/memory here as well.
So ... I think this is is pretty good idea to leave the explicit image names as strings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Pin Docker Hub test images against K8s system-test rate-limit flakes [airflow]
Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on code in PR #66423:
URL: https://github.com/apache/airflow/pull/66423#discussion_r3190224199
##########
kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py:
##########
@@ -218,7 +218,7 @@ def test_config_path_move(self, kubeconfig_path, tmp_path):
def test_working_pod(self):
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
Review Comment:
Can we make this a global variable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org