You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2026/04/19 13:24:14 UTC
[PR] KPO: treat registry 5xx errors as transient during pod startup [airflow]
potiuk opened a new pull request, #65490:
URL: https://github.com/apache/airflow/pull/65490
When a pod is starting, kubelet automatically retries failed image pulls with exponential backoff. `KubernetesPodOperator` monitors the pod via `detect_pod_terminate_early_issues()` in `pod_manager.py` and aborts early on what it deems a fatal pull error — but the transient-error pattern list did not cover registry 5xx / gateway outages. A short Docker Hub outage (502/503/504 from `auth.docker.io` or the registry) was being classified as fatal and would abort the pod before kubelet's next retry, even though the pull would have succeeded once upstream recovered.
Observed in [this run](https://github.com/apache/airflow/actions/runs/24614489670/job/71977719976) where `test_pod_hostnetwork` failed with:
```
failed to authorize: failed to fetch anonymous token:
unexpected status from GET https://auth.docker.io/token?...: 502 Bad Gateway
→ PodLaunchFailedException: Image cannot be pulled, unable to start: ErrImagePull
```
## Changes
- Add canonical gateway error phrases (`bad gateway`, `service unavailable`, `gateway timeout`) to `TRANSIENT_ERROR_PATTERNS` so these conditions let kubelet keep retrying within the caller's `startup_timeout` budget.
- Also treat kubelet's own `ImagePullBackOff` `back-off pulling` cooldown message as transient — without it we would still bail the moment kubelet moves into its retry backoff state.
- Add direct unit tests for `detect_pod_terminate_early_issues` covering both the new transient patterns and the fatal cases.
- Update the existing `test_await_pod_completion_breaks_on_early_termination_issue` to use `InvalidImageName` (still fatal) instead of the now-transient `ImagePullBackOff / back-off pulling` combo.
`startup_timeout` still bounds the overall wait, and genuinely fatal pull errors (`InvalidImageName`, `ErrImageNeverPull`, `manifest unknown`, `unauthorized`, …) remain fatal.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 4.7, 1M context)
Generated-by: Claude Code (Opus 4.7, 1M context) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KPO: treat registry 5xx errors as transient during pod startup [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #65490:
URL: https://github.com/apache/airflow/pull/65490
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KPO: treat registry 5xx errors as transient during pod startup [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #65490:
URL: https://github.com/apache/airflow/pull/65490#discussion_r3106894206
##########
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/utils/pod_manager.py:
##########
@@ -232,6 +232,16 @@ def detect_pod_terminate_early_issues(pod: V1Pod) -> str | None:
"temporarily unavailable",
"timeout",
"account limit",
+ # Upstream registry/auth outages (e.g. Docker Hub auth returning 5xx).
+ # kubelet will retry automatically; the caller's startup_timeout still
+ # bounds the total wait.
+ "bad gateway",
+ "service unavailable",
+ "gateway timeout",
+ # kubelet's canonical message while it is retrying a failed pull with
+ # exponential backoff — treat as transient so we don't bail before
+ # kubelet gets another attempt.
+ "back-off pulling",
Review Comment:
Need to check, I _think_ this was explicitly not added because same is reported if image is not existing. In case of such image is not existing we want to fail fast and not have an endless retry (K8s would retry endless).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KPO: treat registry 5xx errors as transient during pod startup [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #65490:
URL: https://github.com/apache/airflow/pull/65490#discussion_r3106906990
##########
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/utils/pod_manager.py:
##########
@@ -232,6 +232,16 @@ def detect_pod_terminate_early_issues(pod: V1Pod) -> str | None:
"temporarily unavailable",
"timeout",
"account limit",
+ # Upstream registry/auth outages (e.g. Docker Hub auth returning 5xx).
+ # kubelet will retry automatically; the caller's startup_timeout still
+ # bounds the total wait.
+ "bad gateway",
+ "service unavailable",
+ "gateway timeout",
+ # kubelet's canonical message while it is retrying a failed pull with
+ # exponential backoff — treat as transient so we don't bail before
+ # kubelet gets another attempt.
+ "back-off pulling",
Review Comment:
Good point. Let me check it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KPO: treat registry 5xx errors as transient during pod startup [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #65490:
URL: https://github.com/apache/airflow/pull/65490#discussion_r3106917209
##########
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/utils/pod_manager.py:
##########
@@ -232,6 +232,16 @@ def detect_pod_terminate_early_issues(pod: V1Pod) -> str | None:
"temporarily unavailable",
"timeout",
"account limit",
+ # Upstream registry/auth outages (e.g. Docker Hub auth returning 5xx).
+ # kubelet will retry automatically; the caller's startup_timeout still
+ # bounds the total wait.
+ "bad gateway",
+ "service unavailable",
+ "gateway timeout",
+ # kubelet's canonical message while it is retrying a failed pull with
+ # exponential backoff — treat as transient so we don't bail before
+ # kubelet gets another attempt.
+ "back-off pulling",
Review Comment:
You're right — verified against kubelet source:
- **kubelet ≤ 1.31** (`pkg/kubelet/images/image_manager.go`): `ImagePullBackOff` message is just `Back-off pulling image "X"` with no underlying error, so blanket-matching `back-off pulling` would have caused a 120s wait on a missing image.
- **kubelet ≥ 1.32**: The message was changed to `Back-off pulling image "X": <prev_error>` (e.g. `: 502 Bad Gateway`).
Dropped the unconditional `back-off pulling` match. The `bad gateway` / `service unavailable` / `gateway timeout` patterns still cover the real-world failure: on k8s ≥ 1.32 they match both the initial `ErrImagePull` and the follow-on `ImagePullBackOff` (because the 5xx error is appended to the message). On older kubelets we still fail fast on the bare `ImagePullBackOff` — same behaviour as before the PR for missing images. Added a fail-fast test case for that bare message too.
Force-pushed the amend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org