You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "aru-trackunit (via GitHub)" <gi...@apache.org> on 2026/09/16 09:42:14 UTC
[I] Kubernetes Infrastructure failures consume retry attempts. [airflow]
aru-trackunit opened a new issue, #73238:
URL: https://github.com/apache/airflow/issues/73238
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.3.1
### What happened and how to reproduce it?
Executor KubernetesExecutor(parallelism=60) reported that the task instance <TaskInstance: xxxx scheduled__2026-09-16T07:00:00+00:00 map_index=5 [queued] ti_id=01a0a95a-37ed-7177-b212-4e25272f5c2a> finished with state failed, but the task instance's state attribute is queued. Learn more: https://airflow.apache.org/docs/apache-airflow/stable/troubleshooting.html#task-state-changed-externally Extra info: Pod failed because of None [airflow.task] loc=taskinstance.py:1841"
On the screenshot below it says that first attempt is not visible, and it succeeded performing only 2nd attempt.
<img width="790" height="306" alt="Image" src="https://github.com/user-attachments/assets/10befa80-9662-4799-b0a6-a683115f43fe" />
### What you think should happen instead?
It would be valuable to support different retry policies based on the cause of a failure, particularly for application failures versus infrastructure or connectivity failures.
Task authors may want deterministic application errors to fail immediately, since rerunning the same code with the same input is unlikely to succeed. They may also need to disable retries when an operation is non-idempotent or otherwise unsafe to repeat.
However, disabling retries entirely is risky. A task can fail before the operation even begins—for example, because its worker pod cannot start—or because of a transient network problem or temporary third-party service outage. These failures may be safe and worthwhile to retry.
Conversely, applying one retry policy to every failure can cause deterministic application errors to run repeatedly when they should fail immediately.
Ideally, Airflow would allow different behavior for at least the following categories:
Application failures that should fail immediately
Infrastructure or pre-execution failures that should be retried without consuming the application retry budget
Transient connectivity or third-party failures that should follow a configurable retry policy
I am not sure what the ideal implementation would look like, but representing application errors, infrastructure failures, and third-party timeouts with the same generic failed state makes it difficult to choose a safe and effective retry strategy.
Apologies for covering several concerns in one ticket, but I believe they are closely related: without knowing why a task failed, Airflow cannot determine—or allow users to configure—the appropriate retry behavior.
Related:
https://github.com/apache/airflow/issues/73164
https://github.com/apache/airflow/issues/69052
https://github.com/apache/airflow/pull/66405
### Operating System
Debian GNU/Linux 12 (bookworm)
### Deployment
Official Apache Airflow Helm Chart
### Apache Airflow Provider(s)
cncf-kubernetes
### Versions of Apache Airflow Providers
```
apache-airflow-providers-amazon==9.34.0
apache-airflow-providers-cncf-kubernetes==10.21.0
apache-airflow-providers-common-compat==1.18.0
apache-airflow-providers-common-io==1.8.0
apache-airflow-providers-common-sql==2.1.0
apache-airflow-providers-databricks==7.18.1
apache-airflow-providers-fab==3.8.0
apache-airflow-providers-github==2.11.3
apache-airflow-providers-hashicorp==4.8.0
apache-airflow-providers-http==6.0.5
apache-airflow-providers-microsoft-mssql==4.7.0
apache-airflow-providers-mysql==6.6.1
apache-airflow-providers-postgres==7.0.1
apache-airflow-providers-sftp==6.0.1
apache-airflow-providers-slack==9.10.2
apache-airflow-providers-smtp==3.0.3
apache-airflow-providers-ssh==6.0.1
apache-airflow-providers-standard==1.17.0
```
### Official Helm Chart version
1.22.0 (latest released)
### Kubernetes Version
1.34.9
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kubernetes Infrastructure failures consume retry attempts. [airflow]
Posted by "kartik00052 (via GitHub)" <gi...@apache.org>.
kartik00052 commented on issue #73238:
URL: https://github.com/apache/airflow/issues/73238#issuecomment-5773032340
Reading through the retry/failure paths after seeing this issue and #73395 — a couple of things worth pointing at directly with code references, plus a possible way to close the remaining gap.
**A bounded, separately-counted infra retry already exists for one case.**
`KubernetesExecutor._is_pre_execution_failure()` (in `kubernetes_executor.py`) already distinguishes a pod that failed *before* the task instance reached `RUNNING` from one that failed after — it checks `ti_state == QUEUED` at failure time, and if that holds, the pod is requeued on its own counter (`pod_launch_failure_max_retries`) without ever touching `task.retries`. That's exactly the "infra failure shouldn't burn the application's retry budget" behavior this issue is asking for — it just only covers pod-launch failures.
**The mid-execution case — the one in this issue's screenshot — doesn't get that treatment.**
Once a task instance is `RUNNING` and gets killed externally (pod eviction, node drain, SIGTERM), that's the case #73395 is fixing: right now on `main`, `_on_term` in the task runner doesn't raise anything, so `execute()` silently resumes and the task can end up reported `SUCCESS` even though the work was interrupted. #73395 fixes that by raising `AirflowTaskTerminated`.
But looking at how `task_runner.py` handles that exception (the except-chain around `except AirflowTaskTerminated`), it goes straight to `state = TaskInstanceState.FAILED` — it never calls `_handle_current_task_failed`, which is the path that checks `TaskInstance.is_eligible_to_retry()`. So after #73395 merges, a mid-execution infra kill won't silently succeed anymore, but it *will* permanently fail with zero retries even if the task has `retries=5` set. #73395's own PR description flags this directly: *"Whether an infrastructure kill should instead be retry-eligible is a real question — it is what #73238 is about... changing that classification also changes the UI 'mark failed' path, so it is left alone here."*
So the gap this issue is really about, narrowed down: **`AirflowTaskTerminated` needs the same kind of separate, bounded retry budget that pod-launch failures already get — not the application's `task.retries`, and not zero.**
**A possible shape for that**, modeled on the existing pattern rather than inventing something new:
- A new config, e.g. `infrastructure_failure_max_retries`, parallel to `pod_launch_failure_max_retries`.
- The real complication: `pod_launch_failure_max_retries` is tracked in `KubernetesExecutor.pod_launch_attempts`, an in-memory counter on the scheduler process. `AirflowTaskTerminated` is raised inside the task runner process itself — it doesn't have access to that. So this can't just reuse the same storage; the counter would need to live somewhere both sides can see it (task instance / task instance history row seems like the natural place, but I haven't traced whether there's already a suitable field, or whether this needs a new column/attempt-tracking concept).
- Rather than making `AirflowTaskTerminated` unconditionally terminal, it would need to go through something like `_handle_current_task_failed`'s retry-eligibility check, but against the new infra-budget instead of (or in addition to) `task.retries`.
- I'd explicitly leave the UI "mark failed" path alone, same as #73395 did — that's a separate behavior change and probably deserves its own discussion if anyone wants to touch it.
Given this touches task-sdk (where `AirflowTaskTerminated` is raised) and core retry semantics (not just the K8s executor), I'd guess this needs at least a short AIP rather than going straight to a PR, but I'm not certain where the line is for a change this size — happy to be corrected.
Open questions I don't have good answers to yet:
- Where should the infra-retry counter actually live given the cross-process constraint above?
- Should this budget be K8s-executor-specific (like today's pod-launch one) or executor-agnostic, since `AirflowTaskTerminated` itself is raised in executor-agnostic code?
- Does this want to be one unified "infrastructure failure retry budget" covering both pre-execution and mid-execution cases, or should they stay as separate mechanisms since they're detected completely differently?
Happy to take a pass at drafting this out further (or an AIP if that's the right next step) if this direction seems reasonable — wanted to sanity-check the approach here first rather than show up with a PR that reopens a decision #73395 intentionally deferred.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kubernetes Infrastructure failures consume retry attempts. [airflow]
Posted by "namanjain24-sudo (via GitHub)" <gi...@apache.org>.
namanjain24-sudo commented on issue #73238:
URL: https://github.com/apache/airflow/issues/73238#issuecomment-5782869102
Thanks for the detailed breakdown, @kartik00052 — matches what I ran into while working on #73395. Once that lands, I'd like to pick up the infra-retry-budget piece you described (a separate, bounded retry for `AirflowTaskTerminated`, not the application's `task.retries`).
Since it touches task-sdk and core retry semantics across processes, would a short AIP be the right next step here, or is a design write-up in the PR description enough for something this size? Wanted to check before writing code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org