You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "romina-nikolova (via GitHub)" <gi...@apache.org> on 2026/09/09 12:01:25 UTC
[I] KubernetesExecutor: delete_pod fires before the self.running dedup check, causing repeated deletes per TaskInstanceKey [airflow]
romina-nikolova opened a new issue, #72803:
URL: https://github.com/apache/airflow/issues/72803
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.3.1
### What happened and how to reproduce it?
KubernetesExecutor issues 2-3 delete_pod API calls for the same finished worker pod instead of one. The first call succeeds; the rest return 404 from the Kubernetes API. All calls come from the same scheduler process for the same TaskInstanceKey, milliseconds apart.
Observed on a production-scale cluster: over a 15-minute sample, 306 pods received 3 delete calls, 15 received 2, and only 4 received the intended 1. That's ~1.47 delete calls per pod and a 67% 404 rate, generating a large amount of unnecessary kube-apiserver audit-log volume (and associated CloudWatch cost on EKS) at scale. Mapped tasks (non-null map_index) appear to be the common factor in every case we captured.
Root cause
In the result-processing path (kubernetes_executor.py / kubernetes_executor_utils.py in apache-airflow-providers-cncf-kubernetes), delete_pod is invoked before the executor's self.running dedup check runs for that TaskInstanceKey. Because the same task-completion event gets processed more than once before the dedup check has a chance to short-circuit it, the pod delete is issued redundantly on each pass.
### What you think should happen instead?
The dedup check against self.running should happen (or the running-set be updated) before delete_pod is called, so a given TaskInstanceKey only ever triggers one delete call, regardless of how many times its completion event is processed.
### Operating System
_No response_
### Deployment
None
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes==10.21.0
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesExecutor: delete_pod fires before the self.running dedup check, causing repeated deletes per TaskInstanceKey [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #72803:
URL: https://github.com/apache/airflow/issues/72803#issuecomment-5601492074
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org