You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "datasc24 (via GitHub)" <gi...@apache.org> on 2026/04/20 07:00:45 UTC
[D] Kubernetes Executor : Quota exceeded, tasks fail [airflow]
GitHub user datasc24 created a discussion: Kubernetes Executor : Quota exceeded, tasks fail
Hello,
We were using AIrflow 2.7 with Kubernetes Executor and when we used to launch a lot of tasks, they were staying queued. But after migrating to Airflow 3.2, if we exceed our namespace quota, tasks are failing.
Do you have any solution to deal with this ? Is there a way to force task to stay queued, wiaiting for ressources instead of failing ?
GitHub link: https://github.com/apache/airflow/discussions/65520
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Kubernetes Executor : Quota exceeded, tasks fail [airflow]
Posted by "cookesan (via GitHub)" <gi...@apache.org>.
GitHub user cookesan added a comment to the discussion: Kubernetes Executor : Quota exceeded, tasks fail
Kubernetes quota rejection happens before the worker pod exists, so Airflow can only retry or throttle pod creation; it cannot make Kubernetes accept pods beyond the namespace quota.
In the current Kubernetes Executor source, `403` errors whose message contains `exceeded quota` are retried only while `task_publish_max_retries` allows it. If that value is `0`, or if the retry budget is exhausted while the quota is still full, the task is failed.
I would check these first:
- `AIRFLOW__KUBERNETES_EXECUTOR__TASK_PUBLISH_MAX_RETRIES`
- `worker_pods_creation_batch_size`
- global `parallelism`
- DAG/task concurrency
- pools for the high-fanout work
- worker pod CPU/memory requests versus the namespace quota
The durable fix is to keep Airflow's maximum pod demand below the namespace quota, or raise the quota. If you want the old "stay queued until quota frees" behavior, a finite or infinite publish retry setting is the closest Airflow-side lever, but it should be paired with concurrency limits so the scheduler is not constantly asking Kubernetes for pods it cannot create.
GitHub link: https://github.com/apache/airflow/discussions/65520#discussioncomment-16657272
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org