You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "praveer-rc (via GitHub)" <gi...@apache.org> on 2025/10/27 11:57:14 UTC
[I] Schedular getting stuck in parallelism limit reached [airflow]
praveer-rc opened a new issue, #57347:
URL: https://github.com/apache/airflow/issues/57347
### Apache Airflow version
Other Airflow 2/3 version (please specify below)
### If "Other Airflow 2/3 version" selected, which one?
3.0.2
### What happened?
So a little context about my architecture. I have installed airflow using the chart version 1.18.0 on my k8s cluster. The airflow core pods (api, triggerrer, schedular, etc.) are deployed in the airfow namespace. I'm using the kubernetes executor so the worker pods are deployed in the tenant namespace which has the secrets for the specific dag. Now I'm seeing two issues that could be interconnected.
1. The worker pods that are deployed in the tenant namespace. After the task run is complete, the pods remain in 'Completed' state and are never automatically deleted.
2. (The major one) The scheduler is getting stuck with the message "Parallelism limit reached" even though I've set it in thousands and it gets stuck every few hours still with the same error message. This could be because the pod isn't sending a completed signal back to the schedular.
### What you think should happen instead?
_No response_
### How to reproduce
Set - name: AIRFLOW__CORE__PARALLELISM
value: '128'
- name: AIRFLOW__CORE__DEFAULT_POOL_TASK_SLOT_COUNT
value: '256'
- name: AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG
value: '32'
Send down tasks in another namespace using the kubernetes executor and check after 32 tasks for the same pipeline, it starts logging "Parallelization limit reached" and doesn't schedule any pods.
### Operating System
Kubernetes
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Chart version 1.18.0
### Anything else?
The problem occurs every few hours or minutes
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Schedular getting stuck in parallelism limit reached [airflow]
Posted by "zach-overflow (via GitHub)" <gi...@apache.org>.
zach-overflow commented on issue #57347:
URL: https://github.com/apache/airflow/issues/57347#issuecomment-3456856138
I'll chime in that we are observing a similar behavior, and we have a similar setup (mutliple dedicated "tenant" namespaces, and a singular "core" namespace where the scheduler and shared components run), but with the Celery executor, and the majority of tasks running within the celery workers, rather than as ephemeral k8s task pods. We are running on Airflow version 3.0.6 and are observing the same error, despite increasing the `parallelism` value. It almost seems like the way the base executor tracks "running" tasks is not accounting for task removals properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Schedular getting stuck in parallelism limit reached [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #57347:
URL: https://github.com/apache/airflow/issues/57347#issuecomment-3456263817
@praveer-rc , it seems you need this config: https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/configurations-ref.html?utm_source=chatgpt.com#multi-namespace-mode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Schedular getting stuck in parallelism limit reached [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #57347:
URL: https://github.com/apache/airflow/issues/57347#issuecomment-3544432284
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Schedular getting stuck in parallelism limit reached [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #57347:
URL: https://github.com/apache/airflow/issues/57347#issuecomment-3578168669
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Schedular getting stuck in parallelism limit reached [airflow]
Posted by "praveer-rc (via GitHub)" <gi...@apache.org>.
praveer-rc commented on issue #57347:
URL: https://github.com/apache/airflow/issues/57347#issuecomment-3456079955
When I run the tasks in a target namespace, it throws this:
<img width="1498" height="932" alt="Image" src="https://github.com/user-attachments/assets/9694821d-8dcf-46ba-b88a-c8fea365fa32" />
However when I run in the same namespace, it works perfectly fine. The pod creates and after the task pod execution completes, it gets deleted and the schedular isn't getting stuck.
Also, I keep seeing these messages every 30 seconds:
[2025-10-28T09:41:30.924+0000] {kubernetes_executor_utils.py:95} INFO - Kubernetes watch timed out waiting for events. Restarting watch.
[2025-10-28T09:41:31.924+0000] {kubernetes_executor_utils.py:132} INFO - Event: and now my watch begins starting at resource_version: 0
[2025-10-28T09:42:01.959+0000] {kubernetes_executor_utils.py:95} INFO - Kubernetes watch timed out waiting for events. Restarting watch.
[2025-10-28T09:42:02.960+0000] {kubernetes_executor_utils.py:132} INFO - Event: and now my watch begins starting at resource_version: 0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Schedular getting stuck in parallelism limit reached [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #57347:
URL: https://github.com/apache/airflow/issues/57347#issuecomment-3456936827
@zach-overflow I would like to know if you have the multi namespace config enabled: https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/configurations-ref.html?utm_source=chatgpt.com#multi-namespace-mode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Schedular getting stuck in parallelism limit reached [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #57347:
URL: https://github.com/apache/airflow/issues/57347#issuecomment-3455254178
Can you check the difference when you have it in the same namespace? The capacity of your cluster would determine the number of pods that can be run on the cluster. So you can increase parallelism but still be limited by your cluster capacity on the number of pods it can run.
I also recommend increasing `AIRFLOW__KUBERNETES_EXECUTOR__WORKER_PODS_CREATION_BATCH_SIZE` to the size of your cluster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Schedular getting stuck in parallelism limit reached [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #57347:
URL: https://github.com/apache/airflow/issues/57347#issuecomment-3450908965
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Schedular getting stuck in parallelism limit reached [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #57347:
URL: https://github.com/apache/airflow/issues/57347#issuecomment-3460882752
> I might be misunderstanding, but I don't think that configuration is relevant for our situation since we use the Celery executor, rather than the k8s executor. But to answer your question, we do not use that setting.
Your case is slightly different from the issue and both of you might truly have reached your executor parallelism limit. The open slots in the executor is `parallelism - running tasks -queued tasks`. Can both of you check that when you add the queued tasks and running tasks, that they are not equal to the set parallelism?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Schedular getting stuck in parallelism limit reached [airflow]
Posted by "zach-overflow (via GitHub)" <gi...@apache.org>.
zach-overflow commented on issue #57347:
URL: https://github.com/apache/airflow/issues/57347#issuecomment-3457095723
I might be misunderstanding, but I don't think that configuration is relevant for our situation since we use the Celery executor, rather than the k8s executor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org