You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "paramjeet01 (via GitHub)" <gi...@apache.org> on 2024/04/12 15:18:40 UTC
[I] Tasks are in queued state for a longer time [airflow]
paramjeet01 opened a new issue, #38968:
URL: https://github.com/apache/airflow/issues/38968
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.8.3
### What happened?
The tasks are in queued state for longer time than expected. This was working fine in 2.6.3 perfectly.
### What you think should happen instead?
The tasks should be in running state instead being queued.
### How to reproduce
Spin up multiple dags and tasks in parallel in airflow 2.8.3
### Operating System
Amazon Linux 2
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time [airflow]
Posted by "paramjeet01 (via GitHub)" <gi...@apache.org>.
paramjeet01 commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2053914101
This is similar issue to #36998 , #36478
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time [airflow]
Posted by "paramjeet01 (via GitHub)" <gi...@apache.org>.
paramjeet01 commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2053543932
@ephraimbuddy , The above config has improved the performance in scheduling the tasks and the gantt view shows the tasks queue time is lesser than before. Also , please share performance tuning documentation that will be really nice of you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time and executor slots are exhausted often [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2506874533
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time and executor slots are exhausted often [airflow]
Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2105535297
This issue is related to watcher is not able to scale and process the events on time. This leads to so many completed pods over the time.
related: https://github.com/apache/airflow/issues/22612
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2052303193
Without any logs, errors, metrics or details it is impossible to (1) understand your problem and (2) fix anything.
Can you please describe more details?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time and executor slots are exhausted often [airflow]
Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2105534758
@paramjeet01
You can mention airflow num_runs configuration parameter to restart the scheduler container based on your needs.
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time and executor slots are exhausted often [airflow]
Posted by "paramjeet01 (via GitHub)" <gi...@apache.org>.
paramjeet01 commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2063454855
We have also observed that the pods are not cleaned up after completion of the task and all the pods are stuck in SUCCEEDED state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2052382840
> Apologies, I'm relatively new to Airflow. We've checked the scheduler logs thoroughly, and everything seems to be functioning correctly without any errors. Additionally, the scheduler pods are operating within normal CPU and memory limits. Our database, RDS, doesn't indicate any breaches either. Currently, we're running a DAG with 150 parallel DAG runs. However, a significant portion of tasks are remaining in a queued state for an extended period. Specifically, about 140 tasks are queued, while only 39 are actively running. I've already reviewed the configurations for max_active_tasks_per_dag and max_active_runs_per_dag, and they appear to be properly set. We did not face this issue in 2.3.3
Can you try increasing the `[scheduler]max_tis_per_query` to 512? In one performance debugging, we discovered this to work better when increased but it might depend on the environment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time and executor slots are exhausted often [airflow]
Posted by "changqian9 (via GitHub)" <gi...@apache.org>.
changqian9 commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2053928766
We got the same issues for twice. Same observation, this happened when executor open slots < 0.
<img width="1737" alt="Screenshot 2024-04-14 at 14 21 36" src="https://github.com/apache/airflow/assets/10512672/cec289f7-d68d-49a2-80ef-24fb367de635">
cc @paramjeet01
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time [airflow]
Posted by "paramjeet01 (via GitHub)" <gi...@apache.org>.
paramjeet01 commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2052400361
I have updated the config map with `max_tis_per_query = 512` and redeployed the scheduler. Will monitor for some time and let you know , thanks for quick response.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time and executor slots are exhausted often [airflow]
Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2105532327
This issue is related to watcher is not able to scale and process the events on time. This leads to so many completed pods over the time.
related: #22612
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time and executor slots are exhausted often [airflow]
Posted by "paramjeet01 (via GitHub)" <gi...@apache.org>.
paramjeet01 commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2075684549
Sorry , the above comment is false positive. We are customizing our KPO and we missed to add `on_finish_action` so the pods stuck in SUCCEEDED state. After adding it , all the pods are removed properly.
We also able to mitigate the executor slots leak by adding a cronjob to restart our schedulers once in a while.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time and executor slots are exhausted often [airflow]
Posted by "paramjeet01 (via GitHub)" <gi...@apache.org>.
paramjeet01 commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2059521327
After reviewing various GitHub and Stack Overflow discussions, I've made updates to the following configuration and migrated to Airflow version 2.7.2 with apache-airflow-providers-cncf-kubernetes version 8.0.0:
```
[scheduler]
task_queued_timeout : 90
max_dagruns_per_loop_to_schedule : 128
max_dagruns_to_create_per_loop : 128
max_tis_per_query : 1024
```
Additionally, scaling the scheduler to 8 replicas has notably improved performance. The executor slots being exhausted was solved using max_tis_query to max number. Sorry I couldn't find the root cause of the issue but I hope this helps
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time and executor slots are exhausted often [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2457630176
Airflow 2.10.3 is now out an it has fix #42932 that is likely to fix the problems you reported, please <author> upgrade, check if it fixed your problem and report back @paramjeet01 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time and executor slots are exhausted often [airflow]
Posted by "paramjeet01 (via GitHub)" <gi...@apache.org>.
paramjeet01 commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2105580937
@dirrao , airflow num_runs configuration parameter purpose has been changed a while ago AFAIK and cannot be used for restarting the scheduler. We have also removed run_duration which was later used for restarting the scheduler.
https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html#num-runs
https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html#remove-run-duration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time and executor slots are exhausted often [airflow]
Posted by "paramjeet01 (via GitHub)" <gi...@apache.org>.
paramjeet01 commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2056754864
@jscheffl , Can you remove the pending response label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time and executor slots are exhausted often [airflow]
Posted by "paramjeet01 (via GitHub)" <gi...@apache.org>.
paramjeet01 commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2061611229
After observing for some time, we encountered instances where the executor open slots were approaching negative values, leading to tasks becoming stuck in the scheduled state. Restarting all the scheduler pods solved this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time [airflow]
Posted by "paramjeet01 (via GitHub)" <gi...@apache.org>.
paramjeet01 commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2053592268
@ephraimbuddy , Also saw that the dags were in scheduled state , after restarting the scheduler everything works fine now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time and executor slots are exhausted often [airflow]
Posted by "sunank200 (via GitHub)" <gi...@apache.org>.
sunank200 commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2141353538
If I understood this correctly, the performance issues with tasks in the queued state were mitigated by adjusting max_tis_per_query, scaling scheduler replicas, and implementing periodic scheduler restarts. @paramjeet01 tried periodic restarts of all scheduler pods to temporarily resolve the issue.
Related Issues: #36998, #22612
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time [airflow]
Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2053591369
@paramjeet01 This might be helpful
https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/scheduler.html#fine-tuning-your-scheduler-performance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time [airflow]
Posted by "paramjeet01 (via GitHub)" <gi...@apache.org>.
paramjeet01 commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2052336139
Apologies, I'm relatively new to Airflow. We've checked the scheduler logs thoroughly, and everything seems to be functioning correctly without any errors. Additionally, the scheduler pods are operating within normal CPU and memory limits. Our database, RDS, doesn't indicate any breaches either. Currently, we're running a DAG with 150 parallel DAG runs. However, a significant portion of tasks are remaining in a queued state for an extended period. Specifically, about 140 tasks are queued, while only 39 are actively running. I've already reviewed the configurations for max_active_tasks_per_dag and max_active_runs_per_dag, and they appear to be properly set. We did not face this issue in 2.3.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time and executor slots are exhausted often [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2165175179
Can anyone try this patch https://github.com/apache/airflow/pull/40183 for the scheduler restarting issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time and executor slots are exhausted often [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #38968:
URL: https://github.com/apache/airflow/issues/38968#issuecomment-2521796229
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Tasks are in queued state for a longer time and executor slots are exhausted often [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #38968: Tasks are in queued state for a longer time and executor slots are exhausted often
URL: https://github.com/apache/airflow/issues/38968
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org