You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "alkismavridis (via GitHub)" <gi...@apache.org> on 2025/09/24 12:48:52 UTC
[I] Scheduler stops working - tasks stuck in queue state forever [airflow]
alkismavridis opened a new issue, #56045:
URL: https://github.com/apache/airflow/issues/56045
### Apache Airflow version
3.0.6
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
We run airflow 3.0.6 within a Docker container.
From time to time (once per 2-3 days) our scheduler stops to work. The health check command seems to be fine:
```
default@1a4707d0c867:/opt/airflow$ airflow jobs check --job-type SchedulerJob --local
Found one alive job.
```
But in fact, the scheduler seems to be stuck. Multiple Tasks seem to stuck in "Queued" state and nothing gets started. The whole infrastructure comes to a halt.
Scheduler logs repeatidly the following statement:
```
[2025-09-24T12:37:05.471+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:06.536+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:07.589+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:08.652+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:09.717+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:10.777+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:11.775+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:12.836+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:13.885+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:14.939+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:15.998+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:17.067+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:17.347+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:18.415+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:19.469+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:20.523+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:21.575+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:21.829+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:21.963+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:21.984+0000] {scheduler_job_runner.py:2218} INFO - Adopting or resetting orphaned tasks for active dag runs
[2025-09-24T12:37:23.014+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:24.076+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:25.127+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:26.171+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:27.217+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:28.260+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:29.308+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:30.055+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:31.104+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:31.875+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:32.394+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:33.438+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:34.495+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
[2025-09-24T12:37:35.547+0000] {dagrun.py:1039} INFO - Found span_status 'should_end', while updating state for dag_run 'scheduled__2025-09-23T23:45:00+00:00'
```
Please note that the scheduler seems to use all available memory (https://github.com/apache/airflow/issues/55768) so this might be related.
### What you think should happen instead?
Scheduler works and triggers tasks. If scheduler is not healthy, then `airflow jobs check --job-type SchedulerJob --local` should at least be informative enough and fail.
### How to reproduce
Unfortunately we are unsure. Section for the scheduler:
```yaml
airflow-scheduler:
<<: *airflow-common
container_name: airflow-scheduler-test
command: scheduler
healthcheck:
test: ["CMD-SHELL", "airflow jobs check --job-type SchedulerJob --local || bash -c 'kill -s 15 -1 && (sleep 10; kill -s 9 -1)'"]
interval: 30s
timeout: 10s
retries: 5
start_period: 150s
restart: always
depends_on:
<<: *airflow-common-depends-on
airflow-init:
condition: service_completed_successfully
mem_limit: 4000m
```
### Operating System
Linux (+docker)
### Versions of Apache Airflow Providers
```
apache-airflow-providers-amazon==9.12.0
apache-airflow-providers-celery==3.12.2
apache-airflow-providers-cncf-kubernetes==10.7.0
apache-airflow-providers-common-compat==1.7.3
apache-airflow-providers-common-io==1.6.2
apache-airflow-providers-common-messaging==1.0.5
apache-airflow-providers-common-sql==1.27.5
apache-airflow-providers-docker==4.4.2
apache-airflow-providers-elasticsearch==6.3.2
apache-airflow-providers-fab==2.4.1
apache-airflow-providers-ftp==3.13.2
apache-airflow-providers-git==0.0.6
apache-airflow-providers-google==17.1.0
apache-airflow-providers-grpc==3.8.2
apache-airflow-providers-hashicorp==4.3.2
apache-airflow-providers-http==5.3.3
apache-airflow-providers-microsoft-azure==12.6.1
apache-airflow-providers-mysql==6.3.3
apache-airflow-providers-odbc==4.10.2
apache-airflow-providers-openlineage==2.6.1
apache-airflow-providers-postgres==6.2.3
apache-airflow-providers-redis==4.2.0
apache-airflow-providers-sendgrid==4.1.3
apache-airflow-providers-sftp==5.3.4
apache-airflow-providers-slack==9.1.4
apache-airflow-providers-smtp==2.2.0
apache-airflow-providers-snowflake==6.4.0
apache-airflow-providers-ssh==4.1.3
apache-airflow-providers-standard==1.6.0
```
### Deployment
Docker-Compose
### Deployment details
Docker-Compose.
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler stops working - tasks stuck in queue state forever [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #56045:
URL: https://github.com/apache/airflow/issues/56045#issuecomment-4393224451
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler stops working - tasks stuck in queue state forever [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #56045: Scheduler stops working - tasks stuck in queue state forever
URL: https://github.com/apache/airflow/issues/56045
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler stops working - tasks stuck in queue state forever [airflow]
Posted by "LengYue12389 (via GitHub)" <gi...@apache.org>.
LengYue12389 commented on issue #56045:
URL: https://github.com/apache/airflow/issues/56045#issuecomment-3342223087
<img width="1275" height="151" alt="Image" src="https://github.com/user-attachments/assets/5a93f9fa-01c6-446a-b46f-3e152b0e8be4" />
dag-processor will occupy more and more memory over time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler stops working - tasks stuck in queue state forever [airflow]
Posted by "LengYue12389 (via GitHub)" <gi...@apache.org>.
LengYue12389 commented on issue #56045:
URL: https://github.com/apache/airflow/issues/56045#issuecomment-3342224112
This issue still exists in version 3.1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler stops working - tasks stuck in queue state forever [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on issue #56045:
URL: https://github.com/apache/airflow/issues/56045#issuecomment-3443412080
Without memray or without investigations like https://github.com/apache/airflow/issues/56641 -- it is tough (almost impossible) to know what's happening !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler stops working - tasks stuck in queue state forever [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #56045:
URL: https://github.com/apache/airflow/issues/56045#issuecomment-4446342092
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org