[jira] [Created] (AIRFLOW-363) Celery Queue becoming too huge when worker unavailable

["Vineet Goel (JIRA)" <ji...@apache.org>]

Vineet Goel created AIRFLOW-363:
-----------------------------------

             Summary: Celery Queue becoming too huge when worker unavailable
                 Key: AIRFLOW-363
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-363
             Project: Apache Airflow
          Issue Type: Bug
            Reporter: Vineet Goel


The scheduler currently queues executable tasks every run of the scheduler. I ran into an issue where I was using a single worker and my dag was composed of a bunch of tasks that could run simultaneously. I was setting a concurrency on the worker to control how many of those tasks could run at the same time. However, my scheduler kept queueing more and more tasks. This can cause a HUGE back pressure on the celery queue, which becomes unmanageable at times. 

We should set the http://docs.celeryproject.org/en/latest/configuration.html#celery-event-queue-ttl
configuration to handle this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)