You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Vineet Goel (JIRA)" <ji...@apache.org> on 2016/07/25 23:44:20 UTC
[jira] [Commented] (AIRFLOW-363) Celery Queue becoming too huge
when worker unavailable
[ https://issues.apache.org/jira/browse/AIRFLOW-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392866#comment-15392866 ]
Vineet Goel commented on AIRFLOW-363:
-------------------------------------
[~bolke]
[~sanand]
[~criccomini]
> Celery Queue becoming too huge when worker unavailable
> ------------------------------------------------------
>
> Key: AIRFLOW-363
> URL: https://issues.apache.org/jira/browse/AIRFLOW-363
> Project: Apache Airflow
> Issue Type: Bug
> Reporter: Vineet Goel
>
> The scheduler currently queues executable tasks every run of the scheduler. I ran into an issue where I was using a single worker and my dag was composed of a bunch of tasks that could run simultaneously. I was setting a concurrency on the worker to control how many of those tasks could run at the same time. However, my scheduler kept queueing more and more tasks. This can cause a HUGE back pressure on the celery queue, which becomes unmanageable at times.
> We should set the http://docs.celeryproject.org/en/latest/configuration.html#celery-event-queue-ttl
> configuration to handle this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)