You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Ash Berlin-Taylor (JIRA)" <ji...@apache.org> on 2019/06/27 12:56:00 UTC
[jira] [Updated] (AIRFLOW-4553) Scheduler only schedules one task
at a time with CeleryExecutor
[ https://issues.apache.org/jira/browse/AIRFLOW-4553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ash Berlin-Taylor updated AIRFLOW-4553:
---------------------------------------
Attachment: Screenshot 2019-06-27 at 13.54.00.png
> Scheduler only schedules one task at a time with CeleryExecutor
> ---------------------------------------------------------------
>
> Key: AIRFLOW-4553
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4553
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 1.10.2, 1.10.3
> Reporter: Alejandro
> Assignee: Alejandro
> Priority: Major
> Attachments: Screenshot 2019-06-27 at 13.54.00.png
>
>
> I've recently upgraded from Airflow 1.9.0 to Airflow 1.10.3. There has been a change in how the scheduler put tasks up for execution that have affected some of our DAGs. We use the CeleryExecutor.
> On Airflow 1.9.0 the scheduler would set as `scheduled` as many tasks as it could, For example in a DAG with 5 independent tasks and enough concurrency set, the scheduler would queue the 5 tasks for execution and all 5 tasks could potentially start running.
> On Airflow 1.10.3 the scheduler would set as `scheduled` only one task at a time, For example in a DAG with 5 independent tasks and enough concurrency set, the scheduler would queue the 1 task for execution and the other 4 would not stay with a None state.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)