[jira] [Commented] (AIRFLOW-6233) Task get stuck on scheduled on LocalExecutor

["Robert Conroy (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-6233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17015492#comment-17015492 ] 

Robert Conroy commented on AIRFLOW-6233:
----------------------------------------

[~BrechtDeVlieger] I've been unable to reproduce this locally. Do you see this on the first DAG run, or should I be waiting for a future run?

> Task get stuck on scheduled on LocalExecutor
> --------------------------------------------
>
>                 Key: AIRFLOW-6233
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6233
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler
>    Affects Versions: 1.10.6
>            Reporter: Brecht De Vlieger
>            Priority: Blocker
>         Attachments: DAG.png, airflow.cfg, expected_gantt.png, logs.tar.gz, observed_gantt.png, test.py
>
>
> Use case description:
> One DAG with a sequence of tasks run by the LocalExecutor. The number of parallel tasks (6) to be run is higher than the concurrency limit (3) (parallelism and dag_concurrency). An image of the DAG layout is attached below.
> Expected behaviour:
> Scheduler uses the three available slots to run as many tasks concurrently as possible. The DAG finishes in some finite time without restarting the scheduler. Example Gantt chart attached below.
> Observed behaviour:
> The tasks are not optimally scheduled and after a while the tasks no longer move from the 'scheduled' state to 'queued'.  When the scheduler is restarted the tasks are picked up again as expected. The Gantt chart is also in the attachments. Note that the chart does not show the tasks that never get scheduled. You can find the logs in the attachments.
> How to reproduce:
> Install vanilla airflow version 1.10.6 and run with LocalExecutor (I used a mysql database). The airflow.cfg and dag file are added to the attachments. The problem does not exist in 1.10.5.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)