[jira] [Commented] (AIRFLOW-1133) More tasks than the concurrency limit can run

["Gerard Toonstra (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15986765#comment-15986765 ] 

Gerard Toonstra commented on AIRFLOW-1133:
------------------------------------------

Could be related to this issue too.

> More tasks than the concurrency limit can run
> ---------------------------------------------
>
>                 Key: AIRFLOW-1133
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1133
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Alex Guziel
>
> There are two ways to put checks on dag concurrency:
> 1) The scheduler not queueing too many tasks (via checking the amount of tasks running)
> 2) The worker checking that not too many tasks are running (via the db)
> Right now, both have issues.
> 1 doesn't considered queued tasks which may not be running now, but will be running soon. Hopefully, check 2 should catch it, but it does not check the condition properly as it only locks the row, and it seems locking the dag would also be expensive.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)