You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "Bolke de Bruin (JIRA)" <ji...@apache.org> on 2016/05/03 19:29:12 UTC

[jira] [Created] (AIRFLOW-41) SubdagOperators can oversubscribe to pools due to race condition

Bolke de Bruin created AIRFLOW-41:
-------------------------------------

             Summary: SubdagOperators can oversubscribe to pools due to race condition
                 Key: AIRFLOW-41
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-41
             Project: Apache Airflow
          Issue Type: Bug
          Components: subdag, scheduler
    Affects Versions: Airflow 1.7.1
            Reporter: Bolke de Bruin


SubDagOperators essentially create their own mini-scheduler. Which can interfere with the main scheduler. 

SubdagOperators check if there is slot available in a Pool. However this slot is not claimed at the same time leaving room for main scheduler to also check for the slot. Both can then obtain a slot and thus oversubscribe

A solution could be a centralized PoolHandler that gives out slots



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)