You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Aizhamal Nurmamat kyzy (JIRA)" <ji...@apache.org> on 2019/05/17 22:19:04 UTC
[jira] [Updated] (AIRFLOW-41) SubdagOperators can oversubscribe to
pools due to race condition
[ https://issues.apache.org/jira/browse/AIRFLOW-41?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aizhamal Nurmamat kyzy updated AIRFLOW-41:
------------------------------------------
Labels: subdag (was: )
Component/s: operators
Adding 'operators' component and 'subdag' label for component refactor.
> SubdagOperators can oversubscribe to pools due to race condition
> ----------------------------------------------------------------
>
> Key: AIRFLOW-41
> URL: https://issues.apache.org/jira/browse/AIRFLOW-41
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators, scheduler, subdag
> Affects Versions: 1.7.1
> Reporter: Bolke de Bruin
> Priority: Major
> Labels: subdag
>
> SubDagOperators essentially create their own mini-scheduler. Which can interfere with the main scheduler.
> SubdagOperators check if there is slot available in a Pool. However this slot is not claimed at the same time leaving room for main scheduler to also check for the slot. Both can then obtain a slot and thus oversubscribe
> A solution could be a centralized PoolHandler that gives out slots
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)