[jira] [Updated] (AIRFLOW-1965) Like Scheduler dies and subdag locked in running state when sub-subdag fails

["Dylan Gorman (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dylan Gorman updated AIRFLOW-1965:
----------------------------------
    Summary: Like Scheduler dies and subdag locked in running state when sub-subdag fails  (was: Subdag locked in running state when sub-subdag fails)

> Like Scheduler dies and subdag locked in running state when sub-subdag fails
> ----------------------------------------------------------------------------
>
>                 Key: AIRFLOW-1965
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1965
>             Project: Apache Airflow
>          Issue Type: Bug
>    Affects Versions: 1.8.1
>            Reporter: Dylan Gorman
>         Attachments: subdagtest.py
>
>
> We have a problem with subdags getting locked in the paused state. This seems to happen when the subdags contain another level of subdags which fail.
> The dag id corresponding to the subdag gets switched to paused in the database after failure of the sub-subdag. The scheduler as well seems to die when this occurs (trying to clear the task leaves it in the shutdown state). The SubDagOperator task stays in the running state and never switches to failed. This seems to only happen when we set concurrency=1 in the first level subdag.
> We attach a python file which should re-produce the behavior.
> In particular, our graph looks like this:
> Dummy Task_1 >> Subdag >> Dummy Task_2
> The subdag graph looks like:
> Dummy Task_Root >> SubSubdag_1 >> Dummy_Task_final
> Dummy_Task_Root >> SubSubdag_2 >> Dummy_Task_final
> SubSubdag_1 and _2 only contain a failing pythonoperator. When either of the SubSubdags fails, the top level subdag gets then stuck in the paused state and does not allow execution of the other subsubdag.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)