DAG with ExternalTaskSensor won't run after deletion

[Ali Uz <al...@gmail.com>]

Hello,

I have a DAG that has an ExternalTaskSensor that waits for another DAG to
complete before running. Here's my scenario: I run this DAG and while the
external task sensor is running, I pause it, then delete all its tasks and
the DAG Run from the Admin panel. Then I turn it on again, and the DAG
displays as "running" in the UI, but none of its tasks get queued.

Also in the Jobs UI there seems to be a Job stuck in "running" which seems
to be related to the DAG I just deleted.

How can I ensure that my DAG will always run no matter what?

Thanks,

- Ali

Re: DAG with ExternalTaskSensor won't run after deletion

[Ali Uz <al...@gmail.com>]

Hi Joy,

Indeed that is what's happening. I am running airflow (at the moment) on a
single docker instance in AWS ECS and the only way I can bounce the
scheduler is either by initiating a new deploy or by SSHing into the
machine and restarting the scheduler manually, which isn't really a
sustainable method for the long-term.
I guess I can file this one as a bug in JIRA if some other guys also
confirm that, this is indeed a bug and not an intended feature.

Thanks again!

- Ali

On Tue, Sep 12, 2017 at 10:10 PM, Joy Gao <jo...@wepay.com> wrote:

> Hi Ali,
>
> I was able to replicate your scenario above.
>
> The executor maintains a cache of running task instances. When a task
> instance finishes, it is removed from the `running` cache of the executor.
> When you hard-delete the DAG Run and the task instances via the UI, the
> corresponding task instances are still in the 'running' cache. So when the
> scheduler is iterating through the dags, it is told by the executor that
> this particular task instance is already running, and does not need to be
> rescheduled.  If you bounce the scheduler, the task should be picked up
> again as the cache will be regenerated.
>
> I believe this is a bug in the scheduler. So feel free to report it as a
> JIRA issue on this. Although in general,  I try to avoid hard-deleting
> items from the DB when they are in `running` state. Hope this helps!
>
> Joy
>
> On Mon, Sep 11, 2017 at 2:38 PM, Ali Uz <al...@gmail.com> wrote:
>
> > Hello,
> >
> > I have a DAG that has an ExternalTaskSensor that waits for another DAG to
> > complete before running. Here's my scenario: I run this DAG and while the
> > external task sensor is running, I pause it, then delete all its tasks
> and
> > the DAG Run from the Admin panel. Then I turn it on again, and the DAG
> > displays as "running" in the UI, but none of its tasks get queued.
> >
> > Also in the Jobs UI there seems to be a Job stuck in "running" which
> seems
> > to be related to the DAG I just deleted.
> >
> > How can I ensure that my DAG will always run no matter what?
> >
> > Thanks,
> >
> > - Ali
> >
>

Re: DAG with ExternalTaskSensor won't run after deletion

[Joy Gao <jo...@wepay.com>]

Hi Ali,

I was able to replicate your scenario above.

The executor maintains a cache of running task instances. When a task
instance finishes, it is removed from the `running` cache of the executor.
When you hard-delete the DAG Run and the task instances via the UI, the
corresponding task instances are still in the 'running' cache. So when the
scheduler is iterating through the dags, it is told by the executor that
this particular task instance is already running, and does not need to be
rescheduled.  If you bounce the scheduler, the task should be picked up
again as the cache will be regenerated.

I believe this is a bug in the scheduler. So feel free to report it as a
JIRA issue on this. Although in general,  I try to avoid hard-deleting
items from the DB when they are in `running` state. Hope this helps!

Joy

On Mon, Sep 11, 2017 at 2:38 PM, Ali Uz <al...@gmail.com> wrote:

> Hello,
>
> I have a DAG that has an ExternalTaskSensor that waits for another DAG to
> complete before running. Here's my scenario: I run this DAG and while the
> external task sensor is running, I pause it, then delete all its tasks and
> the DAG Run from the Admin panel. Then I turn it on again, and the DAG
> displays as "running" in the UI, but none of its tasks get queued.
>
> Also in the Jobs UI there seems to be a Job stuck in "running" which seems
> to be related to the DAG I just deleted.
>
> How can I ensure that my DAG will always run no matter what?
>
> Thanks,
>
> - Ali
>