You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by "Ryabchuk, Pavlo" <ex...@here.com> on 2016/06/29 13:02:24 UTC
Programmatically stop execution of a task
Hi,
Can someone suggest how can I stop execution/mark(failed or any other status) of all sibling tasks (also can be passed as a list) if one of them failed?
It's just useless for me to process other tasks if one failed, because in my case it's for sure issue with my processing soft and it will lead to fixing and recompilation of it, and in any case reprocessing all tasks with newer version.
I've tried various manipulations with TaskInstances and Jobs states on on_failure_callback but nothing seem to work.
Best,
Paul
Re: Programmatically stop execution of a task
Posted by Maxime Beauchemin <ma...@gmail.com>.
Good point, `shutdown` probably isn't needed anymore. Though I haven't
looked at the code so I'm not 100% sure.
Max
On Thu, Jun 30, 2016 at 2:35 PM, Chris Riccomini <cr...@apache.org>
wrote:
> @Maxime, is SHUTDOWN needed anymore, then?
>
> On Thu, Jun 30, 2016 at 2:34 PM, Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
> > There's a new feature in master where tasks instances call the mothership
> > (the database) at each heartbeat to see if they are still running. If
> they
> > aren't they essentially commit suicide.
> >
> > That means that in theory you could have an failure hook that would set
> the
> > running siblings as failed and they should terminate.
> >
> > Max
> >
> > On Thu, Jun 30, 2016 at 12:50 PM, Chris Riccomini <criccomini@apache.org
> >
> > wrote:
> >
> > > Hey Paul,
> > >
> > > I don't have a solution to this, but I have a suggestion on where you
> > might
> > > look. There's a state that results in a blue square in the tree view.
> > This
> > > happens when a running task has its state cleared. It's apparently a
> > > 'poison pill' that should cause the task to kill itself. I don't recall
> > the
> > > name of the state off the top of my head. I *think* it's the SHUTDOWN
> > > state. Might trace through that code and see if you can figure it out.
> > >
> > > Cheers,
> > > Chris
> > >
> > > On Wed, Jun 29, 2016 at 6:02 AM, Ryabchuk, Pavlo <
> > > ext-pavlo.ryabchuk@here.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > Can someone suggest how can I stop execution/mark(failed or any other
> > > > status) of all sibling tasks (also can be passed as a list) if one of
> > > them
> > > > failed?
> > > > It's just useless for me to process other tasks if one failed,
> because
> > in
> > > > my case it's for sure issue with my processing soft and it will lead
> > to
> > > > fixing and recompilation of it, and in any case reprocessing all
> tasks
> > > with
> > > > newer version.
> > > > I've tried various manipulations with TaskInstances and Jobs states
> on
> > > > on_failure_callback but nothing seem to work.
> > > >
> > > > Best,
> > > > Paul
> > > >
> > >
> >
>
Re: Programmatically stop execution of a task
Posted by Chris Riccomini <cr...@apache.org>.
@Maxime, is SHUTDOWN needed anymore, then?
On Thu, Jun 30, 2016 at 2:34 PM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:
> There's a new feature in master where tasks instances call the mothership
> (the database) at each heartbeat to see if they are still running. If they
> aren't they essentially commit suicide.
>
> That means that in theory you could have an failure hook that would set the
> running siblings as failed and they should terminate.
>
> Max
>
> On Thu, Jun 30, 2016 at 12:50 PM, Chris Riccomini <cr...@apache.org>
> wrote:
>
> > Hey Paul,
> >
> > I don't have a solution to this, but I have a suggestion on where you
> might
> > look. There's a state that results in a blue square in the tree view.
> This
> > happens when a running task has its state cleared. It's apparently a
> > 'poison pill' that should cause the task to kill itself. I don't recall
> the
> > name of the state off the top of my head. I *think* it's the SHUTDOWN
> > state. Might trace through that code and see if you can figure it out.
> >
> > Cheers,
> > Chris
> >
> > On Wed, Jun 29, 2016 at 6:02 AM, Ryabchuk, Pavlo <
> > ext-pavlo.ryabchuk@here.com> wrote:
> >
> > > Hi,
> > >
> > > Can someone suggest how can I stop execution/mark(failed or any other
> > > status) of all sibling tasks (also can be passed as a list) if one of
> > them
> > > failed?
> > > It's just useless for me to process other tasks if one failed, because
> in
> > > my case it's for sure issue with my processing soft and it will lead
> to
> > > fixing and recompilation of it, and in any case reprocessing all tasks
> > with
> > > newer version.
> > > I've tried various manipulations with TaskInstances and Jobs states on
> > > on_failure_callback but nothing seem to work.
> > >
> > > Best,
> > > Paul
> > >
> >
>
Re: Programmatically stop execution of a task
Posted by Maxime Beauchemin <ma...@gmail.com>.
There's a new feature in master where tasks instances call the mothership
(the database) at each heartbeat to see if they are still running. If they
aren't they essentially commit suicide.
That means that in theory you could have an failure hook that would set the
running siblings as failed and they should terminate.
Max
On Thu, Jun 30, 2016 at 12:50 PM, Chris Riccomini <cr...@apache.org>
wrote:
> Hey Paul,
>
> I don't have a solution to this, but I have a suggestion on where you might
> look. There's a state that results in a blue square in the tree view. This
> happens when a running task has its state cleared. It's apparently a
> 'poison pill' that should cause the task to kill itself. I don't recall the
> name of the state off the top of my head. I *think* it's the SHUTDOWN
> state. Might trace through that code and see if you can figure it out.
>
> Cheers,
> Chris
>
> On Wed, Jun 29, 2016 at 6:02 AM, Ryabchuk, Pavlo <
> ext-pavlo.ryabchuk@here.com> wrote:
>
> > Hi,
> >
> > Can someone suggest how can I stop execution/mark(failed or any other
> > status) of all sibling tasks (also can be passed as a list) if one of
> them
> > failed?
> > It's just useless for me to process other tasks if one failed, because in
> > my case it's for sure issue with my processing soft and it will lead to
> > fixing and recompilation of it, and in any case reprocessing all tasks
> with
> > newer version.
> > I've tried various manipulations with TaskInstances and Jobs states on
> > on_failure_callback but nothing seem to work.
> >
> > Best,
> > Paul
> >
>
Re: Programmatically stop execution of a task
Posted by Chris Riccomini <cr...@apache.org>.
Hey Paul,
I don't have a solution to this, but I have a suggestion on where you might
look. There's a state that results in a blue square in the tree view. This
happens when a running task has its state cleared. It's apparently a
'poison pill' that should cause the task to kill itself. I don't recall the
name of the state off the top of my head. I *think* it's the SHUTDOWN
state. Might trace through that code and see if you can figure it out.
Cheers,
Chris
On Wed, Jun 29, 2016 at 6:02 AM, Ryabchuk, Pavlo <
ext-pavlo.ryabchuk@here.com> wrote:
> Hi,
>
> Can someone suggest how can I stop execution/mark(failed or any other
> status) of all sibling tasks (also can be passed as a list) if one of them
> failed?
> It's just useless for me to process other tasks if one failed, because in
> my case it's for sure issue with my processing soft and it will lead to
> fixing and recompilation of it, and in any case reprocessing all tasks with
> newer version.
> I've tried various manipulations with TaskInstances and Jobs states on
> on_failure_callback but nothing seem to work.
>
> Best,
> Paul
>