You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by ra...@gmail.com,
ra...@gmail.com on 2018/04/12 15:26:20 UTC
Cancel a Running dag
Hi All,
We have a use case to cancel the already running DAG. So is there any recommended way to do so.
Thanks,
Raman
Re: Cancel a Running dag
Posted by ra...@gmail.com,
ra...@gmail.com.
We are exploring following approach for DAG cancellation. Please let us know if you see any issue with this
1) Set/create the xcom variable "cancel":"true". It would be set out of the band by updating the xcom Table in metadata store.
2) Operators would have the code to periodically check for the "cancel" flag. They will read it from xcom.
3) If the flag is set the operators would do the necessary clean up and would exit.
4) Through DAG's dependency condition we can ensure that Downstream tasks are not triggered and are skipped.
Thanks,
Raman GUpta
On 2018/04/13 08:33:50, "Driesprong, Fokko" <fo...@driesprong.frl> wrote:
> Like Bolke said, it has been fixed in master. One of the perquisites is
> support by the operator. For example, the Spark operator has implemented
> how to kill the Spark job on YARN, Local and Kubernetes. If you are running
> something else, you might want to check if this is implemented.
>
> Implemented on_kill: https://github.com/apache/incubator-airflow/pull/3204
> An example of the on_kill:
> https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L485-L534
>
> Cheers, Fokko
>
> 2018-04-12 21:19 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
>
> > This is now fixed in master. Clearing tasks will now properly terminate a
> > running task. If you pause the dag run no new tasks will be scheduled.
> >
> > B.
> >
> >
> >
> > Verstuurd vanaf mijn iPad
> >
> > > Op 12 apr. 2018 om 20:23 heeft Laura Lorenz <ll...@industrydive.com>
> > het volgende geschreven:
> > >
> > > That won't stop them if they are already running in a celery worker or
> > > already in your messaging queue backend (e.g. rabbitmq; redis), but it
> > will
> > > prevent the message to do them from being emitted again by the airflow
> > > scheduler to your messaging queue backend. To be thorough you have to do
> > > both - stop the scheduler from scheduling the tasks anymore (by failing
> > > them individually and/or the DagRun in the metadata database) and, if you
> > > want to make sure the tasks that already got picked up stop and don't try
> > > again, you have to kill their worker processes and make sure your
> > messaging
> > > queue is clean of messages of that task type. If you don't care that any
> > > already started or queued up tasks finish, you can simply doctor the
> > > metadata database.
> > >
> > > Laura
> > >
> > > On Thu, Apr 12, 2018 at 12:40 PM, ramandumcs@gmail.com <
> > ramandumcs@gmail.com
> > >> wrote:
> > >
> > >> Thanks Laura,
> > >> We are using the CeleryExecutor. Just wondering if marking the
> > >> TaskInstances as failed in metadata store would also work.
> > >> -Raman
> > >>
> > >>> On 2018/04/12 16:27:00, Laura Lorenz <ll...@industrydive.com> wrote:
> > >>> I use the CeleryExecutor and have used a mix of `celery control` and
> > >>> messaging queue purges to kill the running tasks and prevent them from
> > >>> being picked up by workers again (respectively), and doctor the DagRun
> > to
> > >>> failed to stop the scheduler from repopulating the message. I think if
> > >> you
> > >>> are using the Local or Sequential Executor you'd have to kill the
> > >> scheduler
> > >>> process.
> > >>>
> > >>> Laura
> > >>>
> > >>> On Thu, Apr 12, 2018 at 12:05 PM, Taylor Edmiston <tedmiston@gmail.com
> > >
> > >>> wrote:
> > >>>
> > >>>> I don't think killing a currently running task is possible today.
> > >>>>
> > >>>> Of course you can pause it from the CLI or web UI so that future runs
> > >> don't
> > >>>> get triggered, but it sounds like that's not what you're looking for.
> > >>>>
> > >>>> Best,
> > >>>> Taylor
> > >>>>
> > >>>> *Taylor Edmiston*
> > >>>> Blog <http://blog.tedmiston.com> | Stack Overflow CV
> > >>>> <https://stackoverflow.com/story/taylor> | LinkedIn
> > >>>> <https://www.linkedin.com/in/tedmiston/> | AngelList
> > >>>> <https://angel.co/taylor>
> > >>>>
> > >>>>
> > >>>> On Thu, Apr 12, 2018 at 11:26 AM, ramandumcs@gmail.com <
> > >>>> ramandumcs@gmail.com
> > >>>>> wrote:
> > >>>>
> > >>>>> Hi All,
> > >>>>> We have a use case to cancel the already running DAG. So is there any
> > >>>>> recommended way to do so.
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Raman
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
>
Re: Cancel a Running dag
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Like Bolke said, it has been fixed in master. One of the perquisites is
support by the operator. For example, the Spark operator has implemented
how to kill the Spark job on YARN, Local and Kubernetes. If you are running
something else, you might want to check if this is implemented.
Implemented on_kill: https://github.com/apache/incubator-airflow/pull/3204
An example of the on_kill:
https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L485-L534
Cheers, Fokko
2018-04-12 21:19 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
> This is now fixed in master. Clearing tasks will now properly terminate a
> running task. If you pause the dag run no new tasks will be scheduled.
>
> B.
>
>
>
> Verstuurd vanaf mijn iPad
>
> > Op 12 apr. 2018 om 20:23 heeft Laura Lorenz <ll...@industrydive.com>
> het volgende geschreven:
> >
> > That won't stop them if they are already running in a celery worker or
> > already in your messaging queue backend (e.g. rabbitmq; redis), but it
> will
> > prevent the message to do them from being emitted again by the airflow
> > scheduler to your messaging queue backend. To be thorough you have to do
> > both - stop the scheduler from scheduling the tasks anymore (by failing
> > them individually and/or the DagRun in the metadata database) and, if you
> > want to make sure the tasks that already got picked up stop and don't try
> > again, you have to kill their worker processes and make sure your
> messaging
> > queue is clean of messages of that task type. If you don't care that any
> > already started or queued up tasks finish, you can simply doctor the
> > metadata database.
> >
> > Laura
> >
> > On Thu, Apr 12, 2018 at 12:40 PM, ramandumcs@gmail.com <
> ramandumcs@gmail.com
> >> wrote:
> >
> >> Thanks Laura,
> >> We are using the CeleryExecutor. Just wondering if marking the
> >> TaskInstances as failed in metadata store would also work.
> >> -Raman
> >>
> >>> On 2018/04/12 16:27:00, Laura Lorenz <ll...@industrydive.com> wrote:
> >>> I use the CeleryExecutor and have used a mix of `celery control` and
> >>> messaging queue purges to kill the running tasks and prevent them from
> >>> being picked up by workers again (respectively), and doctor the DagRun
> to
> >>> failed to stop the scheduler from repopulating the message. I think if
> >> you
> >>> are using the Local or Sequential Executor you'd have to kill the
> >> scheduler
> >>> process.
> >>>
> >>> Laura
> >>>
> >>> On Thu, Apr 12, 2018 at 12:05 PM, Taylor Edmiston <tedmiston@gmail.com
> >
> >>> wrote:
> >>>
> >>>> I don't think killing a currently running task is possible today.
> >>>>
> >>>> Of course you can pause it from the CLI or web UI so that future runs
> >> don't
> >>>> get triggered, but it sounds like that's not what you're looking for.
> >>>>
> >>>> Best,
> >>>> Taylor
> >>>>
> >>>> *Taylor Edmiston*
> >>>> Blog <http://blog.tedmiston.com> | Stack Overflow CV
> >>>> <https://stackoverflow.com/story/taylor> | LinkedIn
> >>>> <https://www.linkedin.com/in/tedmiston/> | AngelList
> >>>> <https://angel.co/taylor>
> >>>>
> >>>>
> >>>> On Thu, Apr 12, 2018 at 11:26 AM, ramandumcs@gmail.com <
> >>>> ramandumcs@gmail.com
> >>>>> wrote:
> >>>>
> >>>>> Hi All,
> >>>>> We have a use case to cancel the already running DAG. So is there any
> >>>>> recommended way to do so.
> >>>>>
> >>>>> Thanks,
> >>>>> Raman
> >>>>>
> >>>>
> >>>
> >>
>
Re: Cancel a Running dag
Posted by ra...@gmail.com,
ra...@gmail.com.
Thanks Bolke,
Will it become part of Airflow 1.10 release. Is there any tentative timeline for same.
-Raman Gupta
On 2018/04/12 19:19:07, Bolke de Bruin <bd...@gmail.com> wrote:
> This is now fixed in master. Clearing tasks will now properly terminate a running task. If you pause the dag run no new tasks will be scheduled.
>
> B.
>
>
>
> Verstuurd vanaf mijn iPad
>
> > Op 12 apr. 2018 om 20:23 heeft Laura Lorenz <ll...@industrydive.com> het volgende geschreven:
> >
> > That won't stop them if they are already running in a celery worker or
> > already in your messaging queue backend (e.g. rabbitmq; redis), but it will
> > prevent the message to do them from being emitted again by the airflow
> > scheduler to your messaging queue backend. To be thorough you have to do
> > both - stop the scheduler from scheduling the tasks anymore (by failing
> > them individually and/or the DagRun in the metadata database) and, if you
> > want to make sure the tasks that already got picked up stop and don't try
> > again, you have to kill their worker processes and make sure your messaging
> > queue is clean of messages of that task type. If you don't care that any
> > already started or queued up tasks finish, you can simply doctor the
> > metadata database.
> >
> > Laura
> >
> > On Thu, Apr 12, 2018 at 12:40 PM, ramandumcs@gmail.com <ramandumcs@gmail.com
> >> wrote:
> >
> >> Thanks Laura,
> >> We are using the CeleryExecutor. Just wondering if marking the
> >> TaskInstances as failed in metadata store would also work.
> >> -Raman
> >>
> >>> On 2018/04/12 16:27:00, Laura Lorenz <ll...@industrydive.com> wrote:
> >>> I use the CeleryExecutor and have used a mix of `celery control` and
> >>> messaging queue purges to kill the running tasks and prevent them from
> >>> being picked up by workers again (respectively), and doctor the DagRun to
> >>> failed to stop the scheduler from repopulating the message. I think if
> >> you
> >>> are using the Local or Sequential Executor you'd have to kill the
> >> scheduler
> >>> process.
> >>>
> >>> Laura
> >>>
> >>> On Thu, Apr 12, 2018 at 12:05 PM, Taylor Edmiston <te...@gmail.com>
> >>> wrote:
> >>>
> >>>> I don't think killing a currently running task is possible today.
> >>>>
> >>>> Of course you can pause it from the CLI or web UI so that future runs
> >> don't
> >>>> get triggered, but it sounds like that's not what you're looking for.
> >>>>
> >>>> Best,
> >>>> Taylor
> >>>>
> >>>> *Taylor Edmiston*
> >>>> Blog <http://blog.tedmiston.com> | Stack Overflow CV
> >>>> <https://stackoverflow.com/story/taylor> | LinkedIn
> >>>> <https://www.linkedin.com/in/tedmiston/> | AngelList
> >>>> <https://angel.co/taylor>
> >>>>
> >>>>
> >>>> On Thu, Apr 12, 2018 at 11:26 AM, ramandumcs@gmail.com <
> >>>> ramandumcs@gmail.com
> >>>>> wrote:
> >>>>
> >>>>> Hi All,
> >>>>> We have a use case to cancel the already running DAG. So is there any
> >>>>> recommended way to do so.
> >>>>>
> >>>>> Thanks,
> >>>>> Raman
> >>>>>
> >>>>
> >>>
> >>
>
Re: Cancel a Running dag
Posted by Bolke de Bruin <bd...@gmail.com>.
This is now fixed in master. Clearing tasks will now properly terminate a running task. If you pause the dag run no new tasks will be scheduled.
B.
Verstuurd vanaf mijn iPad
> Op 12 apr. 2018 om 20:23 heeft Laura Lorenz <ll...@industrydive.com> het volgende geschreven:
>
> That won't stop them if they are already running in a celery worker or
> already in your messaging queue backend (e.g. rabbitmq; redis), but it will
> prevent the message to do them from being emitted again by the airflow
> scheduler to your messaging queue backend. To be thorough you have to do
> both - stop the scheduler from scheduling the tasks anymore (by failing
> them individually and/or the DagRun in the metadata database) and, if you
> want to make sure the tasks that already got picked up stop and don't try
> again, you have to kill their worker processes and make sure your messaging
> queue is clean of messages of that task type. If you don't care that any
> already started or queued up tasks finish, you can simply doctor the
> metadata database.
>
> Laura
>
> On Thu, Apr 12, 2018 at 12:40 PM, ramandumcs@gmail.com <ramandumcs@gmail.com
>> wrote:
>
>> Thanks Laura,
>> We are using the CeleryExecutor. Just wondering if marking the
>> TaskInstances as failed in metadata store would also work.
>> -Raman
>>
>>> On 2018/04/12 16:27:00, Laura Lorenz <ll...@industrydive.com> wrote:
>>> I use the CeleryExecutor and have used a mix of `celery control` and
>>> messaging queue purges to kill the running tasks and prevent them from
>>> being picked up by workers again (respectively), and doctor the DagRun to
>>> failed to stop the scheduler from repopulating the message. I think if
>> you
>>> are using the Local or Sequential Executor you'd have to kill the
>> scheduler
>>> process.
>>>
>>> Laura
>>>
>>> On Thu, Apr 12, 2018 at 12:05 PM, Taylor Edmiston <te...@gmail.com>
>>> wrote:
>>>
>>>> I don't think killing a currently running task is possible today.
>>>>
>>>> Of course you can pause it from the CLI or web UI so that future runs
>> don't
>>>> get triggered, but it sounds like that's not what you're looking for.
>>>>
>>>> Best,
>>>> Taylor
>>>>
>>>> *Taylor Edmiston*
>>>> Blog <http://blog.tedmiston.com> | Stack Overflow CV
>>>> <https://stackoverflow.com/story/taylor> | LinkedIn
>>>> <https://www.linkedin.com/in/tedmiston/> | AngelList
>>>> <https://angel.co/taylor>
>>>>
>>>>
>>>> On Thu, Apr 12, 2018 at 11:26 AM, ramandumcs@gmail.com <
>>>> ramandumcs@gmail.com
>>>>> wrote:
>>>>
>>>>> Hi All,
>>>>> We have a use case to cancel the already running DAG. So is there any
>>>>> recommended way to do so.
>>>>>
>>>>> Thanks,
>>>>> Raman
>>>>>
>>>>
>>>
>>
Re: Cancel a Running dag
Posted by Laura Lorenz <ll...@industrydive.com>.
That won't stop them if they are already running in a celery worker or
already in your messaging queue backend (e.g. rabbitmq; redis), but it will
prevent the message to do them from being emitted again by the airflow
scheduler to your messaging queue backend. To be thorough you have to do
both - stop the scheduler from scheduling the tasks anymore (by failing
them individually and/or the DagRun in the metadata database) and, if you
want to make sure the tasks that already got picked up stop and don't try
again, you have to kill their worker processes and make sure your messaging
queue is clean of messages of that task type. If you don't care that any
already started or queued up tasks finish, you can simply doctor the
metadata database.
Laura
On Thu, Apr 12, 2018 at 12:40 PM, ramandumcs@gmail.com <ramandumcs@gmail.com
> wrote:
> Thanks Laura,
> We are using the CeleryExecutor. Just wondering if marking the
> TaskInstances as failed in metadata store would also work.
> -Raman
>
> On 2018/04/12 16:27:00, Laura Lorenz <ll...@industrydive.com> wrote:
> > I use the CeleryExecutor and have used a mix of `celery control` and
> > messaging queue purges to kill the running tasks and prevent them from
> > being picked up by workers again (respectively), and doctor the DagRun to
> > failed to stop the scheduler from repopulating the message. I think if
> you
> > are using the Local or Sequential Executor you'd have to kill the
> scheduler
> > process.
> >
> > Laura
> >
> > On Thu, Apr 12, 2018 at 12:05 PM, Taylor Edmiston <te...@gmail.com>
> > wrote:
> >
> > > I don't think killing a currently running task is possible today.
> > >
> > > Of course you can pause it from the CLI or web UI so that future runs
> don't
> > > get triggered, but it sounds like that's not what you're looking for.
> > >
> > > Best,
> > > Taylor
> > >
> > > *Taylor Edmiston*
> > > Blog <http://blog.tedmiston.com> | Stack Overflow CV
> > > <https://stackoverflow.com/story/taylor> | LinkedIn
> > > <https://www.linkedin.com/in/tedmiston/> | AngelList
> > > <https://angel.co/taylor>
> > >
> > >
> > > On Thu, Apr 12, 2018 at 11:26 AM, ramandumcs@gmail.com <
> > > ramandumcs@gmail.com
> > > > wrote:
> > >
> > > > Hi All,
> > > > We have a use case to cancel the already running DAG. So is there any
> > > > recommended way to do so.
> > > >
> > > > Thanks,
> > > > Raman
> > > >
> > >
> >
>
Re: Cancel a Running dag
Posted by ra...@gmail.com,
ra...@gmail.com.
Thanks Laura,
We are using the CeleryExecutor. Just wondering if marking the TaskInstances as failed in metadata store would also work.
-Raman
On 2018/04/12 16:27:00, Laura Lorenz <ll...@industrydive.com> wrote:
> I use the CeleryExecutor and have used a mix of `celery control` and
> messaging queue purges to kill the running tasks and prevent them from
> being picked up by workers again (respectively), and doctor the DagRun to
> failed to stop the scheduler from repopulating the message. I think if you
> are using the Local or Sequential Executor you'd have to kill the scheduler
> process.
>
> Laura
>
> On Thu, Apr 12, 2018 at 12:05 PM, Taylor Edmiston <te...@gmail.com>
> wrote:
>
> > I don't think killing a currently running task is possible today.
> >
> > Of course you can pause it from the CLI or web UI so that future runs don't
> > get triggered, but it sounds like that's not what you're looking for.
> >
> > Best,
> > Taylor
> >
> > *Taylor Edmiston*
> > Blog <http://blog.tedmiston.com> | Stack Overflow CV
> > <https://stackoverflow.com/story/taylor> | LinkedIn
> > <https://www.linkedin.com/in/tedmiston/> | AngelList
> > <https://angel.co/taylor>
> >
> >
> > On Thu, Apr 12, 2018 at 11:26 AM, ramandumcs@gmail.com <
> > ramandumcs@gmail.com
> > > wrote:
> >
> > > Hi All,
> > > We have a use case to cancel the already running DAG. So is there any
> > > recommended way to do so.
> > >
> > > Thanks,
> > > Raman
> > >
> >
>
Re: Cancel a Running dag
Posted by Laura Lorenz <ll...@industrydive.com>.
I use the CeleryExecutor and have used a mix of `celery control` and
messaging queue purges to kill the running tasks and prevent them from
being picked up by workers again (respectively), and doctor the DagRun to
failed to stop the scheduler from repopulating the message. I think if you
are using the Local or Sequential Executor you'd have to kill the scheduler
process.
Laura
On Thu, Apr 12, 2018 at 12:05 PM, Taylor Edmiston <te...@gmail.com>
wrote:
> I don't think killing a currently running task is possible today.
>
> Of course you can pause it from the CLI or web UI so that future runs don't
> get triggered, but it sounds like that's not what you're looking for.
>
> Best,
> Taylor
>
> *Taylor Edmiston*
> Blog <http://blog.tedmiston.com> | Stack Overflow CV
> <https://stackoverflow.com/story/taylor> | LinkedIn
> <https://www.linkedin.com/in/tedmiston/> | AngelList
> <https://angel.co/taylor>
>
>
> On Thu, Apr 12, 2018 at 11:26 AM, ramandumcs@gmail.com <
> ramandumcs@gmail.com
> > wrote:
>
> > Hi All,
> > We have a use case to cancel the already running DAG. So is there any
> > recommended way to do so.
> >
> > Thanks,
> > Raman
> >
>
Re: Cancel a Running dag
Posted by Taylor Edmiston <te...@gmail.com>.
I don't think killing a currently running task is possible today.
Of course you can pause it from the CLI or web UI so that future runs don't
get triggered, but it sounds like that's not what you're looking for.
Best,
Taylor
*Taylor Edmiston*
Blog <http://blog.tedmiston.com> | Stack Overflow CV
<https://stackoverflow.com/story/taylor> | LinkedIn
<https://www.linkedin.com/in/tedmiston/> | AngelList
<https://angel.co/taylor>
On Thu, Apr 12, 2018 at 11:26 AM, ramandumcs@gmail.com <ramandumcs@gmail.com
> wrote:
> Hi All,
> We have a use case to cancel the already running DAG. So is there any
> recommended way to do so.
>
> Thanks,
> Raman
>