About how to pause the running task

[Song Liu <so...@outlook.com>]

Hi,

A DAG is composed of many tasks, when this DAG is started, how to pause the current running task ?

Thanks,
Song

Re: About how to pause the running task

[Ruiqin Yang <yr...@gmail.com>]

I don't think we have a way to pause but we can stop the running tasks by
changing its state. Currently you can change the state of a running task in
the UI by click on the task instance and mark it success/ clear it, change
its state to failed/success in the CRUD, or clear it through the cli. I'm
currently working on adding the support (PR
<https://github.com/apache/incubator-airflow/pull/3255>) to mark a DagRun
as failed and thus it would be easier to stop all running task instances
for one DagRun.

Cheers,
Kevin Yang

On Thu, Apr 26, 2018 at 10:14 AM, Song Liu <so...@outlook.com> wrote:

> Hi,
>
> A DAG is composed of many tasks, when this DAG is started, how to pause
> the current running task ?
>
> Thanks,
> Song
>

Re: About how to pause the running task

[Maxime Beauchemin <ma...@gmail.com>]

There are no semantics or concept of pause within a task, though you can
clear/kill tasks which is essentially sending a poison pill to kill the
task subprocess.

Even if BaseOperator (common to all tasks) was to implement pausing
semantics, they probably wouldn't be implemented for most operators. Is it
even possible to pause a bash script externally (through SIGTSTP
<https://en.wikipedia.org/wiki/SIGTSTP>, but does that always work)? a
python script? Hive job? A Spark job? Personally I don't think that feature
would get much use. You end up freezing and not releasing a worker slot.

You can pause a DAG though. FYI when a DAG is paused, it only means it will
stop scheduling new tasks.

Max

On Thu, Apr 26, 2018 at 10:14 AM, Song Liu <so...@outlook.com> wrote:

> Hi,
>
> A DAG is composed of many tasks, when this DAG is started, how to pause
> the current running task ?
>
> Thanks,
> Song
>