You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Zsolt Tóth <to...@gmail.com> on 2017/06/02 10:28:12 UTC

Executing Dags atomically

Hi,

is there a way to execute all tasks of a dag without interruption? Right
now the scheduler jumps between dags, and eventually they will all finish.
What I want is to force the scheduler to finish a dag completely, and jump
to the next one afterwards.
I think I could set all tasks of the dag to the same priority, giving each
dag a different prio this way, but then I'll also have a precedence between
DAGs, which I don't want.

Thanks,
Zsolt

Re: Executing Dags atomically

Posted by Zsolt Tóth <to...@gmail.com>.
Hi Scott,

thanks for the answer. However, I absolutely don't want to make one Dag
dependent of the other. I don't care if Dag1 fails, I still want to execute
Dag2. I also don't care about the execution order of the Dags.

Zsolt

2017-06-02 15:21 GMT+02:00 Scott Halgrim <sc...@zapier.com>:

> Try an ExternalTaskSensor
>
> Thanks,
>
> Scott
>
> On Jun 2, 2017, 3:28 AM -0700, Zsolt Tóth <to...@gmail.com>,
> wrote:
> > Hi,
> >
> > is there a way to execute all tasks of a dag without interruption? Right
> > now the scheduler jumps between dags, and eventually they will all
> finish.
> > What I want is to force the scheduler to finish a dag completely, and
> jump
> > to the next one afterwards.
> > I think I could set all tasks of the dag to the same priority, giving
> each
> > dag a different prio this way, but then I'll also have a precedence
> between
> > DAGs, which I don't want.
> >
> > Thanks,
> > Zsolt
>

Re: Executing Dags atomically

Posted by Scott Halgrim <sc...@zapier.com>.
Try an ExternalTaskSensor

Thanks,

Scott

On Jun 2, 2017, 3:28 AM -0700, Zsolt Tóth <to...@gmail.com>, wrote:
> Hi,
>
> is there a way to execute all tasks of a dag without interruption? Right
> now the scheduler jumps between dags, and eventually they will all finish.
> What I want is to force the scheduler to finish a dag completely, and jump
> to the next one afterwards.
> I think I could set all tasks of the dag to the same priority, giving each
> dag a different prio this way, but then I'll also have a precedence between
> DAGs, which I don't want.
>
> Thanks,
> Zsolt