You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@airflow.apache.org by Michael Gong <go...@hotmail.com> on 2017/01/04 20:08:16 UTC

airflow dags share the same database?

Hi,


Say due to some policy, I have to run 2 dag instances on 2 different servers.


If I want task 1 from dag1 depends on task 2 from dag2, how can I do it ?


One idea I can  think of is that the 2 dag instances share the same MySQL db, will it work ?



Any ideas are welcomed.


thanks.

Michael

Re: airflow dags share the same database?

Posted by Michael Gong <go...@hotmail.com>.

Hi, Bolke,

Thanks for the reply.

Can you clarify some points ?

>* SQLSensor pointing to the other database

Do you mean after task2(dag2) is done, it run a SQLSensor to update the db for dag1 ?

> * Put a file somewhere, where it is being polled by the other dag/task

> * Trigger the dag1 from task2 (easier with the Rest API in master)

Do you mean task2 can trigger the entire dag1 or can trigger the task1 in dag1 only ?

Thanks.
Michael

________________________________
From: Bolke de Bruin <bd...@gmail.com>
Sent: Wednesday, January 4, 2017 8:32 PM
To: dev@airflow.incubator.apache.org
Subject: Re: airflow dags share the same database?

Some suggestions:

* SQLSensor pointing to the other database
* Put a file somewhere, where it is being polled by the other dag/task
* Trigger the dag1 from task2 (easier with the Rest API in master)

Bolke

> On 4 Jan 2017, at 21:08, Michael Gong <go...@hotmail.com> wrote:
>
> Hi,
>
>
> Say due to some policy, I have to run 2 dag instances on 2 different servers.
>
>
> If I want task 1 from dag1 depends on task 2 from dag2, how can I do it ?
>
>
> One idea I can  think of is that the 2 dag instances share the same MySQL db, will it work ?
>
>
>
> Any ideas are welcomed.
>
>
> thanks.
>
> Michael
>
>
>
>
>
>
>
>

Re: airflow dags share the same database?

Posted by Bolke de Bruin <bd...@gmail.com>.

Some suggestions:

* SQLSensor pointing to the other database
* Put a file somewhere, where it is being polled by the other dag/task
* Trigger the dag1 from task2 (easier with the Rest API in master)

Bolke

> On 4 Jan 2017, at 21:08, Michael Gong <go...@hotmail.com> wrote:
> 
> Hi,
> 
> 
> Say due to some policy, I have to run 2 dag instances on 2 different servers.
> 
> 
> If I want task 1 from dag1 depends on task 2 from dag2, how can I do it ?
> 
> 
> One idea I can  think of is that the 2 dag instances share the same MySQL db, will it work ?
> 
> 
> 
> Any ideas are welcomed.
> 
> 
> thanks.
> 
> Michael
> 
> 
> 
> 
> 
> 
> 
>