You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Rob Froetscher <rf...@lumoslabs.com> on 2016/11/17 18:52:54 UTC
Re: external task dependency across 2 different servers
Are you saying you have multiple installations of Airflow, or simply that
your airflow setup is distributed, with celery workers on their own nodes?
If the latter, ExternalTaskSensor should still do the trick for you, as I
believe it just checks airflow's operational database for the status of
this external task.
On Thu, Nov 17, 2016 at 10:40 AM, Michael Gong <go...@hotmail.com> wrote:
> Hi,
>
>
> Is it possible that 1 task can depend on another task running on another
> DAG on another sever?
>
>
> For example, I have task1 in dag1, and task2 in dag2.
>
>
> If both dag1 and dag2 running on the same airflow instance on the same
> server, airflow provides the ExternalTaskSensor operator.
>
>
> But if dag2 runs on another airflow instance on another sever, can airflow
> allow dag1's task1 depend on dag2's task2?
>
>
>
> Any suggestions are welcomed.
>
>
> thanks
>
>
>
Re: external task dependency across 2 different servers
Posted by Bolke de Bruin <bd...@gmail.com>.
Looking ahead forward to a not too distant future: I’m currently finishing up the foundations for a Rest API. It would be trivial to add a call that returns the state of a task in this API. Then having a federated or even fully decoupled HA becomes a possibility. Obviously it is experimental at the moment cause it will need to settle a bit.
https://github.com/apache/incubator-airflow/pull/1783
Bolke
> Op 17 nov. 2016, om 22:47 heeft Rob Froetscher <rf...@lumoslabs.com> het volgende geschreven:
>
> Hey Michael,
>
> If you have separate installations with separate operational databases,
> then the ExternalTaskSensor will not do what you need. However, I'm not
> sure I understand why you would need separate Airflow installations that
> rely upon each other. Using Celery, you should be able to scale a single
> installation of Airflow as much as you need.
>
> Regarding setting up celery worker nodes. There is some information on that
> in the Airflow Documentation
> <https://airflow.incubator.apache.org/configuration.html#scaling-out-with-celery>
> .
>
> On Thu, Nov 17, 2016 at 11:48 AM, Michael Gong <go...@hotmail.com> wrote:
>
>> Rob,
>>
>> thanks for reply.
>>
>>
>> If I have multiple installations of airflow on different servers, will
>> this dependency be done ?
>>
>>
>> For the second setup, ie. 1 installation of airflow and it is distributed,
>> can you advise how to set it up ? I am not familiar with that.
>>
>>
>> Thanks.
>>
>>
>>
>> ________________________________
>> From: Rob Froetscher <rf...@lumoslabs.com>
>> Sent: Thursday, November 17, 2016 6:52 PM
>> To: dev@airflow.incubator.apache.org
>> Subject: Re: external task dependency across 2 different servers
>>
>> Are you saying you have multiple installations of Airflow, or simply that
>> your airflow setup is distributed, with celery workers on their own nodes?
>>
>> If the latter, ExternalTaskSensor should still do the trick for you, as I
>> believe it just checks airflow's operational database for the status of
>> this external task.
>>
>> On Thu, Nov 17, 2016 at 10:40 AM, Michael Gong <go...@hotmail.com> wrote:
>>
>>> Hi,
>>>
>>>
>>> Is it possible that 1 task can depend on another task running on another
>>> DAG on another sever?
>>>
>>>
>>> For example, I have task1 in dag1, and task2 in dag2.
>>>
>>>
>>> If both dag1 and dag2 running on the same airflow instance on the same
>>> server, airflow provides the ExternalTaskSensor operator.
>>>
>>>
>>> But if dag2 runs on another airflow instance on another sever, can
>> airflow
>>> allow dag1's task1 depend on dag2's task2?
>>>
>>>
>>>
>>> Any suggestions are welcomed.
>>>
>>>
>>> thanks
>>>
>>>
>>>
>>
Re: external task dependency across 2 different servers
Posted by Maxime Beauchemin <ma...@gmail.com>.
You can simply use SqlSensor poking at the Airflow metadata database of
your other cluster.
https://airflow.apache.org/code.html?highlight=sqlsensor#airflow.operators.SqlSensor
Max
On Fri, Nov 18, 2016 at 7:26 AM, Michael Gong <go...@hotmail.com> wrote:
> Hey Rob,
>
>
> Thanks for the valuable information.
>
>
> Currently, our airflow configuration does use Celery executor and Redis
> backend.
>
> But the worker processes are all created locally.
>
>
> I assume you mean that Celery support distributed worker process to
> different servers/nodes.
>
> If that's the case, do you know any guideline of how to setup the Celery
> with distributed worker ?
>
>
> Any advise is welcomed.
>
>
> thanks.
>
> Michael
>
>
>
>
>
> ________________________________
> From: Rob Froetscher <rf...@lumoslabs.com>
> Sent: Thursday, November 17, 2016 9:47 PM
> To: dev@airflow.incubator.apache.org
> Subject: Re: external task dependency across 2 different servers
>
> Hey Michael,
>
> If you have separate installations with separate operational databases,
> then the ExternalTaskSensor will not do what you need. However, I'm not
> sure I understand why you would need separate Airflow installations that
> rely upon each other. Using Celery, you should be able to scale a single
> installation of Airflow as much as you need.
>
> Regarding setting up celery worker nodes. There is some information on that
> in the Airflow Documentation
> <https://airflow.incubator.apache.org/configuration.html#
> scaling-out-with-celery>
> .
>
> On Thu, Nov 17, 2016 at 11:48 AM, Michael Gong <go...@hotmail.com> wrote:
>
> > Rob,
> >
> > thanks for reply.
> >
> >
> > If I have multiple installations of airflow on different servers, will
> > this dependency be done ?
> >
> >
> > For the second setup, ie. 1 installation of airflow and it is
> distributed,
> > can you advise how to set it up ? I am not familiar with that.
> >
> >
> > Thanks.
> >
> >
> >
> > ________________________________
> > From: Rob Froetscher <rf...@lumoslabs.com>
> > Sent: Thursday, November 17, 2016 6:52 PM
> > To: dev@airflow.incubator.apache.org
> > Subject: Re: external task dependency across 2 different servers
> >
> > Are you saying you have multiple installations of Airflow, or simply that
> > your airflow setup is distributed, with celery workers on their own
> nodes?
> >
> > If the latter, ExternalTaskSensor should still do the trick for you, as I
> > believe it just checks airflow's operational database for the status of
> > this external task.
> >
> > On Thu, Nov 17, 2016 at 10:40 AM, Michael Gong <go...@hotmail.com>
> wrote:
> >
> > > Hi,
> > >
> > >
> > > Is it possible that 1 task can depend on another task running on
> another
> > > DAG on another sever?
> > >
> > >
> > > For example, I have task1 in dag1, and task2 in dag2.
> > >
> > >
> > > If both dag1 and dag2 running on the same airflow instance on the same
> > > server, airflow provides the ExternalTaskSensor operator.
> > >
> > >
> > > But if dag2 runs on another airflow instance on another sever, can
> > airflow
> > > allow dag1's task1 depend on dag2's task2?
> > >
> > >
> > >
> > > Any suggestions are welcomed.
> > >
> > >
> > > thanks
> > >
> > >
> > >
> >
>
Re: external task dependency across 2 different servers
Posted by Michael Gong <go...@hotmail.com>.
Hey Rob,
Thanks for the valuable information.
Currently, our airflow configuration does use Celery executor and Redis backend.
But the worker processes are all created locally.
I assume you mean that Celery support distributed worker process to different servers/nodes.
If that's the case, do you know any guideline of how to setup the Celery with distributed worker ?
Any advise is welcomed.
thanks.
Michael
________________________________
From: Rob Froetscher <rf...@lumoslabs.com>
Sent: Thursday, November 17, 2016 9:47 PM
To: dev@airflow.incubator.apache.org
Subject: Re: external task dependency across 2 different servers
Hey Michael,
If you have separate installations with separate operational databases,
then the ExternalTaskSensor will not do what you need. However, I'm not
sure I understand why you would need separate Airflow installations that
rely upon each other. Using Celery, you should be able to scale a single
installation of Airflow as much as you need.
Regarding setting up celery worker nodes. There is some information on that
in the Airflow Documentation
<https://airflow.incubator.apache.org/configuration.html#scaling-out-with-celery>
.
On Thu, Nov 17, 2016 at 11:48 AM, Michael Gong <go...@hotmail.com> wrote:
> Rob,
>
> thanks for reply.
>
>
> If I have multiple installations of airflow on different servers, will
> this dependency be done ?
>
>
> For the second setup, ie. 1 installation of airflow and it is distributed,
> can you advise how to set it up ? I am not familiar with that.
>
>
> Thanks.
>
>
>
> ________________________________
> From: Rob Froetscher <rf...@lumoslabs.com>
> Sent: Thursday, November 17, 2016 6:52 PM
> To: dev@airflow.incubator.apache.org
> Subject: Re: external task dependency across 2 different servers
>
> Are you saying you have multiple installations of Airflow, or simply that
> your airflow setup is distributed, with celery workers on their own nodes?
>
> If the latter, ExternalTaskSensor should still do the trick for you, as I
> believe it just checks airflow's operational database for the status of
> this external task.
>
> On Thu, Nov 17, 2016 at 10:40 AM, Michael Gong <go...@hotmail.com> wrote:
>
> > Hi,
> >
> >
> > Is it possible that 1 task can depend on another task running on another
> > DAG on another sever?
> >
> >
> > For example, I have task1 in dag1, and task2 in dag2.
> >
> >
> > If both dag1 and dag2 running on the same airflow instance on the same
> > server, airflow provides the ExternalTaskSensor operator.
> >
> >
> > But if dag2 runs on another airflow instance on another sever, can
> airflow
> > allow dag1's task1 depend on dag2's task2?
> >
> >
> >
> > Any suggestions are welcomed.
> >
> >
> > thanks
> >
> >
> >
>
Re: external task dependency across 2 different servers
Posted by Rob Froetscher <rf...@lumoslabs.com>.
Hey Michael,
If you have separate installations with separate operational databases,
then the ExternalTaskSensor will not do what you need. However, I'm not
sure I understand why you would need separate Airflow installations that
rely upon each other. Using Celery, you should be able to scale a single
installation of Airflow as much as you need.
Regarding setting up celery worker nodes. There is some information on that
in the Airflow Documentation
<https://airflow.incubator.apache.org/configuration.html#scaling-out-with-celery>
.
On Thu, Nov 17, 2016 at 11:48 AM, Michael Gong <go...@hotmail.com> wrote:
> Rob,
>
> thanks for reply.
>
>
> If I have multiple installations of airflow on different servers, will
> this dependency be done ?
>
>
> For the second setup, ie. 1 installation of airflow and it is distributed,
> can you advise how to set it up ? I am not familiar with that.
>
>
> Thanks.
>
>
>
> ________________________________
> From: Rob Froetscher <rf...@lumoslabs.com>
> Sent: Thursday, November 17, 2016 6:52 PM
> To: dev@airflow.incubator.apache.org
> Subject: Re: external task dependency across 2 different servers
>
> Are you saying you have multiple installations of Airflow, or simply that
> your airflow setup is distributed, with celery workers on their own nodes?
>
> If the latter, ExternalTaskSensor should still do the trick for you, as I
> believe it just checks airflow's operational database for the status of
> this external task.
>
> On Thu, Nov 17, 2016 at 10:40 AM, Michael Gong <go...@hotmail.com> wrote:
>
> > Hi,
> >
> >
> > Is it possible that 1 task can depend on another task running on another
> > DAG on another sever?
> >
> >
> > For example, I have task1 in dag1, and task2 in dag2.
> >
> >
> > If both dag1 and dag2 running on the same airflow instance on the same
> > server, airflow provides the ExternalTaskSensor operator.
> >
> >
> > But if dag2 runs on another airflow instance on another sever, can
> airflow
> > allow dag1's task1 depend on dag2's task2?
> >
> >
> >
> > Any suggestions are welcomed.
> >
> >
> > thanks
> >
> >
> >
>
Re: external task dependency across 2 different servers
Posted by "Wong, Catherine" <Ca...@nike.com>.
We are thinking about the same thing. We are going to write a custom plugins similar to the ExternalTaskSensor and check for the task success on the different backend database.
Please let me know if you have better idea.
Thanks,
Catherine
On 11/17/16, 11:48 AM, "Michael Gong" <go...@hotmail.com> wrote:
Rob,
thanks for reply.
If I have multiple installations of airflow on different servers, will this dependency be done ?
For the second setup, ie. 1 installation of airflow and it is distributed, can you advise how to set it up ? I am not familiar with that.
Thanks.
________________________________
From: Rob Froetscher <rf...@lumoslabs.com>
Sent: Thursday, November 17, 2016 6:52 PM
To: dev@airflow.incubator.apache.org
Subject: Re: external task dependency across 2 different servers
Are you saying you have multiple installations of Airflow, or simply that
your airflow setup is distributed, with celery workers on their own nodes?
If the latter, ExternalTaskSensor should still do the trick for you, as I
believe it just checks airflow's operational database for the status of
this external task.
On Thu, Nov 17, 2016 at 10:40 AM, Michael Gong <go...@hotmail.com> wrote:
> Hi,
>
>
> Is it possible that 1 task can depend on another task running on another
> DAG on another sever?
>
>
> For example, I have task1 in dag1, and task2 in dag2.
>
>
> If both dag1 and dag2 running on the same airflow instance on the same
> server, airflow provides the ExternalTaskSensor operator.
>
>
> But if dag2 runs on another airflow instance on another sever, can airflow
> allow dag1's task1 depend on dag2's task2?
>
>
>
> Any suggestions are welcomed.
>
>
> thanks
>
>
>
Re: external task dependency across 2 different servers
Posted by Michael Gong <go...@hotmail.com>.
Rob,
thanks for reply.
If I have multiple installations of airflow on different servers, will this dependency be done ?
For the second setup, ie. 1 installation of airflow and it is distributed, can you advise how to set it up ? I am not familiar with that.
Thanks.
________________________________
From: Rob Froetscher <rf...@lumoslabs.com>
Sent: Thursday, November 17, 2016 6:52 PM
To: dev@airflow.incubator.apache.org
Subject: Re: external task dependency across 2 different servers
Are you saying you have multiple installations of Airflow, or simply that
your airflow setup is distributed, with celery workers on their own nodes?
If the latter, ExternalTaskSensor should still do the trick for you, as I
believe it just checks airflow's operational database for the status of
this external task.
On Thu, Nov 17, 2016 at 10:40 AM, Michael Gong <go...@hotmail.com> wrote:
> Hi,
>
>
> Is it possible that 1 task can depend on another task running on another
> DAG on another sever?
>
>
> For example, I have task1 in dag1, and task2 in dag2.
>
>
> If both dag1 and dag2 running on the same airflow instance on the same
> server, airflow provides the ExternalTaskSensor operator.
>
>
> But if dag2 runs on another airflow instance on another sever, can airflow
> allow dag1's task1 depend on dag2's task2?
>
>
>
> Any suggestions are welcomed.
>
>
> thanks
>
>
>