You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Diogo Franco <di...@gmail.com> on 2017/07/28 14:12:19 UTC
Custom BashSensor
Hi,
I was looking for a sensor that allowed me to customize the sensing logic.
I couldn't find any way of doing this and wrote a BashSensor, which returns
True or False based on a command/script's return code. The implementation
is similar to the BashOperator, and it seems to me that it would fit pretty
naturally within airflow.
Is there another way to achieve this that I don't know about? If you think
it is worthwhile, I'm happy to work on some tests and a PR.
Thank you!
Re: Custom BashSensor
Posted by Stefano Baghino <st...@teralytics.ch>.
I see, thanks for the clarification!
On Sun, Jul 30, 2017 at 3:24 PM, Diogo Franco <di...@gmail.com>
wrote:
> I believe that wouldn't be equivalent, as the use case here is to keep
> poking until a script/bash command returns 0.
>
> To clarify, I needed this because I wanted to wait for a certain condition
> to become True, but checking that condition meant SSH'ing into a machine
> and looking for the presence of a few files, with some logic in between.
>
> Cheers,
>
> On 28 July 2017 at 17:44, Stefano Baghino <st...@teralytics.ch>
> wrote:
>
> > Couldn't this be achieved by simply making the script that needs the
> sensor
> > a dependency of the one that emits the return code?
> >
> > On Fri, Jul 28, 2017 at 4:12 PM, Diogo Franco <diogoalexfranco@gmail.com
> >
> > wrote:
> >
> > > Hi,
> > >
> > > I was looking for a sensor that allowed me to customize the sensing
> > logic.
> > > I couldn't find any way of doing this and wrote a BashSensor, which
> > returns
> > > True or False based on a command/script's return code. The
> implementation
> > > is similar to the BashOperator, and it seems to me that it would fit
> > pretty
> > > naturally within airflow.
> > >
> > > Is there another way to achieve this that I don't know about? If you
> > think
> > > it is worthwhile, I'm happy to work on some tests and a PR.
> > >
> > > Thank you!
> > >
> >
> >
> >
> > --
> > Stefano Baghino | TERALYTICS
> > *software engineer*
> >
> > Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
> > phone: +41 43 508 24 57
> > email: stefano.baghino@teralytics.ch
> > www.teralytics.net
> >
> > Company registration number: CH-020.3.037.709-7 | Trade register Canton
> > Zurich
> > Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz,
> Yann
> > de Vries
> >
> > This e-mail message contains confidential information which is for the
> sole
> > attention and use of the intended recipient. Please notify us at once if
> > you think that it may not be intended for you and delete it immediately.
> >
>
--
Stefano Baghino | TERALYTICS
*software engineer*
Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
phone: +41 43 508 24 57
email: stefano.baghino@teralytics.ch
www.teralytics.net
Company registration number: CH-020.3.037.709-7 | Trade register Canton
Zurich
Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
de Vries
This e-mail message contains confidential information which is for the sole
attention and use of the intended recipient. Please notify us at once if
you think that it may not be intended for you and delete it immediately.
Re: Custom BashSensor
Posted by Diogo Franco <di...@gmail.com>.
I believe that wouldn't be equivalent, as the use case here is to keep
poking until a script/bash command returns 0.
To clarify, I needed this because I wanted to wait for a certain condition
to become True, but checking that condition meant SSH'ing into a machine
and looking for the presence of a few files, with some logic in between.
Cheers,
On 28 July 2017 at 17:44, Stefano Baghino <st...@teralytics.ch>
wrote:
> Couldn't this be achieved by simply making the script that needs the sensor
> a dependency of the one that emits the return code?
>
> On Fri, Jul 28, 2017 at 4:12 PM, Diogo Franco <di...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I was looking for a sensor that allowed me to customize the sensing
> logic.
> > I couldn't find any way of doing this and wrote a BashSensor, which
> returns
> > True or False based on a command/script's return code. The implementation
> > is similar to the BashOperator, and it seems to me that it would fit
> pretty
> > naturally within airflow.
> >
> > Is there another way to achieve this that I don't know about? If you
> think
> > it is worthwhile, I'm happy to work on some tests and a PR.
> >
> > Thank you!
> >
>
>
>
> --
> Stefano Baghino | TERALYTICS
> *software engineer*
>
> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
> phone: +41 43 508 24 57
> email: stefano.baghino@teralytics.ch
> www.teralytics.net
>
> Company registration number: CH-020.3.037.709-7 | Trade register Canton
> Zurich
> Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
> de Vries
>
> This e-mail message contains confidential information which is for the sole
> attention and use of the intended recipient. Please notify us at once if
> you think that it may not be intended for you and delete it immediately.
>
Re: Custom BashSensor
Posted by Stefano Baghino <st...@teralytics.ch>.
Couldn't this be achieved by simply making the script that needs the sensor
a dependency of the one that emits the return code?
On Fri, Jul 28, 2017 at 4:12 PM, Diogo Franco <di...@gmail.com>
wrote:
> Hi,
>
> I was looking for a sensor that allowed me to customize the sensing logic.
> I couldn't find any way of doing this and wrote a BashSensor, which returns
> True or False based on a command/script's return code. The implementation
> is similar to the BashOperator, and it seems to me that it would fit pretty
> naturally within airflow.
>
> Is there another way to achieve this that I don't know about? If you think
> it is worthwhile, I'm happy to work on some tests and a PR.
>
> Thank you!
>
--
Stefano Baghino | TERALYTICS
*software engineer*
Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
phone: +41 43 508 24 57
email: stefano.baghino@teralytics.ch
www.teralytics.net
Company registration number: CH-020.3.037.709-7 | Trade register Canton
Zurich
Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
de Vries
This e-mail message contains confidential information which is for the sole
attention and use of the intended recipient. Please notify us at once if
you think that it may not be intended for you and delete it immediately.
Re: Custom BashSensor
Posted by Diogo Franco <di...@gmail.com>.
Could someone look into merging this?
https://github.com/apache/incubator-airflow/pull/2489
I've been actively maintaining it conflict free for some months.
Thanks!
Cheers,
Diogo
On 4 August 2017 at 15:50, Diogo Franco <di...@gmail.com> wrote:
> A PR implementing this is here https://github.com/
> apache/incubator-airflow/pull/2489
>
> Cheers,
> Diogo
>
> On 30 July 2017 at 14:29, Diogo Franco <di...@gmail.com> wrote:
>
>> Hi Maxime,
>>
>> Awesome, I also believe PythonSensor makes sense in the core package of
>> course. These also allow someone trying out airflow to easily build dag's
>> with sensors without setting up connections, for example.
>>
>> I'll open a different ticket for PythonSensor and do it separately, if
>> that is ok.
>>
>> Cheers.
>>
>> On 28 July 2017 at 17:46, Maxime Beauchemin <ma...@gmail.com>
>> wrote:
>>
>>> BashSensor would be a good contribution.
>>>
>>> I'm guessing that the reason why it doesn't exist or (PythonSensor
>>> doesn't
>>> exist for that matter) is that it's pretty easy to derive
>>> BaseSensorOperator and make your own. It still would be neat to have it
>>> in
>>> the core package and in the docs.
>>>
>>> Max
>>>
>>> On Fri, Jul 28, 2017 at 7:12 AM, Diogo Franco <diogoalexfranco@gmail.com
>>> >
>>> wrote:
>>>
>>> > Hi,
>>> >
>>> > I was looking for a sensor that allowed me to customize the sensing
>>> logic.
>>> > I couldn't find any way of doing this and wrote a BashSensor, which
>>> returns
>>> > True or False based on a command/script's return code. The
>>> implementation
>>> > is similar to the BashOperator, and it seems to me that it would fit
>>> pretty
>>> > naturally within airflow.
>>> >
>>> > Is there another way to achieve this that I don't know about? If you
>>> think
>>> > it is worthwhile, I'm happy to work on some tests and a PR.
>>> >
>>> > Thank you!
>>> >
>>>
>>
>>
>
Re: Custom BashSensor
Posted by Diogo Franco <di...@gmail.com>.
A PR implementing this is here
https://github.com/apache/incubator-airflow/pull/2489
Cheers,
Diogo
On 30 July 2017 at 14:29, Diogo Franco <di...@gmail.com> wrote:
> Hi Maxime,
>
> Awesome, I also believe PythonSensor makes sense in the core package of
> course. These also allow someone trying out airflow to easily build dag's
> with sensors without setting up connections, for example.
>
> I'll open a different ticket for PythonSensor and do it separately, if
> that is ok.
>
> Cheers.
>
> On 28 July 2017 at 17:46, Maxime Beauchemin <ma...@gmail.com>
> wrote:
>
>> BashSensor would be a good contribution.
>>
>> I'm guessing that the reason why it doesn't exist or (PythonSensor doesn't
>> exist for that matter) is that it's pretty easy to derive
>> BaseSensorOperator and make your own. It still would be neat to have it in
>> the core package and in the docs.
>>
>> Max
>>
>> On Fri, Jul 28, 2017 at 7:12 AM, Diogo Franco <di...@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > I was looking for a sensor that allowed me to customize the sensing
>> logic.
>> > I couldn't find any way of doing this and wrote a BashSensor, which
>> returns
>> > True or False based on a command/script's return code. The
>> implementation
>> > is similar to the BashOperator, and it seems to me that it would fit
>> pretty
>> > naturally within airflow.
>> >
>> > Is there another way to achieve this that I don't know about? If you
>> think
>> > it is worthwhile, I'm happy to work on some tests and a PR.
>> >
>> > Thank you!
>> >
>>
>
>
Re: Custom BashSensor
Posted by Diogo Franco <di...@gmail.com>.
Hi Maxime,
Awesome, I also believe PythonSensor makes sense in the core package of
course. These also allow someone trying out airflow to easily build dag's
with sensors without setting up connections, for example.
I'll open a different ticket for PythonSensor and do it separately, if that
is ok.
Cheers.
On 28 July 2017 at 17:46, Maxime Beauchemin <ma...@gmail.com>
wrote:
> BashSensor would be a good contribution.
>
> I'm guessing that the reason why it doesn't exist or (PythonSensor doesn't
> exist for that matter) is that it's pretty easy to derive
> BaseSensorOperator and make your own. It still would be neat to have it in
> the core package and in the docs.
>
> Max
>
> On Fri, Jul 28, 2017 at 7:12 AM, Diogo Franco <di...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I was looking for a sensor that allowed me to customize the sensing
> logic.
> > I couldn't find any way of doing this and wrote a BashSensor, which
> returns
> > True or False based on a command/script's return code. The implementation
> > is similar to the BashOperator, and it seems to me that it would fit
> pretty
> > naturally within airflow.
> >
> > Is there another way to achieve this that I don't know about? If you
> think
> > it is worthwhile, I'm happy to work on some tests and a PR.
> >
> > Thank you!
> >
>
Re: Custom BashSensor
Posted by Maxime Beauchemin <ma...@gmail.com>.
BashSensor would be a good contribution.
I'm guessing that the reason why it doesn't exist or (PythonSensor doesn't
exist for that matter) is that it's pretty easy to derive
BaseSensorOperator and make your own. It still would be neat to have it in
the core package and in the docs.
Max
On Fri, Jul 28, 2017 at 7:12 AM, Diogo Franco <di...@gmail.com>
wrote:
> Hi,
>
> I was looking for a sensor that allowed me to customize the sensing logic.
> I couldn't find any way of doing this and wrote a BashSensor, which returns
> True or False based on a command/script's return code. The implementation
> is similar to the BashOperator, and it seems to me that it would fit pretty
> naturally within airflow.
>
> Is there another way to achieve this that I don't know about? If you think
> it is worthwhile, I'm happy to work on some tests and a PR.
>
> Thank you!
>