You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@airflow.apache.org by Niels Zeilemaker <ni...@zeilemaker.nl> on 2017/07/05 14:03:07 UTC

Airflow Notifiers

Hi All,

I’ve opened a pull request
(https://github.com/apache/incubator-airflow/pull/2412) which
introduces the concept of notifiers.

I’ve made this change as I have a requirement to push status changes
of failed/retried jobs to more than only email. Eg, I want to use
slack in this case.

Currently, we can only pass an email notifier, which I feel is pretty limited.

My pull-request changes this and introduces Notifiers, which are
triggered on status changes. I think it’s almost there, but I have
some problems trying to migrate the SLA stuff into the same concept.
Let me know if you have any comments. Or if you feel this is the wrong
approach to my problem.

Niels

Re: Airflow Notifiers

Posted by Niels Zeilemaker <ni...@zeilemaker.nl>.

Hi Maxime,

I've pushed some changes to the pull request. And partially agree with
your comment.
Using the on_..._callbacks you could achieve something similar, but it
would be much more cumbersome I think.
The notifiers you can pass to the DAG using the default_args argument,
which to me seems to be a cleaner solution.

Niels

2017-07-09 6:23 GMT+02:00 Maxime Beauchemin <ma...@gmail.com>:
> I left a few comments on the PR. Note that this approach is marginally more
> flexible/expressive to what can already be accomplished with
> `on_success_callback`, `on_failure_callback` and `on_retry_callback`.
>
> On Wed, Jul 5, 2017 at 7:03 AM, Niels Zeilemaker <ni...@zeilemaker.nl>
> wrote:
>
>> Hi All,
>>
>> I’ve opened a pull request
>> (https://github.com/apache/incubator-airflow/pull/2412) which
>> introduces the concept of notifiers.
>>
>> I’ve made this change as I have a requirement to push status changes
>> of failed/retried jobs to more than only email. Eg, I want to use
>> slack in this case.
>>
>> Currently, we can only pass an email notifier, which I feel is pretty
>> limited.
>>
>> My pull-request changes this and introduces Notifiers, which are
>> triggered on status changes. I think it’s almost there, but I have
>> some problems trying to migrate the SLA stuff into the same concept.
>> Let me know if you have any comments. Or if you feel this is the wrong
>> approach to my problem.
>>
>> Niels
>>

Re: Airflow Notifiers

Posted by Maxime Beauchemin <ma...@gmail.com>.

I left a few comments on the PR. Note that this approach is marginally more
flexible/expressive to what can already be accomplished with
`on_success_callback`, `on_failure_callback` and `on_retry_callback`.

On Wed, Jul 5, 2017 at 7:03 AM, Niels Zeilemaker <ni...@zeilemaker.nl>
wrote:

> Hi All,
>
> I’ve opened a pull request
> (https://github.com/apache/incubator-airflow/pull/2412) which
> introduces the concept of notifiers.
>
> I’ve made this change as I have a requirement to push status changes
> of failed/retried jobs to more than only email. Eg, I want to use
> slack in this case.
>
> Currently, we can only pass an email notifier, which I feel is pretty
> limited.
>
> My pull-request changes this and introduces Notifiers, which are
> triggered on status changes. I think it’s almost there, but I have
> some problems trying to migrate the SLA stuff into the same concept.
> Let me know if you have any comments. Or if you feel this is the wrong
> approach to my problem.
>
> Niels
>