Re: Alertmanager Sink Connector

[Konstantin Knauf <kn...@apache.org>]

cc user@, bcc dev@

Hi Dhruv,

Yes, this should be totally possible.

For 1, I would use a ProcessFunction to buffer the alerts and register
timers per alert for the repeated firing (or to clear it from state if it
is resolved). From this ProcessFunction you send records to an
AlertManagerSink.

For 2. the AsyncSink, Fabian Paul (cc) might be the best person to judge if
it is a good fit. There is PR [1] to add a blog post about the Async Sink
that might be of help for you in the meantime.

Cheers,

Konstantin

[1] https://github.com/apache/flink-web/pull/517


On Thu, Apr 14, 2022 at 9:43 AM Dhruv Patel <dh...@gmail.com>
wrote:

> Hi,
>    We have a use case where we want to send alerts to Prometheus
> Alertmanager (https://prometheus.io/docs/alerting/latest/alertmanager/)
> from Flink. Each of these alerts have a startsAt, endsAt field along with
> alertMetadata. Alertmanager expects clients to send all the FIRING alerts
> every X minutes (X is configurable) with an updated endsAt time (greater
> than X since alertmanager would drop the alert once endsAt time is reached)
> and once the alert is in RESOLVED state stop sending it. The state updates
> of the alerts would come from Kafka. So the pipeline is something like this
> State Updates to Alerts (Kafka) -> Flink (Some Enrichment) -> Alertmanager
> Sink
>  They provide a rest endpoint where we can post these alerts. I have some
> questions to see if its achievable to develop a sink connector for
> alertmanager in flink?
>
> 1. Is it possible to send data to a sink every X minutes from a custom sink
> connector since I have to somehow simulate a behavior of continuously
> sending the same alerts even because state updates are only received from
> Kafka for FIRING -> RESOLVED state and not for FIRING -> FIRING state? I
> was thinking of having a state of active alerts and somehow the sink
> connector would get the state every X minutes, batch it and then send it to
> alertmanager. However, I am not able to find resources to write some
> implementation around it.
>
> 2. In Flink 1.15, there are Async Sinks (
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink)
> but not much documentation around. Also don't know if it would be
> achievable to write the continuous firing logic in alertmanager
>
> Any other ideas are welcomed.
>
> --
> *Thanks,*
> *Dhruv*
>


-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

Re: Alertmanager Sink Connector

[Fabian Paul <fp...@apache.org>]

Hi Dhruv,

Yes, the AsyncSink might be a good fit to implement your use case. In
general, the AsyncSink is meant to be used for writing to external
systems that do not support native transactions and can therefore not
guarantee exactly-once semantics. As a starting point, you can look at
the existing KinesisStreamsSink [1]. You only have to keep in mind
that duplicate records might be sent to Prometheus after failover.

Best,
Fabian

[1] https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-aws-kinesis-streams/src/main/java/org/apache/flink/connector/kinesis/sink/KinesisStreamsSinkWriter.java

On Wed, Apr 20, 2022 at 1:17 PM Konstantin Knauf <kn...@apache.org> wrote:
>
> cc user@, bcc dev@
>
> Hi Dhruv,
>
> Yes, this should be totally possible.
>
> For 1, I would use a ProcessFunction to buffer the alerts and register timers per alert for the repeated firing (or to clear it from state if it is resolved). From this ProcessFunction you send records to an AlertManagerSink.
>
> For 2. the AsyncSink, Fabian Paul (cc) might be the best person to judge if it is a good fit. There is PR [1] to add a blog post about the Async Sink that might be of help for you in the meantime.
>
> Cheers,
>
> Konstantin
>
> [1] https://github.com/apache/flink-web/pull/517
>
>
> On Thu, Apr 14, 2022 at 9:43 AM Dhruv Patel <dh...@gmail.com> wrote:
>>
>> Hi,
>>    We have a use case where we want to send alerts to Prometheus
>> Alertmanager (https://prometheus.io/docs/alerting/latest/alertmanager/)
>> from Flink. Each of these alerts have a startsAt, endsAt field along with
>> alertMetadata. Alertmanager expects clients to send all the FIRING alerts
>> every X minutes (X is configurable) with an updated endsAt time (greater
>> than X since alertmanager would drop the alert once endsAt time is reached)
>> and once the alert is in RESOLVED state stop sending it. The state updates
>> of the alerts would come from Kafka. So the pipeline is something like this
>> State Updates to Alerts (Kafka) -> Flink (Some Enrichment) -> Alertmanager
>> Sink
>>  They provide a rest endpoint where we can post these alerts. I have some
>> questions to see if its achievable to develop a sink connector for
>> alertmanager in flink?
>>
>> 1. Is it possible to send data to a sink every X minutes from a custom sink
>> connector since I have to somehow simulate a behavior of continuously
>> sending the same alerts even because state updates are only received from
>> Kafka for FIRING -> RESOLVED state and not for FIRING -> FIRING state? I
>> was thinking of having a state of active alerts and somehow the sink
>> connector would get the state every X minutes, batch it and then send it to
>> alertmanager. However, I am not able to find resources to write some
>> implementation around it.
>>
>> 2. In Flink 1.15, there are Async Sinks (
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink)
>> but not much documentation around. Also don't know if it would be
>> achievable to write the continuous firing logic in alertmanager
>>
>> Any other ideas are welcomed.
>>
>> --
>> *Thanks,*
>> *Dhruv*
>
>
>
> --
>
> Konstantin Knauf
>
> https://twitter.com/snntrable
>
> https://github.com/knaufk