You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by jaxbihani <ja...@helpshift.com> on 2017/04/11 10:15:59 UTC
Custom timer implementation using Flink
I have a use case which I am trying to solve using Flink. Need an advice to
decide on the correct approach.
Use case:
----------
I have a stream of events partitioned by a key. For some events, I need to
start a timer (conside this as a SLA i.e. if something is not done in x
secs/minutes do something). Now when that timer expires I need to perform
some arbitrary action (like writing to database etc). There will be some
events which can cancel the timers. (i.e. if event comes before x secs we
need not run SLA violation action etc.). We are considering flink because
then we can reuse the scaling, fault tolerance provided by the engine rather
than building our own. Current rps is ~ 200-300 but it can be expected to
increase quickly.
Solutions in mind:
-------------------
1. We can think it like CEP use case, where with encoding like "event1
followed by event2" with "not" in x seconds. i.e. when event 2 is "not"
arrived in x seconds. I assume there will be NOT operator support. I am not
sure about memory consumption in CEP. Because x seconds can be x days as
well and I do not need any batching of events in memory. I just need to
start a timer of x days/hours (lets say) and when it is fired just trigger
something. So there is no notion of window as such. Can CEP fit in this type
of use case? If the timer between events is in days, how about the memory
consumption?
2. Use Flink for event processing and delegate the tasks of timers to
another service i.e. when event occurs send it to kafka with timer
information and then another service handles timers and send back the event
again once that is done etc. Looks like many hops in this process and
latency will be high if SLA is in seconds (I am thinking of using Kafka
here).
Is anyone aware of a better way of doing this in flink?
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Custom-timer-implementation-using-Flink-tp12581.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Custom timer implementation using Flink
Posted by jaxbihani <ja...@helpshift.com>.
Hi Gordon
I somehow missed this in the docs. Looks really helpful for this use case.
Thanks.
I now have some other ideas for this use case. I will post here if I use
this or not. If not, will post the approach which I will take.
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Custom-timer-implementation-using-Flink-tp12581p12592.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Custom timer implementation using Flink
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi,
I just need to
start a timer of x days/hours (lets say) and when it is fired just trigger
something.
Flink’s lower-level ProcessFunction [1] should be very suitable to implement this. Have you taken a look at this and see if it suits your case?
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/stream/process_function.html
Cheers,
Gordon
On April 11, 2017 at 3:25:39 AM, jaxbihani (jagadish@helpshift.com) wrote:
I have a use case which I am trying to solve using Flink. Need an advice to
decide on the correct approach.
Use case:
----------
I have a stream of events partitioned by a key. For some events, I need to
start a timer (conside this as a SLA i.e. if something is not done in x
secs/minutes do something). Now when that timer expires I need to perform
some arbitrary action (like writing to database etc). There will be some
events which can cancel the timers. (i.e. if event comes before x secs we
need not run SLA violation action etc.). We are considering flink because
then we can reuse the scaling, fault tolerance provided by the engine rather
than building our own. Current rps is ~ 200-300 but it can be expected to
increase quickly.
Solutions in mind:
-------------------
1. We can think it like CEP use case, where with encoding like "event1
followed by event2" with "not" in x seconds. i.e. when event 2 is "not"
arrived in x seconds. I assume there will be NOT operator support. I am not
sure about memory consumption in CEP. Because x seconds can be x days as
well and I do not need any batching of events in memory. I just need to
start a timer of x days/hours (lets say) and when it is fired just trigger
something. So there is no notion of window as such. Can CEP fit in this type
of use case? If the timer between events is in days, how about the memory
consumption?
2. Use Flink for event processing and delegate the tasks of timers to
another service i.e. when event occurs send it to kafka with timer
information and then another service handles timers and send back the event
again once that is done etc. Looks like many hops in this process and
latency will be high if SLA is in seconds (I am thinking of using Kafka
here).
Is anyone aware of a better way of doing this in flink?
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Custom-timer-implementation-using-Flink-tp12581.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.