Dynamic Rule Evaluation in Flink

[Aarti Gupta <aa...@qualys.com>]

Hi,

We are currently evaluating Flink to build a real time rule engine that
looks at events in a stream and evaluates them against a set of rules.

The rules are dynamically configured and can be of three types -
1. Simple Conditions - these require you to look inside a single event.
Example, match rule if A happens.
2. Aggregations - these require you to aggregate multiple events. Example,
match rule if more than five A's happen.
3. Complex patterns - these require you to look at multiple events and
detect patterns. Example, match rule if A happens and then B happens.

Since the rules are dynamically configured, we cannot use CEP.

As an alternative, we are using connected streams and the CoFlatMap
function to store the rules in shared state, and evaluate each incoming
event against the stored rules.  Implementation is similar to what's
outlined here
<https://data-artisans.com/blog/bettercloud-dynamic-alerting-apache-flink>.

My questions -

   1. Since the CoFlatMap function works on a single event, how do we
   evaluate rules that require aggregations across events. (Match rule if more
   than 5 A events happen)
   2. Since the CoFlatMap function works on a single event, how do we
   evaluate rules that require pattern detection across events (Match rule if
   A happens, followed by B).
   3. How do you dynamically define a window function.


--Aarti



-- 
Aarti Gupta <https://www.linkedin.com/company/qualys>
Director, Engineering, Correlation


aagupta@qualys.com
T


Qualys, Inc. – Blog <https://qualys.com/blog> | Community
<https://community.qualys.com> | Twitter <https://twitter.com/qualys>


<https://www.qualys.com/email-banner>

Re: Dynamic Rule Evaluation in Flink

[Fabian Hueske <fh...@gmail.com>]

Hi Aarti,

Please post your questions to the user@flink.apache.org mailing list.
The community list is meant for community related issues (such as events)
and not for questions regarding the software.

Thanks,
Fabian

2018-07-02 10:52 GMT+02:00 Aarti Gupta <aa...@qualys.com>:

> Hi,
>
> We are currently evaluating Flink to build a real time rule engine that
> looks at events in a stream and evaluates them against a set of rules.
>
> The rules are dynamically configured and can be of three types -
> 1. Simple Conditions - these require you to look inside a single event.
> Example, match rule if A happens.
> 2. Aggregations - these require you to aggregate multiple events. Example,
> match rule if more than five A's happen.
> 3. Complex patterns - these require you to look at multiple events and
> detect patterns. Example, match rule if A happens and then B happens.
>
> Since the rules are dynamically configured, we cannot use CEP.
>
> As an alternative, we are using connected streams and the CoFlatMap
> function to store the rules in shared state, and evaluate each incoming
> event against the stored rules.  Implementation is similar to what's
> outlined here
> <https://data-artisans.com/blog/bettercloud-dynamic-alerting-apache-flink
> >.
>
> My questions -
>
>    1. Since the CoFlatMap function works on a single event, how do we
>    evaluate rules that require aggregations across events. (Match rule if
> more
>    than 5 A events happen)
>    2. Since the CoFlatMap function works on a single event, how do we
>    evaluate rules that require pattern detection across events (Match rule
> if
>    A happens, followed by B).
>    3. How do you dynamically define a window function.
>
>
> --Aarti
>
>
>
> --
> Aarti Gupta <https://www.linkedin.com/company/qualys>
> Director, Engineering, Correlation
>
>
> aagupta@qualys.com
> T
>
>
> Qualys, Inc. – Blog <https://qualys.com/blog> | Community
> <https://community.qualys.com> | Twitter <https://twitter.com/qualys>
>
>
> <https://www.qualys.com/email-banner>
>