Using Contextual Data

[Doron Keller <do...@exabeam.com>]

Hello Flink,

Is it possible to use contextual data as part of Event Processing (CEP)?

I would like to create some tables associated with each key. These tables would need to be updated as information streams in. The values in these tables would also have to be referenced as part of the rule logic. E.g. if a certain value was seen more than x times, trigger.

Finally, when the system is starting up these tables would have to be loaded from a database.

Thanks,
Doron

Re: Using Contextual Data

[Doron Keller <do...@exabeam.com>]

Thank you for your response.

Please let me know if this is doable in CEP, or if you need more information.

Doron

From: "Tzu-Li (Gordon) Tai" <tz...@apache.org>>
Date: Sunday, April 16, 2017 at 11:05 PM
To: "user@flink.apache.org<ma...@flink.apache.org>" <us...@flink.apache.org>>, "kkloudas@apache.org<ma...@apache.org>" <kk...@apache.org>>
Subject: Re: Using Contextual Data

Hi,

It seems like you want to load the context tables from external databases when the CEP operators start, and allow pattern detection logic to refer to those tables. I’m not entirely sure if this possible with the Flink CEP library.
I’ve looped in Klou who is more familiar with the CEP library and would probably have more insight on this (its currently the national holidays so we might not be responsive at the moment).

Otherwise, this is definitely achievable using the DataStream API. More specifically, you would want to implement the rich variant functions and load the contextual tables in `open()`. The occurrence frequency of events would be kept as registered state in the case you described.

Hope this helps!

Cheers,
Gordon



On 14 April 2017 at 7:41:46 AM, Doron Keller (doron@exabeam.com<ma...@exabeam.com>) wrote:
Hello Flink,

Is it possible to use contextual data as part of Event Processing (CEP)?

I would like to create some tables associated with each key. These tables would need to be updated as information streams in. The values in these tables would also have to be referenced as part of the rule logic. E.g. if a certain value was seen more than x times, trigger.

Finally, when the system is starting up these tables would have to be loaded from a database.

Thanks,
Doron

Re: Using Contextual Data

["Tzu-Li (Gordon) Tai" <tz...@apache.org>]

Hi,

It seems like you want to load the context tables from external databases when the CEP operators start, and allow pattern detection logic to refer to those tables. I’m not entirely sure if this possible with the Flink CEP library.
I’ve looped in Klou who is more familiar with the CEP library and would probably have more insight on this (its currently the national holidays so we might not be responsive at the moment).

Otherwise, this is definitely achievable using the DataStream API. More specifically, you would want to implement the rich variant functions and load the contextual tables in `open()`. The occurrence frequency of events would be kept as registered state in the case you described.

Hope this helps!

Cheers,
Gordon


On 14 April 2017 at 7:41:46 AM, Doron Keller (doron@exabeam.com) wrote:

Hello Flink,

 

Is it possible to use contextual data as part of Event Processing (CEP)?

 

I would like to create some tables associated with each key. These tables would need to be updated as information streams in. The values in these tables would also have to be referenced as part of the rule logic. E.g. if a certain value was seen more than x times, trigger.

 

Finally, when the system is starting up these tables would have to be loaded from a database.

 

Thanks,

Doron