You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Kostas Kloudas <k....@data-artisans.com> on 2018/03/01 09:35:23 UTC
Re: Questions about the FlinkCEP
Hi Esa,
The answers to the questions are inlined.
> On Feb 28, 2018, at 8:32 PM, Esa Heikkinen <he...@student.tut.fi> wrote:
>
> Hi
>
> I have tried to learn FlinkCEP [1], but i have yet not found the clear answers for questions:
> 1) Whether the pattern of CEP is meant only for one data stream at the same time ?
Yes.
> 2) If i have many different parallel data streams (or sources), should i combine them into one data stream (and is this possible ?), if i want to use same CEP pattern for all parallel streams at the same time ?
Yes you should somehow combine them. This can be done with .union if this is ok for you logic but
how to do it depends on your job. Or you can just apply the same pattern to all your streams and then
union the result into a single output stream.
> 3) What is the format of data stream of events for CEP ?
The input can be an arbitrary data stream.
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/cep.html <https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/cep.html>
https://flink.apache.org/news/2016/04/06/cep-monitoring.html <https://flink.apache.org/news/2016/04/06/cep-monitoring.html>
https://data-artisans.com/blog/complex-event-processing-flink-cep-update <https://data-artisans.com/blog/complex-event-processing-flink-cep-update>
>
> 4) Can i directly supply a data stream from env.socketTextStream() to CEP ?
Yes. Why don’t you try it out ;)
> 5) Can one event in stream include many "attributes" and values ? Or is it only key-value-pair ? Timestamp of event ?
An event can have arbitrary format. It is up to you to interpret your data.
> 6) Can CEP save the found values of events for use in later "steps" ? For example in pattern:
>
> val pattern = Pattern.begin("start").where(_.getId == 42)
>
>
> .next("middle").subtype(classOf[SubEvent]).where(_.getVolume >= 10.0)
>
>
> .followedBy("end").where(_.getName == "end")
> Whether value of Id in"start"-step can be saved and check whether it (Id) is same in "end"-step ?
>
If I get the question right, you can do it with Iterative conditions, as described in the documentation.
> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/cep.html
>
> Best, Esa
Best,
Kostas
Re: Questions about the FlinkCEP
Posted by Kostas Kloudas <k....@data-artisans.com>.
Hi,
So yes you can do it with IterativeConditions.
Cheers,
Kostas
> On Mar 1, 2018, at 1:15 PM, Esa Heikkinen <es...@student.tut.fi> wrote:
>
>
> Hi
>
> 6) I meant that in the first step the CEP pattern queries value for “Id” and stores the value to (global) variable for later use in the same pattern or even other places in the application. Is this possible ?
>
> Best, Esa
>
> From: Kostas Kloudas [mailto:k.kloudas@data-artisans.com]
> Sent: Thursday, March 1, 2018 11:35 AM
> To: Esa Heikkinen <es...@student.tut.fi>
> Cc: user@flink.apache.org
> Subject: Re: Questions about the FlinkCEP
>
> Hi Esa,
>
> The answers to the questions are inlined.
>
>
> On Feb 28, 2018, at 8:32 PM, Esa Heikkinen <heikkin2@student.tut.fi <ma...@student.tut.fi>> wrote:
>
> Hi
>
> I have tried to learn FlinkCEP [1], but i have yet not found the clear answers for questions:
> 1) Whether the pattern of CEP is meant only for one data stream at the same time ?
> Yes.
>
> 2) If i have many different parallel data streams (or sources), should i combine them into one data stream (and is this possible ?), if i want to use same CEP pattern for all parallel streams at the same time ?
>
> Yes you should somehow combine them. This can be done with .union if this is ok for you logic but
> how to do it depends on your job. Or you can just apply the same pattern to all your streams and then
> union the result into a single output stream.
>
>
> 3) What is the format of data stream of events for CEP ?
>
> The input can be an arbitrary data stream.
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/cep.html <https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/cep.html>
> https://flink.apache.org/news/2016/04/06/cep-monitoring.html <https://flink.apache.org/news/2016/04/06/cep-monitoring.html>
> https://data-artisans.com/blog/complex-event-processing-flink-cep-update <https://data-artisans.com/blog/complex-event-processing-flink-cep-update>
>
>
>
> 4) Can i directly supply a data stream from env.socketTextStream() to CEP ?
>
> Yes. Why don’t you try it out ;)
>
>
> 5) Can one event in stream include many "attributes" and values ? Or is it only key-value-pair ? Timestamp of event ?
>
> An event can have arbitrary format. It is up to you to interpret your data.
>
>
> 6) Can CEP save the found values of events for use in later "steps" ? For example in pattern:
>
> val pattern = Pattern.begin("start").where(_.getId == 42)
>
>
> .next("middle").subtype(classOf[SubEvent]).where(_.getVolume >= 10.0)
>
>
> .followedBy("end").where(_.getName == "end")
> Whether value of Id in"start"-step can be saved and check whether it (Id) is same in "end"-step ?
>
>
> If I get the question right, you can do it with Iterative conditions, as described in the documentation.
>
>
> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/cep.html <https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/cep.html>
>
> Best, Esa
>
> Best,
> Kostas
>
RE: Questions about the FlinkCEP
Posted by Esa Heikkinen <es...@student.tut.fi>.
Hi
6) I meant that in the first step the CEP pattern queries value for “Id” and stores the value to (global) variable for later use in the same pattern or even other places in the application. Is this possible ?
Best, Esa
From: Kostas Kloudas [mailto:k.kloudas@data-artisans.com]
Sent: Thursday, March 1, 2018 11:35 AM
To: Esa Heikkinen <es...@student.tut.fi>
Cc: user@flink.apache.org
Subject: Re: Questions about the FlinkCEP
Hi Esa,
The answers to the questions are inlined.
On Feb 28, 2018, at 8:32 PM, Esa Heikkinen <he...@student.tut.fi>> wrote:
Hi
I have tried to learn FlinkCEP [1], but i have yet not found the clear answers for questions:
1) Whether the pattern of CEP is meant only for one data stream at the same time ?
Yes.
2) If i have many different parallel data streams (or sources), should i combine them into one data stream (and is this possible ?), if i want to use same CEP pattern for all parallel streams at the same time ?
Yes you should somehow combine them. This can be done with .union if this is ok for you logic but
how to do it depends on your job. Or you can just apply the same pattern to all your streams and then
union the result into a single output stream.
3) What is the format of data stream of events for CEP ?
The input can be an arbitrary data stream.
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/cep.html
https://flink.apache.org/news/2016/04/06/cep-monitoring.html
https://data-artisans.com/blog/complex-event-processing-flink-cep-update
4) Can i directly supply a data stream from env.socketTextStream() to CEP ?
Yes. Why don’t you try it out ;)
5) Can one event in stream include many "attributes" and values ? Or is it only key-value-pair ? Timestamp of event ?
An event can have arbitrary format. It is up to you to interpret your data.
6) Can CEP save the found values of events for use in later "steps" ? For example in pattern:
val pattern = Pattern.begin("start").where(_.getId == 42)
.next("middle").subtype(classOf[SubEvent]).where(_.getVolume >= 10.0)
.followedBy("end").where(_.getName == "end")
Whether value of Id in"start"-step can be saved and check whether it (Id) is same in "end"-step ?
If I get the question right, you can do it with Iterative conditions, as described in the documentation.
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/cep.html
Best, Esa
Best,
Kostas