You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Titus Rakkesh <ti...@gmail.com> on 2019/02/08 06:43:33 UTC

Couldn't figure out - How to do this in Flink? - Pls assist with suggestions

Dears,

I have a data stream continuously coming,

DataStream<Tuple3<String, Integer, Double>> splitZTuple;

Eg  - (775168263,113182,0.0)

I have to store this for 24 hrs expiry in somewhere (Window or somewhere)
to check against another stream.

The second stream is

DataStream<Tuple2<String, Double>> splittedVomsTuple which also
continuously receiving one.

Eg. (775168263,100.0)


We need to accumulate the third element in (775168263,113182,*0.0*) in the
WINDOW (If the corresponding first element match happened with the incoming
second streams second element 775168263,*100.0*)

While keeping this WINDOW session if any (775168263,113182,*175*) third
element in the Window Stream exceed a value (Eg >150) we need to call back
a another rest point to send an alert ------- (775168263,113182,*175*)
match the criteria. Simply a CEP call back.


In Flink how we can do this kind of operations? Or do I need to think about
any other framework? Please advise.

Thanks...

Re: Couldn't figure out - How to do this in Flink? - Pls assist with suggestions

Posted by Titus Rakkesh <ti...@gmail.com>.

Thanks Chesnay. I will try that and let you know.

Thanks.

On Sun, Feb 10, 2019 at 2:31 PM Chesnay Schepler <ch...@apache.org> wrote:

> You should be able to use a KeyedProcessFunction
> <https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/process_function.html#the-processfunction>for
> that.
> Find matching elements via keyBy() on the first field.
> Aggregate into ValueState, send alert if necessary.
> Upon encountering a new key, setup a timer to remove the entry in 24h.
>
> On 08.02.2019 07:43, Titus Rakkesh wrote:
>
> Dears,
>
> I have a data stream continuously coming,
>
> DataStream<Tuple3<String, Integer, Double>> splitZTuple;
>
> Eg  - (775168263,113182,0.0)
>
> I have to store this for 24 hrs expiry in somewhere (Window or somewhere)
> to check against another stream.
>
> The second stream is
>
> DataStream<Tuple2<String, Double>> splittedVomsTuple which also
> continuously receiving one.
>
> Eg. (775168263,100.0)
>
>
> We need to accumulate the third element in (775168263,113182,*0.0*) in
> the WINDOW (If the corresponding first element match happened with the
> incoming second streams second element 775168263,*100.0*)
>
> While keeping this WINDOW session if any (775168263,113182,*175*) third
> element in the Window Stream exceed a value (Eg >150) we need to call back
> a another rest point to send an alert ------- (775168263,113182,*175*)
> match the criteria. Simply a CEP call back.
>
>
> In Flink how we can do this kind of operations? Or do I need to think
> about any other framework? Please advise.
>
> Thanks...
>
>
>

Re: Couldn't figure out - How to do this in Flink? - Pls assist with suggestions

Posted by Chesnay Schepler <ch...@apache.org>.

You should be able to use a KeyedProcessFunction 
<https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/process_function.html#the-processfunction>for 
that.
Find matching elements via keyBy() on the first field.
Aggregate into ValueState, send alert if necessary.
Upon encountering a new key, setup a timer to remove the entry in 24h.

On 08.02.2019 07:43, Titus Rakkesh wrote:
>
> Dears,
>
> I have a data stream continuously coming,
>
> DataStream<Tuple3<String, Integer, Double>> splitZTuple;
>
> Eg  - (775168263,113182,0.0)
>
> I have to store this for 24 hrs expiry in somewhere (Window or 
> somewhere) to check against another stream.
>
> The second stream is
>
> DataStream<Tuple2<String, Double>> splittedVomsTuple which also 
> continuously receiving one.
>
> Eg. (775168263,100.0)
>
>
> We need to accumulate the third element in (775168263,113182,*/0.0/*) 
> in the WINDOW (If the corresponding first element match happened with 
> the incoming second streams second element 775168263,*/100.0/*)
>
> While keeping this WINDOW session if any (775168263,113182,*/175/*) 
> third element in the Window Stream exceed a value (Eg >150) we need to 
> call back a another rest point to send an alert ------- 
> (775168263,113182,*/175/*) match the criteria. Simply a CEP call back.
>
>
> In Flink how we can do this kind of operations? Or do I need to think 
> about any other framework? Please advise.
>
> Thanks...
>