You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Manas Kale <ma...@gmail.com> on 2020/04/02 05:03:32 UTC

Perform processing only when watermark updates, buffer data otherwise

Hi,
I want to perform some processing on events only when the watermark is
updated. Otherwise, for all other events, I want to keep buffering them
till the watermark arrives.
The main motivation behind doing this is that I have several operators that
emit events/messages to a downstream operator. Since the order in which
events arrive at the downstream operator is not guaranteed to be in
chronological event time, I want to manually sort events when the watermark
arrives and only then proceed.

Specifically, I want to first combine multiple streams and then do the
above. Something like :
stream1.union(stream2, steream3)...

One solution I am exploring is using a global window with a trigger that
will fire only when the watermark updates.
stream1.union(stream2, steream3).
keyBy(...).
window(GlobalWindows.create()).
trigger(new OnWatermarkUpdateTrigger()).
process(...)

I will store the latest watermark in the trigger's state store. In the
onElement() method, I will FIRE if the current watermark is different than
the stored one.

Is this the best way to implement the functionality described above?

Re: Perform processing only when watermark updates, buffer data otherwise

Posted by Manas Kale <ma...@gmail.com>.
Hi Timo,
Thanks for the information.

On Thu, Apr 2, 2020 at 9:30 PM Timo Walther <tw...@apache.org> wrote:

> Hi Manas,
>
> first of all, after assigning watermarks at the source level, usually
> Flink operators make sure to handle the watermarks.
>
> In case of a `union()`, the subsequent operator will increment its
> internal event-time clock and emit a new watermark only if all input
> streams (and their parallel instances) have reached a common event-time.
>
> Your sorting use case can be easily done with a KeyedProcessFunction
> [1]. You can buffer your events in a list state, and process them when a
> timer fires. The documentation also explains how to set a timer.
>
> If you want to fire when the next watermark arrives, you can set a timer
> like:
>
> ctx.timerService().currentWatermark() + 1
>
> The `union()` is meant for combining streams of the same data into one
> where the order of the event does not matter. However, watermarks are
> still arriving in order so a sorting by event-time should not be a problem.
>
> connect() is broader than a join (see also the answer here [2]).
>
> I hope I could answer most of your questions. Feel free to ask further
> questions.
>
> Regards,
> Timo
>
>
> [1]
>
> https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/process_function.html#process-function
> [2]
>
> https://stackoverflow.com/questions/52885926/what-is-the-difference-between-flink-join-and-connect
>
>
>
> On 02.04.20 12:11, Manas Kale wrote:
> > Also
> >
> >   *   What happens to watermarks after a union operation? Do I have to
> >     assignTimestampsAndWatermarks() again? I guess I will have to since
> >     multiple streams are being combined and Flink needs to know how to
> >     resolve individual watermarks.
> >   * What is the difference between union() and connect()?
> >
> >
> > On Thu, Apr 2, 2020 at 10:33 AM Manas Kale <manaskale96@gmail.com
> > <ma...@gmail.com>> wrote:
> >
> >     Hi,
> >     I want to perform some processing on events only when the watermark
> >     is updated. Otherwise, for all other events, I want to keep
> >     buffering them till the watermark arrives.
> >     The main motivation behind doing this is that I have several
> >     operators that emit events/messages to a downstream operator. Since
> >     the order in which events arrive at the downstream operator is not
> >     guaranteed to be in chronological event time, I want to manually
> >     sort events when the watermark arrives and only then proceed.
> >
> >     Specifically, I want to first combine multiple streams and then do
> >     the above. Something like :
> >     stream1.union(stream2, steream3)...
> >
> >     One solution I am exploring is using a global window with a trigger
> >     that will fire only when the watermark updates.
> >     stream1.union(stream2, steream3).
> >     keyBy(...).
> >     window(GlobalWindows.create()).
> >     trigger(new OnWatermarkUpdateTrigger()).
> >     process(...)
> >
> >     I will store the latest watermark in the trigger's state store. In
> >     the onElement() method, I will FIRE if the current watermark is
> >     different than the stored one.
> >
> >     Is this the best way to implement the functionality described above?
> >
>
>

Re: Perform processing only when watermark updates, buffer data otherwise

Posted by Timo Walther <tw...@apache.org>.
Hi Manas,

first of all, after assigning watermarks at the source level, usually 
Flink operators make sure to handle the watermarks.

In case of a `union()`, the subsequent operator will increment its 
internal event-time clock and emit a new watermark only if all input 
streams (and their parallel instances) have reached a common event-time.

Your sorting use case can be easily done with a KeyedProcessFunction 
[1]. You can buffer your events in a list state, and process them when a 
timer fires. The documentation also explains how to set a timer.

If you want to fire when the next watermark arrives, you can set a timer 
like:

ctx.timerService().currentWatermark() + 1

The `union()` is meant for combining streams of the same data into one 
where the order of the event does not matter. However, watermarks are 
still arriving in order so a sorting by event-time should not be a problem.

connect() is broader than a join (see also the answer here [2]).

I hope I could answer most of your questions. Feel free to ask further 
questions.

Regards,
Timo


[1] 
https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/process_function.html#process-function
[2] 
https://stackoverflow.com/questions/52885926/what-is-the-difference-between-flink-join-and-connect



On 02.04.20 12:11, Manas Kale wrote:
> Also
> 
>   *   What happens to watermarks after a union operation? Do I have to
>     assignTimestampsAndWatermarks() again? I guess I will have to since
>     multiple streams are being combined and Flink needs to know how to
>     resolve individual watermarks.
>   * What is the difference between union() and connect()?
> 
> 
> On Thu, Apr 2, 2020 at 10:33 AM Manas Kale <manaskale96@gmail.com 
> <ma...@gmail.com>> wrote:
> 
>     Hi,
>     I want to perform some processing on events only when the watermark
>     is updated. Otherwise, for all other events, I want to keep
>     buffering them till the watermark arrives.
>     The main motivation behind doing this is that I have several
>     operators that emit events/messages to a downstream operator. Since
>     the order in which events arrive at the downstream operator is not
>     guaranteed to be in chronological event time, I want to manually
>     sort events when the watermark arrives and only then proceed.
> 
>     Specifically, I want to first combine multiple streams and then do
>     the above. Something like :
>     stream1.union(stream2, steream3)...
> 
>     One solution I am exploring is using a global window with a trigger
>     that will fire only when the watermark updates.
>     stream1.union(stream2, steream3).
>     keyBy(...).
>     window(GlobalWindows.create()).
>     trigger(new OnWatermarkUpdateTrigger()).
>     process(...)
> 
>     I will store the latest watermark in the trigger's state store. In
>     the onElement() method, I will FIRE if the current watermark is
>     different than the stored one.
> 
>     Is this the best way to implement the functionality described above?
> 


Re: Perform processing only when watermark updates, buffer data otherwise

Posted by Manas Kale <ma...@gmail.com>.
Also

   -  What happens to watermarks after a union operation? Do I have to
   assignTimestampsAndWatermarks() again? I guess I will have to since
   multiple streams are being combined and Flink needs to know how to resolve
   individual watermarks.
   - What is the difference between union() and connect()?


On Thu, Apr 2, 2020 at 10:33 AM Manas Kale <ma...@gmail.com> wrote:

> Hi,
> I want to perform some processing on events only when the watermark is
> updated. Otherwise, for all other events, I want to keep buffering them
> till the watermark arrives.
> The main motivation behind doing this is that I have several operators
> that emit events/messages to a downstream operator. Since the order in
> which events arrive at the downstream operator is not guaranteed to be in
> chronological event time, I want to manually sort events when the watermark
> arrives and only then proceed.
>
> Specifically, I want to first combine multiple streams and then do the
> above. Something like :
> stream1.union(stream2, steream3)...
>
> One solution I am exploring is using a global window with a trigger that
> will fire only when the watermark updates.
> stream1.union(stream2, steream3).
> keyBy(...).
> window(GlobalWindows.create()).
> trigger(new OnWatermarkUpdateTrigger()).
> process(...)
>
> I will store the latest watermark in the trigger's state store. In the
> onElement() method, I will FIRE if the current watermark is different than
> the stored one.
>
> Is this the best way to implement the functionality described above?
>
>