You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Ozan DENİZ <oz...@outlook.com> on 2016/03/24 09:03:02 UTC

Apache Flink: aligning watermark among parallel tasks

We are using periodic event time window with watermark. We have currently 4 parallel tasks in our Flink App.

During the streaming process, all the 4 tasks' watermark values must be close to trigger window event.





  For example;

  
  Task 1 watermark value = 8

  
  Task 2 watermark value = 1

  
  Task 3 watermark value = 8

  
  Task 4 watermark value = 8





Task 2 is waiting for log to update its watermark. However, 
the condition can occur before Task 2's update and we want to fire the 
window event before it. 




Is there any mechanism to align all the parallel tasks' watermarks or fire the window event without waiting for other tasks?
 		 	   		  

Re: Apache Flink: aligning watermark among parallel tasks

Posted by Maximilian Michels <mx...@apache.org>.
Hi Ozan,

You probably want to look at a custom Trigger implementation. Please see
the different triggers in
org/apache/flink/streaming/api/windowing/triggers/. You can write your own
event-based trigger. Best thing would be to extend the EventTimeTrigger
with your logic.

Then you can use windowedStream.trigger(new MyCustomTrigger()) to include
your trigger logic on your windowed stream.

Cheers,
Max


On Thu, Mar 24, 2016 at 10:51 AM, Ozan DENİZ <oz...@outlook.com> wrote:

> We are using periodic event time window with watermark. We have currently
> 4 parallel tasks in our Flink App.
>
> During the streaming process, all the 4 tasks' watermark values must be
> close to trigger window event.
>
>
>
>
>
>   For example;
>
>
>   Task 1 watermark value = 8
>
>
>   Task 2 watermark value = 1
>
>
>   Task 3 watermark value = 8
>
>
>   Task 4 watermark value = 8
>
>
>
>
>
> Task 2 is waiting for log to update its watermark. However,
> the condition can occur before Task 2's update and we want to fire the
> window event before it.
>
>
>
>
> Is there any mechanism to align all the parallel tasks' watermarks or fire
> the window event without waiting for other tasks?
>
>
>

Apache Flink: aligning watermark among parallel tasks

Posted by Ozan DENİZ <oz...@outlook.com>.
We are using periodic event time window with watermark. We have currently 4 parallel tasks in our Flink App.

During the streaming process, all the 4 tasks' watermark values must be close to trigger window event.





  For example;

  
  Task 1 watermark value = 8

  
  Task 2 watermark value = 1

  
  Task 3 watermark value = 8

  
  Task 4 watermark value = 8





Task 2 is waiting for log to update its watermark. However, 
the condition can occur before Task 2's update and we want to fire the 
window event before it. 




Is there any mechanism to align all the parallel tasks' watermarks or fire the window event without waiting for other tasks?