Watermark advancement in late side output

[Ori Popowski <or...@gmail.com>]

Let's say I have an event-time stream with a window and a side output for
late data, and in the side output of the late data, I further assign
timestamps and do windowing - what is the watermark situation here?

The main stream has its own watermark advancement but the side output has
its own. Do they maintain separate watermarks? Or they intermingle?

Thanks

Re: Watermark advancement in late side output

[orips <or...@gmail.com>]

Great, thanks



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Watermark advancement in late side output

[Timo Walther <tw...@apache.org>]

Hi Ori,

first of all, watermarks are sent to all side outputs (this is tested 
here [1]). Thus, operators in the side output branch of the pipeline 
will work similar to operators in the main branch.

When calling `assignTimestampsAndWatermarks`, the inserted operator will 
erase incoming watermarks and only emit self-generated ones. The logic 
can be found here [2]. Thus, downstream operators in the side output 
will only consider the newly assigned one (+ the end watermark Long.MAX).

I hope this helps.

Regards,
Timo

[1] 
https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/SideOutputITCase.java
[2] 
https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/TimestampsAndWatermarksOperator.java#L114

On 21.09.20 12:21, Ori Popowski wrote:
> Let's say I have an event-time stream with a window and a side output 
> for late data, and in the side output of the late data, I further assign 
> timestamps and do windowing - what is the watermark situation here?
> 
> The main stream has its own watermark advancement but the side output 
> has its own. Do they maintain separate watermarks? Or they intermingle?
> 
> Thanks