You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Björn Zachrisson <bj...@gmail.com> on 2017/10/02 11:06:47 UTC
Flink Watermark and timing
Hi,
I have a question regarding timing of events.
According to;
https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/event_time.html#event-time-and-watermarks
All events up to and equal to watermark should be handled in "the prevoius
window".
In my case I use event-timestamp.
I'm testing the timing out.
The case is events from 2000-01-01 02:00:00 and up to 2000-01-01 02:20:00
where eavh event is 2 minutes apart. I try to group the events in 5 minute
windows
2000-01-01 02:00:00 => 2000-01-01 02:05:00
2000-01-01 02:05:00 => 2000-01-01 02:10:00
2000-01-01 02:10:00 => 2000-01-01 02:15:00
2000-01-01 02:15:00 => 2000-01-01 02:20:00
How ever, events at the exakt time 02:10:00 (946692600000) is put in the
Window "2000-01-01 02:10:00 => 2000-01-01 02:15:00" which is not according
to what i can read on the wiki.
This is the exakt result;
2000-01-01 02:00:00, 946692000000
2000-01-01 02:02:00, 946692120000
2000-01-01 02:04:00, 946692240000
2000-01-01 02:06:00, 946692360000
2000-01-01 02:08:00, 946692480000
2000-01-01 02:10:00, 946692600000
2000-01-01 02:12:00, 946692720000
2000-01-01 02:14:00, 946692840000
2000-01-01 02:16:00, 946692960000
2000-01-01 02:18:00, 946693080000
2000-01-01 02:20:00, 946693200000
Is this due to that I'm using event time extractor or what might be the
case?
Regards
Björn
Re: Flink Watermark and timing
Posted by Timo Walther <tw...@apache.org>.
Hi Björn,
I don't know if I get your example correctly, but I think your
explanation "All events up to and equal to watermark should be handled
in the prevoius window" is not 100% correct. Watermarks just indicate
the progress ("until here we have seen all events with lower timestamp
than X") and trigger the evaluation of a window. The assignment of
events to windows is based on the timestamp not the watermark. The
documentation will be improved for the upcoming release:
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/windows.html#window-assigners
"Time-based windows have a start timestamp (inclusive) and an end
timestamp (exclusive) that together describe the size of the window. "
I hope this helps.
Regards,
Timo
Am 10/2/17 um 1:06 PM schrieb Björn Zachrisson:
> Hi,
>
> I have a question regarding timing of events.
>
> According to;
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/event_time.html#event-time-and-watermarks
>
> All events up to and equal to watermark should be handled in "the
> prevoius window".
>
> In my case I use event-timestamp.
>
>
> I'm testing the timing out.
>
> The case is events from 2000-01-01 02:00:00 and up to 2000-01-01
> 02:20:00 where eavh event is 2 minutes apart. I try to group the
> events in 5 minute windows
>
> 2000-01-01 02:00:00 => 2000-01-01 02:05:00
> 2000-01-01 02:05:00 => 2000-01-01 02:10:00
> 2000-01-01 02:10:00 => 2000-01-01 02:15:00
> 2000-01-01 02:15:00 => 2000-01-01 02:20:00
>
> How ever, events at the exakt time 02:10:00 (946692600000) is put in
> the Window "2000-01-01 02:10:00 => 2000-01-01 02:15:00" which is not
> according to what i can read on the wiki.
>
> This is the exakt result;
> 2000-01-01 02:00:00, 946692000000
> 2000-01-01 02:02:00, 946692120000
> 2000-01-01 02:04:00, 946692240000
>
> 2000-01-01 02:06:00, 946692360000
> 2000-01-01 02:08:00, 946692480000
>
> 2000-01-01 02:10:00, 946692600000
> 2000-01-01 02:12:00, 946692720000
> 2000-01-01 02:14:00, 946692840000
>
> 2000-01-01 02:16:00, 946692960000
> 2000-01-01 02:18:00, 946693080000
>
> 2000-01-01 02:20:00, 946693200000
>
> Is this due to that I'm using event time extractor or what might be
> the case?
>
> Regards
> Björn
>
>