Question about Watermarks within a KeyedProcessFunction

[Marco Villalobos <mv...@kineteque.com>]

My source is a Kafka topic.
I am using Event Time.
I assign the event time with an AscendingTimestampExtractor

I noticed when debugging that in the KeyedProcessFunction that
after my highest known event time of:  2020-06-23T00:46:30.000Z

the processElement method had a watermark with an impossible date of:
-292275055-05-16T16:47:04.192Z

but in the onTimer method it had a more reasonable value that trails the
highest known event time by 1 millisecond, which is this
value:  2020-06-23T00:46:29.999Z

I want to know, why does the processElement method have an impossible
watermark value?

Re: Question about Watermarks within a KeyedProcessFunction

[David Anderson <da...@alpinegizmo.com>]

With an AscendingTimestampExtractor, watermarks are not created for every
event, and as your job starts up, some events will be processed before the
first watermark is generated.

The impossible value you see is an initial value that's in place until the
first real watermark is available. On the other hand, onTimer can not be
called until some timer is triggered by the arrival of a watermark, at
which point the watermark will have a reasonable value.

On Sat, Jun 27, 2020 at 2:37 AM Marco Villalobos <mv...@kineteque.com>
wrote:

>
> My source is a Kafka topic.
> I am using Event Time.
> I assign the event time with an AscendingTimestampExtractor
>
> I noticed when debugging that in the KeyedProcessFunction that
> after my highest known event time of:  2020-06-23T00:46:30.000Z
>
> the processElement method had a watermark with an impossible date of:
> -292275055-05-16T16:47:04.192Z
>
> but in the onTimer method it had a more reasonable value that trails the
> highest known event time by 1 millisecond, which is this
> value:  2020-06-23T00:46:29.999Z
>
> I want to know, why does the processElement method have an impossible
> watermark value?
>
>
>