You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Marco Villalobos <mv...@kineteque.com> on 2021/09/07 06:23:46 UTC
What is the event time of an element produced in a timer?
If an event time timer is registered to fire exactly every 15 minutes,
starting from exactly at the top of the hour (exactly 00:00, 00:15, 00:30,
00:45 for example), and within that timer it produces an element in the
stream, what event time will that element have, and what window will it
belong to if I am using a 15 minute tumbling event time window?
Re: What is the event time of an element produced in a timer?
Posted by JING ZHANG <be...@gmail.com>.
Hi Maro,
Do you wanna to do a cascade Tumbling windowed stream Job? Do you hope the
data generated in the first Tumbling windowed stream could belongs to same
window in the second stream?
We could understand your demands better if you could share the code.
I would try to explain some detailed problems in your email. Hope it helps.
> when the watermark is GREATER than 00:45:00.000, which means that the
event time created by a timer that fires at 00:45:00.000
If first windowed stream receives watermark 00:45:00.000, it means there
should be no more elements from the input stream with a timestamp which is
older or equals to 00:45:00.000. So it's safe to trigger event timer
00:45:00.000. [1]
However, we need register a timer 00:44:59.999 instead of 00:45:00.000
because 00:44:59.999 represents the inclusive upper of the window . This
is also the behavior in Window Operator which registers a window trigger
timer (00:44:59.999) for window [00:30:00, 00:45:00) .
Window end is exclusive upper bound of the trigged window while window max
timestamp is inclusive upper bound of the trigged window.
[1]
https://ci.apache.org/projects/flink/flink-docs-master/docs/concepts/time/#event-time-and-watermarks
Best,
JING ZHANG
Marco Villalobos <mv...@kineteque.com> 于2021年9月7日周二 下午11:45写道：
> I am guessing the answer is right infront of me.
>
> If the window has these attributes:
>
> window start: 00:30:00.000.
> window end: 00:45:00.000
> max timestamp: 00:44:59.999
>
> Then perhaps I need the timer to fire at 00:44:59.999 if I want an element
> that it generates too belong to this window.
>
> I'll test my hypothesis later today.
>
>
> On Sep 7, 2021, at 2:07 AM, JING ZHANG <be...@gmail.com> wrote:
>
> Hi Marco,
> I'm not sure which API or SQL query do you use.
> If you use Windowed Stream API in DataStream [1]. The input data would be
> assigned to a Window based on which Window Assigner do you use. If using
> tumbling window, the input data would be assigned to a Window based on it's
> event-time and registers an event timer trigger for the Window. You could
> find more information in document [1]. When watermark passed the end of
> Window, the window would be triggered. The event time of the output element
> is inclusive upper bound of the trigged window.
> If you use Window Aggregation based on Flink SQL [2]. The situation is
> very similar.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/operators/windows/
> [2]
> https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/sql/queries/window-agg/
>
> Best regards,
> JING ZHANG
>
> Marco Villalobos <mv...@kineteque.com> 于2021年9月7日周二 下午2:24写道：
>
>> If an event time timer is registered to fire exactly every 15 minutes,
>> starting from exactly at the top of the hour (exactly 00:00, 00:15, 00:30,
>> 00:45 for example), and within that timer it produces an element in the
>> stream, what event time will that element have, and what window will it
>> belong to if I am using a 15 minute tumbling event time window?
>>
>
>
Re: What is the event time of an element produced in a timer?
Posted by Marco Villalobos <mv...@kineteque.com>.
I am guessing the answer is right infront of me.
If the window has these attributes:
window start: 00:30:00.000.
window end: 00:45:00.000
max timestamp: 00:44:59.999
Then perhaps I need the timer to fire at 00:44:59.999 if I want an element that it generates too belong to this window.
I'll test my hypothesis later today.
> On Sep 7, 2021, at 2:07 AM, JING ZHANG <be...@gmail.com> wrote:
>
> Hi Marco,
> I'm not sure which API or SQL query do you use.
> If you use Windowed Stream API in DataStream [1]. The input data would be assigned to a Window based on which Window Assigner do you use. If using tumbling window, the input data would be assigned to a Window based on it's event-time and registers an event timer trigger for the Window. You could find more information in document [1]. When watermark passed the end of Window, the window would be triggered. The event time of the output element is inclusive upper bound of the trigged window.
> If you use Window Aggregation based on Flink SQL [2]. The situation is very similar.
>
> [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/operators/windows/ <https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/operators/windows/>
> [2] https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/sql/queries/window-agg/ <https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/sql/queries/window-agg/>
>
> Best regards,
> JING ZHANG
>
> Marco Villalobos <mvillalobos@kineteque.com <ma...@kineteque.com>> 于2021年9月7日周二 下午2:24写道：
> If an event time timer is registered to fire exactly every 15 minutes, starting from exactly at the top of the hour (exactly 00:00, 00:15, 00:30, 00:45 for example), and within that timer it produces an element in the stream, what event time will that element have, and what window will it belong to if I am using a 15 minute tumbling event time window?
Re: What is the event time of an element produced in a timer?
Posted by Marco Villalobos <mv...@kineteque.com>.
Right.
What confuses me is that data produced within a timer is not input data. It
is synthesized within the timer and collected.
I just realized that my original question was a bit of nonsense.
KeyedProcessFuntion has timers, and it is not a window a function. What I
meant to say is that the operator prior and after are window functions with
tumbling windows.
[source] --> [window function with tumbling window 15 minutes] --> [keyed
process function that uses timers] --> [window function with tumbling
window 15 minutes] ...
My keyed process function creates data every15 minutes if the system is
idle.
Let's say the windows tumble at this time...
00:15:00.000 00:30:00.000. 00:45:00.000. 01:00:00.000
and window 00:30:00.000. 00:45:00.000, that has these attributes:
window start: 00:30:00.000.
window end: 00:45:00.000
max timestamp: 00:44:59.999
that window will fire and purge when the watermark, if using a
monotonous ascending timestamps,
when the watermark is GREATER than 00:45:00.000, which means that the event
time created by a timer that fires at 00:45:00.000
will NOT belong to window start: 00:30:00.000. window end: 00:45:00.000
when using the standard Tumbling Window assigner.
Is my understanding correct? What if I want data generated in the timer
that fires at 00:45:00.000 to belong to the window start: 00:30:00.000.
window end: 00:45:00.000 , what must I do?
It seems as though the event time will always be higher than the timestamp
that triggers the timer firing. Must I create a special window assigner
that accepts late data?
On Tue, Sep 7, 2021 at 2:07 AM JING ZHANG <be...@gmail.com> wrote:
> Hi Marco,
> I'm not sure which API or SQL query do you use.
> If you use Windowed Stream API in DataStream [1]. The input data would be
> assigned to a Window based on which Window Assigner do you use. If using
> tumbling window, the input data would be assigned to a Window based on it's
> event-time and registers an event timer trigger for the Window. You could
> find more information in document [1]. When watermark passed the end of
> Window, the window would be triggered. The event time of the output element
> is inclusive upper bound of the trigged window.
> If you use Window Aggregation based on Flink SQL [2]. The situation is
> very similar.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/operators/windows/
> [2]
> https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/sql/queries/window-agg/
>
> Best regards,
> JING ZHANG
>
> Marco Villalobos <mv...@kineteque.com> 于2021年9月7日周二 下午2:24写道：
>
>> If an event time timer is registered to fire exactly every 15 minutes,
>> starting from exactly at the top of the hour (exactly 00:00, 00:15, 00:30,
>> 00:45 for example), and within that timer it produces an element in the
>> stream, what event time will that element have, and what window will it
>> belong to if I am using a 15 minute tumbling event time window?
>>
>
Re: What is the event time of an element produced in a timer?
Posted by JING ZHANG <be...@gmail.com>.
Hi Marco,
I'm not sure which API or SQL query do you use.
If you use Windowed Stream API in DataStream [1]. The input data would be
assigned to a Window based on which Window Assigner do you use. If using
tumbling window, the input data would be assigned to a Window based on it's
event-time and registers an event timer trigger for the Window. You could
find more information in document [1]. When watermark passed the end of
Window, the window would be triggered. The event time of the output element
is inclusive upper bound of the trigged window.
If you use Window Aggregation based on Flink SQL [2]. The situation is very
similar.
[1]
https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/operators/windows/
[2]
https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/sql/queries/window-agg/
Best regards,
JING ZHANG
Marco Villalobos <mv...@kineteque.com> 于2021年9月7日周二 下午2:24写道：
> If an event time timer is registered to fire exactly every 15 minutes,
> starting from exactly at the top of the hour (exactly 00:00, 00:15, 00:30,
> 00:45 for example), and within that timer it produces an element in the
> stream, what event time will that element have, and what window will it
> belong to if I am using a 15 minute tumbling event time window?
>