You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by M Singh <ma...@yahoo.com> on 2017/11/13 19:19:20 UTC
Apache Flink - Question about TriggerResult.FIRE
Hi Flink Users
I have a few questions about triggers:
If a trigger returns TriggerResult.FIRE from say the onProcessingTime method - the window computation is triggered but elements are kept in the window. If there a second invocation of the onProcessingTime method will the elements from the previous window (which were not purged) a part of the new window computation along with new events added since the last FIRE event ?
Secondly, how does the FIRE option affect the sliding window computation ?
If there are any other insights/pitfalls while dealing with this, please let me know.
Thanks
Mans
Re: Apache Flink - Question about TriggerResult.FIRE
Posted by M Singh <ma...@yahoo.com>.
Thanks Aljoscha.
On Sunday, November 26, 2017 11:21 PM, Aljoscha Krettek <al...@apache.org> wrote:
Hi,
Yes, after the watermark (or processing-time) pass "end-of-window + allowed-lateness" everything that is stored for a window is deleted.
Best,Aljoscha
On 25. Nov 2017, at 18:07, M Singh <ma...@yahoo.com> wrote:
Hi:
Another question - what happens if the trigger never calls PURGE or FIRE_AND_PURGE and only calls FIRE ? Are the window and it's contents removed after the end time + lateness are exceeded ?
Thanks
On Monday, November 20, 2017 2:18 AM, Stefan Richter <s....@data-artisans.com> wrote:
Hi,
"In the first case, it is a new window without the previous elements, in the second case the window reflects the old contents plus all changes since the last trigger."
I am assuming the first case is FIRE and second case is FIRE_AND_PURGE - I was thinking that in the first case (FIRE), there would be elements from previous window since we did not purge and the second case it would have only new elements since it was purged. Am I missing something ?
No, with first case, I meant the „truly new window case“ and the second case is „another triggering of the previous (non-purged) window“. So the second case is a simple FIRE without PURGE.
Best,Stefan
Re: Apache Flink - Question about TriggerResult.FIRE
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
Yes, after the watermark (or processing-time) pass "end-of-window + allowed-lateness" everything that is stored for a window is deleted.
Best,
Aljoscha
> On 25. Nov 2017, at 18:07, M Singh <ma...@yahoo.com> wrote:
>
> Hi:
>
> Another question - what happens if the trigger never calls PURGE or FIRE_AND_PURGE and only calls FIRE ? Are the window and it's contents removed after the end time + lateness are exceeded ?
>
> Thanks
>
>
> On Monday, November 20, 2017 2:18 AM, Stefan Richter <s....@data-artisans.com> wrote:
>
>
> Hi,
>
>>
>> "In the first case, it is a new window without the previous elements, in the second case the window reflects the old contents plus all changes since the last trigger."
>>
>> I am assuming the first case is FIRE and second case is FIRE_AND_PURGE - I was thinking that in the first case (FIRE), there would be elements from previous window since we did not purge and the second case it would have only new elements since it was purged. Am I missing something ?
>>>
>
> No, with first case, I meant the „truly new window case“ and the second case is „another triggering of the previous (non-purged) window“. So the second case is a simple FIRE without PURGE.
>
> Best,
> Stefan
>
>
Re: Apache Flink - Question about TriggerResult.FIRE
Posted by M Singh <ma...@yahoo.com>.
Hi:
Another question - what happens if the trigger never calls PURGE or FIRE_AND_PURGE and only calls FIRE ? Are the window and it's contents removed after the end time + lateness are exceeded ?
Thanks
On Monday, November 20, 2017 2:18 AM, Stefan Richter <s....@data-artisans.com> wrote:
Hi,
"In the first case, it is a new window without the previous elements, in the second case the window reflects the old contents plus all changes since the last trigger."
I am assuming the first case is FIRE and second case is FIRE_AND_PURGE - I was thinking that in the first case (FIRE), there would be elements from previous window since we did not purge and the second case it would have only new elements since it was purged. Am I missing something ?
No, with first case, I meant the „truly new window case“ and the second case is „another triggering of the previous (non-purged) window“. So the second case is a simple FIRE without PURGE.
Best,Stefan
Re: Apache Flink - Question about TriggerResult.FIRE
Posted by Aljoscha Krettek <al...@apache.org>.
Hi Mans,
For understanding the difference between FIRE and FIRE_AND_PURGE it's helpful to look at the cases where it really makes a difference. In my opinion this only makes a difference when you have event-time windowing and when you have multiple firing for the same window (i.e. multiple firings for the window [12:00,13:00) for a key x). You have this either when you have late data or when you have a custom trigger that can fire speculatively. Let's look at the first case and see where FIRE/FIRE_AND_PURGE act differently.
Assume we have window assigner = "tumbling windows of 10 ms" and allowed lateness 5 ms. We get this data:
event1, ts = 1
watermark, ts = 11
event2, ts = 2
What we get from this is a firing when the watermark arrives with the window contents (event1) for window [0, 9). Then, when event2 arrives we get a second firing (because the default event-time trigger will FIRE when late data arrives for a window) for window [0, 9) with contents (event1, event2) if our trigger returns FIRE or with contents (event2) if our trigger returns FIRE_AND_PURGE.
If we had allowed lateness 0 ms we wouldn't have the second firing for the late event and we wouldn't see a difference between FIRE and FIRE_AND_PURGE.
This can be extended to the second case of a custom trigger that might fire early (based on the count of events in the window or processing time).
I hope this helps.
Aljoscha
> On 20. Nov 2017, at 11:18, Stefan Richter <s....@data-artisans.com> wrote:
>
> Hi,
>
>>
>> "In the first case, it is a new window without the previous elements, in the second case the window reflects the old contents plus all changes since the last trigger."
>>
>> I am assuming the first case is FIRE and second case is FIRE_AND_PURGE - I was thinking that in the first case (FIRE), there would be elements from previous window since we did not purge and the second case it would have only new elements since it was purged. Am I missing something ?
>>>
>
> No, with first case, I meant the „truly new window case“ and the second case is „another triggering of the previous (non-purged) window“. So the second case is a simple FIRE without PURGE.
>
> Best,
> Stefan
Re: Apache Flink - Question about TriggerResult.FIRE
Posted by Stefan Richter <s....@data-artisans.com>.
Hi,
>
> "In the first case, it is a new window without the previous elements, in the second case the window reflects the old contents plus all changes since the last trigger."
>
> I am assuming the first case is FIRE and second case is FIRE_AND_PURGE - I was thinking that in the first case (FIRE), there would be elements from previous window since we did not purge and the second case it would have only new elements since it was purged. Am I missing something ?
>>
No, with first case, I meant the „truly new window case“ and the second case is „another triggering of the previous (non-purged) window“. So the second case is a simple FIRE without PURGE.
Best,
Stefan
Re: Apache Flink - Question about TriggerResult.FIRE
Posted by M Singh <ma...@yahoo.com>.
Also, Stefan - You mentioned
"In the first case, it is a new window without the previous elements, in the second case the window reflects the old contents plus all changes since the last trigger."
I am assuming the first case is FIRE and second case is FIRE_AND_PURGE - I was thinking that in the first case (FIRE), there would be elements from previous window since we did not purge and the second case it would have only new elements since it was purged. Am I missing something ?
Once again, if there are any examples it would greatly help me in understanding the semantics/usage scenarios.
Mans
On Friday, November 17, 2017 10:33 AM, M Singh <ma...@yahoo.com> wrote:
Thanks Stefan and Aljoscha for your responses.
Stefan - When I mentioned "new window" - I meant the next window being created.
Eg: if the event was in w1 based processing time and the trigger returned FIRE - then after the window function is computed, what happens to the events in that window (w1). Are they (elements in w1) propagated to the next processing time window (w2) ? If not, then what is the difference between FIRE and FIRE_AND_PURGE and when do we use FIRE vs FIRE_AND PURGE ? Are there any examples to demonstrated these differences ?
Thanks again for your help.
Mans
On Thursday, November 16, 2017 5:16 AM, Aljoscha Krettek <al...@apache.org> wrote:
Yes, all of this is correct. Sliding windows in fact look like completely separate windows to the windowing system.
Best,Aljoscha
On 16. Nov 2017, at 10:15, Stefan Richter <s....@data-artisans.com> wrote:
Hi,
I think the effect is pretty straight forward, the elements in a window are not purged if the trigger is only FIRE and not FIRE_AND_PURGE. Unfortunately, your question is a bit unclear about what exactly you mean by „new window“: a truly „new“ window or another triggering of the previous (non-purged) window? In the first case, it is a new window without the previous elements, in the second case the window reflects the old contents plus all changes since the last trigger.
For sliding windows, if I remember correctly, every slide is actually a different window and elements are just added repeatedly to all windows in which they belong. So window n+1 should not be affected by whether or not window n purges or not. Maybe Aljoscha (in CC) can confirm this for us.
Best,Stefan
Am 13.11.2017 um 20:19 schrieb M Singh <ma...@yahoo.com>:
Hi Flink Users
I have a few questions about triggers:
If a trigger returns TriggerResult.FIRE from say the onProcessingTime method - the window computation is triggered but elements are kept in the window. If there a second invocation of the onProcessingTime method will the elements from the previous window (which were not purged) a part of the new window computation along with new events added since the last FIRE event ?
Secondly, how does the FIRE option affect the sliding window computation ?
If there are any other insights/pitfalls while dealing with this, please let me know.
Thanks
Mans
Re: Apache Flink - Question about TriggerResult.FIRE
Posted by M Singh <ma...@yahoo.com>.
Thanks Stefan and Aljoscha for your responses.
Stefan - When I mentioned "new window" - I meant the next window being created.
Eg: if the event was in w1 based processing time and the trigger returned FIRE - then after the window function is computed, what happens to the events in that window (w1). Are they (elements in w1) propagated to the next processing time window (w2) ? If not, then what is the difference between FIRE and FIRE_AND_PURGE and when do we use FIRE vs FIRE_AND PURGE ? Are there any examples to demonstrated these differences ?
Thanks again for your help.
Mans
On Thursday, November 16, 2017 5:16 AM, Aljoscha Krettek <al...@apache.org> wrote:
Yes, all of this is correct. Sliding windows in fact look like completely separate windows to the windowing system.
Best,Aljoscha
On 16. Nov 2017, at 10:15, Stefan Richter <s....@data-artisans.com> wrote:
Hi,
I think the effect is pretty straight forward, the elements in a window are not purged if the trigger is only FIRE and not FIRE_AND_PURGE. Unfortunately, your question is a bit unclear about what exactly you mean by „new window“: a truly „new“ window or another triggering of the previous (non-purged) window? In the first case, it is a new window without the previous elements, in the second case the window reflects the old contents plus all changes since the last trigger.
For sliding windows, if I remember correctly, every slide is actually a different window and elements are just added repeatedly to all windows in which they belong. So window n+1 should not be affected by whether or not window n purges or not. Maybe Aljoscha (in CC) can confirm this for us.
Best,Stefan
Am 13.11.2017 um 20:19 schrieb M Singh <ma...@yahoo.com>:
Hi Flink Users
I have a few questions about triggers:
If a trigger returns TriggerResult.FIRE from say the onProcessingTime method - the window computation is triggered but elements are kept in the window. If there a second invocation of the onProcessingTime method will the elements from the previous window (which were not purged) a part of the new window computation along with new events added since the last FIRE event ?
Secondly, how does the FIRE option affect the sliding window computation ?
If there are any other insights/pitfalls while dealing with this, please let me know.
Thanks
Mans
Re: Apache Flink - Question about TriggerResult.FIRE
Posted by Aljoscha Krettek <al...@apache.org>.
Yes, all of this is correct. Sliding windows in fact look like completely separate windows to the windowing system.
Best,
Aljoscha
> On 16. Nov 2017, at 10:15, Stefan Richter <s....@data-artisans.com> wrote:
>
> Hi,
>
> I think the effect is pretty straight forward, the elements in a window are not purged if the trigger is only FIRE and not FIRE_AND_PURGE. Unfortunately, your question is a bit unclear about what exactly you mean by „new window“: a truly „new“ window or another triggering of the previous (non-purged) window? In the first case, it is a new window without the previous elements, in the second case the window reflects the old contents plus all changes since the last trigger.
>
> For sliding windows, if I remember correctly, every slide is actually a different window and elements are just added repeatedly to all windows in which they belong. So window n+1 should not be affected by whether or not window n purges or not. Maybe Aljoscha (in CC) can confirm this for us.
>
> Best,
> Stefan
>
>> Am 13.11.2017 um 20:19 schrieb M Singh <mans2singh@yahoo.com <ma...@yahoo.com>>:
>>
>> Hi Flink Users
>>
>> I have a few questions about triggers:
>>
>> If a trigger returns TriggerResult.FIRE from say the onProcessingTime method - the window computation is triggered but elements are kept in the window. If there a second invocation of the onProcessingTime method will the elements from the previous window (which were not purged) a part of the new window computation along with new events added since the last FIRE event ?
>>
>> Secondly, how does the FIRE option affect the sliding window computation ?
>>
>> If there are any other insights/pitfalls while dealing with this, please let me know.
>>
>> Thanks
>>
>> Mans
>>
>>
>
Re: Apache Flink - Question about TriggerResult.FIRE
Posted by Stefan Richter <s....@data-artisans.com>.
Hi,
I think the effect is pretty straight forward, the elements in a window are not purged if the trigger is only FIRE and not FIRE_AND_PURGE. Unfortunately, your question is a bit unclear about what exactly you mean by „new window“: a truly „new“ window or another triggering of the previous (non-purged) window? In the first case, it is a new window without the previous elements, in the second case the window reflects the old contents plus all changes since the last trigger.
For sliding windows, if I remember correctly, every slide is actually a different window and elements are just added repeatedly to all windows in which they belong. So window n+1 should not be affected by whether or not window n purges or not. Maybe Aljoscha (in CC) can confirm this for us.
Best,
Stefan
> Am 13.11.2017 um 20:19 schrieb M Singh <ma...@yahoo.com>:
>
> Hi Flink Users
>
> I have a few questions about triggers:
>
> If a trigger returns TriggerResult.FIRE from say the onProcessingTime method - the window computation is triggered but elements are kept in the window. If there a second invocation of the onProcessingTime method will the elements from the previous window (which were not purged) a part of the new window computation along with new events added since the last FIRE event ?
>
> Secondly, how does the FIRE option affect the sliding window computation ?
>
> If there are any other insights/pitfalls while dealing with this, please let me know.
>
> Thanks
>
> Mans
>
>
Re: Apache Flink - Question about TriggerResult.FIRE
Posted by M Singh <ma...@yahoo.com>.
Hi Guys
Is there any insight into this ?
Thanks
Mans
On Monday, November 13, 2017 11:19 AM, M Singh <ma...@yahoo.com> wrote:
Hi Flink Users
I have a few questions about triggers:
If a trigger returns TriggerResult.FIRE from say the onProcessingTime method - the window computation is triggered but elements are kept in the window. If there a second invocation of the onProcessingTime method will the elements from the previous window (which were not purged) a part of the new window computation along with new events added since the last FIRE event ?
Secondly, how does the FIRE option affect the sliding window computation ?
If there are any other insights/pitfalls while dealing with this, please let me know.
Thanks
Mans