You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Maxim <mf...@gmail.com> on 2016/04/21 01:31:35 UTC
Control triggering on empty window
I have the following use case:
Input stream of timestamped "on" and "off" events received out of order.
I need to produce an event with time that system was "on" every 15 minutes.
Events should be produced only for intervals that system was "on".
When 15 minute window has at least one record it is triggered and the
required aggregate is created, but when no event is received within 15
minute period window is not triggered and nothing is produced.
I understand that it is not feasible to trigger on empty windows when the
set of keys is unbounded. But it would be nice to give the control for such
triggering to a window function. In my case the window function could
enable the empty triggering for the current key when the last event in the
evaluated window is "on" and disable it if is "off".
The strawman API for such feature:
public void apply(String key, TimeWindow window, Iterable<OnOffEvent>
input, Collector<Aggregate> out) throws Exception {
...
RuntimeContext context = this.getRuntimeContext();
if (lastEvent.isOn()) {
context.enableEmptyWindowTriggering();
} else {
context.disableEmptyWindowTriggering();
}
}
I could implement the same logic using global window and custom trigger and
evictor, but it looks like ugly workaround to me.
Is there any better way to solve this use case?
Thanks,
Maxim.
Re: Control triggering on empty window
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
I'm sorry but I still think this is not possible. Windows are usually
associated with a key, so if there is no element to which we can assign a
window then there is also no key to which the window would belong.
Cheers,
Aljoscha
On Thu, 21 Apr 2016 at 22:35 Maxim <mf...@gmail.com> wrote:
> I think the best way to support such a feature is to extend WindowAssigner
> with ability to be called on timer and checkpoint its state the same way it
> is done by the Trigger.
> Such WindowAssigner would be able to create Windows based on time even if
> no event is received.
>
> On Thu, Apr 21, 2016 at 1:57 AM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
>> Hi,
>> I'm afraid this is not possible with our windowing model (expect with
>> hacks using GlobalWindow, as you mentioned). The reason is, that windows
>> only come into existence once there is an element that has a window. Before
>> that, the system has no reference point about what windows there should
>> exist because there is no knowledge about time except when looking at
>> elements.
>>
>> Cheers,
>> Aljoscha
>>
>> On Thu, 21 Apr 2016 at 01:31 Maxim <mf...@gmail.com> wrote:
>>
>>> I have the following use case:
>>>
>>> Input stream of timestamped "on" and "off" events received out of order.
>>> I need to produce an event with time that system was "on" every 15
>>> minutes. Events should be produced only for intervals that system was "on".
>>>
>>> When 15 minute window has at least one record it is triggered and the
>>> required aggregate is created, but when no event is received within 15
>>> minute period window is not triggered and nothing is produced.
>>>
>>> I understand that it is not feasible to trigger on empty windows when
>>> the set of keys is unbounded. But it would be nice to give the control for
>>> such triggering to a window function. In my case the window function could
>>> enable the empty triggering for the current key when the last event in the
>>> evaluated window is "on" and disable it if is "off".
>>> The strawman API for such feature:
>>>
>>> public void apply(String key, TimeWindow window, Iterable<OnOffEvent> input, Collector<Aggregate> out) throws Exception {
>>>
>>> ...
>>>
>>> RuntimeContext context = this.getRuntimeContext();
>>>
>>> if (lastEvent.isOn()) {
>>>
>>> context.enableEmptyWindowTriggering();
>>>
>>> } else {
>>>
>>> context.disableEmptyWindowTriggering();
>>>
>>> }
>>>
>>> }
>>>
>>> I could implement the same logic using global window and custom trigger
>>> and evictor, but it looks like ugly workaround to me.
>>>
>>> Is there any better way to solve this use case?
>>>
>>> Thanks,
>>>
>>> Maxim.
>>>
>>>
>
Re: Control triggering on empty window
Posted by Maxim <mf...@gmail.com>.
I think the best way to support such a feature is to extend WindowAssigner
with ability to be called on timer and checkpoint its state the same way it
is done by the Trigger.
Such WindowAssigner would be able to create Windows based on time even if
no event is received.
On Thu, Apr 21, 2016 at 1:57 AM, Aljoscha Krettek <al...@apache.org>
wrote:
> Hi,
> I'm afraid this is not possible with our windowing model (expect with
> hacks using GlobalWindow, as you mentioned). The reason is, that windows
> only come into existence once there is an element that has a window. Before
> that, the system has no reference point about what windows there should
> exist because there is no knowledge about time except when looking at
> elements.
>
> Cheers,
> Aljoscha
>
> On Thu, 21 Apr 2016 at 01:31 Maxim <mf...@gmail.com> wrote:
>
>> I have the following use case:
>>
>> Input stream of timestamped "on" and "off" events received out of order.
>> I need to produce an event with time that system was "on" every 15
>> minutes. Events should be produced only for intervals that system was "on".
>>
>> When 15 minute window has at least one record it is triggered and the
>> required aggregate is created, but when no event is received within 15
>> minute period window is not triggered and nothing is produced.
>>
>> I understand that it is not feasible to trigger on empty windows when the
>> set of keys is unbounded. But it would be nice to give the control for such
>> triggering to a window function. In my case the window function could
>> enable the empty triggering for the current key when the last event in the
>> evaluated window is "on" and disable it if is "off".
>> The strawman API for such feature:
>>
>> public void apply(String key, TimeWindow window, Iterable<OnOffEvent> input, Collector<Aggregate> out) throws Exception {
>>
>> ...
>>
>> RuntimeContext context = this.getRuntimeContext();
>>
>> if (lastEvent.isOn()) {
>>
>> context.enableEmptyWindowTriggering();
>>
>> } else {
>>
>> context.disableEmptyWindowTriggering();
>>
>> }
>>
>> }
>>
>> I could implement the same logic using global window and custom trigger
>> and evictor, but it looks like ugly workaround to me.
>>
>> Is there any better way to solve this use case?
>>
>> Thanks,
>>
>> Maxim.
>>
>>
Re: Control triggering on empty window
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
I'm afraid this is not possible with our windowing model (expect with hacks
using GlobalWindow, as you mentioned). The reason is, that windows only
come into existence once there is an element that has a window. Before
that, the system has no reference point about what windows there should
exist because there is no knowledge about time except when looking at
elements.
Cheers,
Aljoscha
On Thu, 21 Apr 2016 at 01:31 Maxim <mf...@gmail.com> wrote:
> I have the following use case:
>
> Input stream of timestamped "on" and "off" events received out of order.
> I need to produce an event with time that system was "on" every 15
> minutes. Events should be produced only for intervals that system was "on".
>
> When 15 minute window has at least one record it is triggered and the
> required aggregate is created, but when no event is received within 15
> minute period window is not triggered and nothing is produced.
>
> I understand that it is not feasible to trigger on empty windows when the
> set of keys is unbounded. But it would be nice to give the control for such
> triggering to a window function. In my case the window function could
> enable the empty triggering for the current key when the last event in the
> evaluated window is "on" and disable it if is "off".
> The strawman API for such feature:
>
> public void apply(String key, TimeWindow window, Iterable<OnOffEvent> input, Collector<Aggregate> out) throws Exception {
>
> ...
>
> RuntimeContext context = this.getRuntimeContext();
>
> if (lastEvent.isOn()) {
>
> context.enableEmptyWindowTriggering();
>
> } else {
>
> context.disableEmptyWindowTriggering();
>
> }
>
> }
>
> I could implement the same logic using global window and custom trigger
> and evictor, but it looks like ugly workaround to me.
>
> Is there any better way to solve this use case?
>
> Thanks,
>
> Maxim.
>
>