Propagating delta from window upon trigger

[Nikhil Goyal <no...@gmail.com>]

Hi guys,

Is there a way in Flink to only propagate the changes which happened in the
window's state rather than dumbing the contents of the window again and
again upon trigger?

Thanks
Nikhil

Re: Propagating delta from window upon trigger

[Nikhil Goyal <no...@gmail.com>]

Thanks Rong, I will look at the links you mentioned.


On Wed, May 22, 2019 at 10:39 AM Rong Rong <wa...@gmail.com> wrote:

> Hi Miki,
>
> Upon trigger, window will be fired with all of its content in its state
> invoking the "emitWindowContent" method. which will further invoke the
> window function you define.
> Thus if your goal is to only emit the delta, one thing is to do so in your
> window function. One challenge you might face is how to make your window
> function stateful, you might want to checkout this document[1] for more
> details.
>
> If your goal is to limit the number of content stored in the window state
> to only the delta. you can also look at incremental aggregation state [2]
> of the window, but you will have to design your window function in such a
> way that it can take delta into account.
>
> --
> Rong
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#processwindowfunction
> [2]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#processwindowfunction-with-incremental-aggregation
>
>
> On Sun, May 19, 2019 at 8:00 PM Congxian Qiu <qc...@gmail.com>
> wrote:
>
>> Hi, Nikhil
>> Window will emit all state to downstream. Can you clear the state while
>> triggering?
>>
>> Nikhil Goyal <no...@gmail.com>于2019年5月19日 周日01:03写道：
>>
>>> I have window of 1 hour and trigger of 5 min. I want to know if every 5
>>> min Flink is writing the entire window or only the keys which changed.
>>>
>>> On Sat, May 18, 2019, 9:40 AM miki haiat <mi...@gmail.com> wrote:
>>>
>>>> Can you elaborate more what  is you use case ?
>>>>
>>>>
>>>> On Sat, May 18, 2019 at 12:47 AM Nikhil Goyal <no...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi guys,
>>>>>
>>>>> Is there a way in Flink to only propagate the changes which happened
>>>>> in the window's state rather than dumbing the contents of the window again
>>>>> and again upon trigger?
>>>>>
>>>>> Thanks
>>>>> Nikhil
>>>>>
>>>> --
>> Best,
>> Congxian
>>
>

Re: Propagating delta from window upon trigger

[Rong Rong <wa...@gmail.com>]

Hi Miki,

Upon trigger, window will be fired with all of its content in its state
invoking the "emitWindowContent" method. which will further invoke the
window function you define.
Thus if your goal is to only emit the delta, one thing is to do so in your
window function. One challenge you might face is how to make your window
function stateful, you might want to checkout this document[1] for more
details.

If your goal is to limit the number of content stored in the window state
to only the delta. you can also look at incremental aggregation state [2]
of the window, but you will have to design your window function in such a
way that it can take delta into account.

--
Rong

[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#processwindowfunction
[2]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#processwindowfunction-with-incremental-aggregation


On Sun, May 19, 2019 at 8:00 PM Congxian Qiu <qc...@gmail.com> wrote:

> Hi, Nikhil
> Window will emit all state to downstream. Can you clear the state while
> triggering?
>
> Nikhil Goyal <no...@gmail.com>于2019年5月19日 周日01:03写道：
>
>> I have window of 1 hour and trigger of 5 min. I want to know if every 5
>> min Flink is writing the entire window or only the keys which changed.
>>
>> On Sat, May 18, 2019, 9:40 AM miki haiat <mi...@gmail.com> wrote:
>>
>>> Can you elaborate more what  is you use case ?
>>>
>>>
>>> On Sat, May 18, 2019 at 12:47 AM Nikhil Goyal <no...@gmail.com>
>>> wrote:
>>>
>>>> Hi guys,
>>>>
>>>> Is there a way in Flink to only propagate the changes which happened in
>>>> the window's state rather than dumbing the contents of the window again and
>>>> again upon trigger?
>>>>
>>>> Thanks
>>>> Nikhil
>>>>
>>> --
> Best,
> Congxian
>

Re: Propagating delta from window upon trigger

[Congxian Qiu <qc...@gmail.com>]

Hi, Nikhil
Window will emit all state to downstream. Can you clear the state while
triggering?

Nikhil Goyal <no...@gmail.com>于2019年5月19日 周日01:03写道：

> I have window of 1 hour and trigger of 5 min. I want to know if every 5
> min Flink is writing the entire window or only the keys which changed.
>
> On Sat, May 18, 2019, 9:40 AM miki haiat <mi...@gmail.com> wrote:
>
>> Can you elaborate more what  is you use case ?
>>
>>
>> On Sat, May 18, 2019 at 12:47 AM Nikhil Goyal <no...@gmail.com>
>> wrote:
>>
>>> Hi guys,
>>>
>>> Is there a way in Flink to only propagate the changes which happened in
>>> the window's state rather than dumbing the contents of the window again and
>>> again upon trigger?
>>>
>>> Thanks
>>> Nikhil
>>>
>> --
Best,
Congxian

Re: Propagating delta from window upon trigger

[Nikhil Goyal <no...@gmail.com>]

I have window of 1 hour and trigger of 5 min. I want to know if every 5 min
Flink is writing the entire window or only the keys which changed.

On Sat, May 18, 2019, 9:40 AM miki haiat <mi...@gmail.com> wrote:

> Can you elaborate more what  is you use case ?
>
>
> On Sat, May 18, 2019 at 12:47 AM Nikhil Goyal <no...@gmail.com> wrote:
>
>> Hi guys,
>>
>> Is there a way in Flink to only propagate the changes which happened in
>> the window's state rather than dumbing the contents of the window again and
>> again upon trigger?
>>
>> Thanks
>> Nikhil
>>
>

Re: Propagating delta from window upon trigger

[miki haiat <mi...@gmail.com>]

Can you elaborate more what  is you use case ?


On Sat, May 18, 2019 at 12:47 AM Nikhil Goyal <no...@gmail.com> wrote:

> Hi guys,
>
> Is there a way in Flink to only propagate the changes which happened in
> the window's state rather than dumbing the contents of the window again and
> again upon trigger?
>
> Thanks
> Nikhil
>