Regarding caching the evicted elements and re-emitting them to the next window

[Abdul Salam Shaikh <ab...@gmail.com>]

Hi,

I am using 1.2-Snapshot version of Apache Flink which provides the new
enhanced Evictor functionality and using customized triggers for Global
Window. I have a use case where I am evicting the unwanted event(element)
for the current window before it is evaluated. However, I am looking for
options to cache this evicted element and re-use it in the next window. Is
there a possibility which can help me achieve this in the context of Flink
or in a more generic programming approach.

Thanks in anticipation!

Re: Regarding caching the evicted elements and re-emitting them to the next window

[Shaoxuan Wang <ws...@gmail.com>]

Hi Abdul,
You may want to check out FLIP13 "side output" https://goo.gl/6KSYd0 . Once
we have this feature, you should be able to collect the data to the
external distributed storage, and use these data later on demand.
BTW, can you explain your use case in more details, such that people here
may help you figure out a better solution (it perhaps may just need some
tunings on your query plan).

Shaoxuan


On Sat, Jan 14, 2017 at 12:22 PM, Aljoscha Krettek <al...@apache.org>
wrote:

> Hi,
> I'm afraid there is no functionality for this in Flink. What you can do,
> however, is to not evict these elements from the window buffer but instead
> ignore them when processing your elements in the WindowFunction. This way
> they will be preserved for the next firing. You have to make sure to
> eventually evict some elements, however. Otherwise you would have a memory
> leak.
>
> Aljoscha
>
> On Sun, Jan 8, 2017, 23:47 Abdul Salam Shaikh <ab...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am using 1.2-Snapshot version of Apache Flink which provides the new
>> enhanced Evictor functionality and using customized triggers for Global
>> Window. I have a use case where I am evicting the unwanted event(element)
>> for the current window before it is evaluated. However, I am looking for
>> options to cache this evicted element and re-use it in the next window. Is
>> there a possibility which can help me achieve this in the context of Flink
>> or in a more generic programming approach.
>>
>> Thanks in anticipation!
>>
>

Re: Regarding caching the evicted elements and re-emitting them to the next window

[Aljoscha Krettek <al...@apache.org>]

Hi,
I'm afraid there is no functionality for this in Flink. What you can do,
however, is to not evict these elements from the window buffer but instead
ignore them when processing your elements in the WindowFunction. This way
they will be preserved for the next firing. You have to make sure to
eventually evict some elements, however. Otherwise you would have a memory
leak.

Aljoscha

On Sun, Jan 8, 2017, 23:47 Abdul Salam Shaikh <ab...@gmail.com>
wrote:

> Hi,
>
> I am using 1.2-Snapshot version of Apache Flink which provides the new
> enhanced Evictor functionality and using customized triggers for Global
> Window. I have a use case where I am evicting the unwanted event(element)
> for the current window before it is evaluated. However, I am looking for
> options to cache this evicted element and re-use it in the next window. Is
> there a possibility which can help me achieve this in the context of Flink
> or in a more generic programming approach.
>
> Thanks in anticipation!
>