Does heap memory gets released after computing windows?

[Rahul Raj <ra...@gmail.com>]

Hi All,

I have been facing "GC Overhead limit exceeded" exception for quite a time
with my flink program. In my flink program, I have created a session window
with time gap of 10 minutes and I am performing some custom reduce
operation using Window function. The current statebackend is File System.
The program works fine for 15-20 mins and then fails with "GC overhead
limited exception" . My understanding of flink is that, the task manager's
memory will be released as soon as operation under window function will be
completed. But then why I am getting this error? What should I do to avoid
this?

Rahul Raj

Re: Does heap memory gets released after computing windows?

[Piotr Nowojski <pi...@data-artisans.com>]

Hi,

Memory used by session windows should be released once window is triggered (allowedLateness can prolong window’s life). Unless your code introduces some memory leak (by not releasing references) everything should be garbage collected.

Keep in mind that session windows with time gap of 10 minute, doesn’t mean that you will have a constant/fixed memory usage. As the times goes by and you process more and more elements your memory usage can increase indefinitely as events with new keys arrive and if older windows are still being kept alive (by arrival of events with existing keys).

Also check what you are doing in your reduce function. If you collect incoming events on some collection then your memory footprint also can grow over time.

Piotrek

> On 20 Oct 2017, at 12:55, Rahul Raj <ra...@gmail.com> wrote:
> 
> Hi All,
> 
> I have been facing "GC Overhead limit exceeded" exception for quite a time with my flink program. In my flink program, I have created a session window with time gap of 10 minutes and I am performing some custom reduce operation using Window function. The current statebackend is File System. The program works fine for 15-20 mins and then fails with "GC overhead limited exception" . My understanding of flink is that, the task manager's memory will be released as soon as operation under window function will be completed. But then why I am getting this error? What should I do to avoid this?
> 
> Rahul Raj