Overview of Memory Management in Flink

[Stephan Ewen <se...@apache.org>]

Hi all!

Here is a first version of the documentation how memory management works in
Flink.

I hope it sheds some light on the magic we do. Let me know if certain
sections are still confusing.

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=53741525

Greetings,
Stephan

Re: Overview of Memory Management in Flink

[Maximilian Michels <mx...@apache.org>]

We should have all parts of Flink documented that well. Very nice overview,
Stephan!

Two comments:

Considering the lazy memory segment allocation. Besides, the problem of
initial garbage collection of these dynamically allocated segments,
wouldn't it also have the disadvantage that we could not guarantee a fixed
amount of memory? How would we check that the user memory does not take
away the managed memory?

I would add a note to why java.misc.Unsafe is faster for longs. I would
guess because of the coding of the long (little or big endian).

Best,
Max

On Wed, Mar 18, 2015 at 5:21 PM, Stephan Ewen <se...@apache.org> wrote:

> Hi all!
>
> Here is a first version of the documentation how memory management works in
> Flink.
>
> I hope it sheds some light on the magic we do. Let me know if certain
> sections are still confusing.
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=53741525
>
> Greetings,
> Stephan
>