[jira] [Created] (FLINK-18188) Document asymmetric allocation of Flink memory

["Chesnay Schepler (Jira)" <ji...@apache.org>]

Chesnay Schepler created FLINK-18188:
----------------------------------------

             Summary: Document asymmetric allocation of Flink memory
                 Key: FLINK-18188
                 URL: https://issues.apache.org/jira/browse/FLINK-18188
             Project: Flink
          Issue Type: Improvement
          Components: Runtime / Coordination
            Reporter: Chesnay Schepler
             Fix For: 1.11.0


I stumbled on some non-documented and surprising behavior while configuring the heap/off-heap memory sizes.

Let's start with a base line, where we configure the total process memory to 1024mb:
{code}
Final Master Memory configuration:
  Total Process Memory:      1024.000mb (1073741824 bytes)
    Total Flink Memory:      576.000mb (603979776 bytes)
      Flink Heap Memory:     448.000mb (469762048 bytes)
      Flink Off-heap Memory: 128.000mb (134217728 bytes)
    JVM Metaspace Memory:    256.000mb (268435456 bytes)
    JVM Overhead Memory:     192.000mb (201326592 bytes)
{code}

If we reduce the off-heap memory to 16mb, then the gained memory is allocated to the heap memory:
{code}
Final Master Memory configuration:
  Total Process Memory:      1024.000mb (1073741824 bytes)
    Total Flink Memory:      576.000mb (603979776 bytes)
      Flink Heap Memory:     560.000mb (587202560 bytes)
      Flink Off-heap Memory: 16.000mb (16777216 bytes)
    JVM Metaspace Memory:    256.000mb (268435456 bytes)
    JVM Overhead Memory:     192.000mb (201326592 bytes)
{code}

Conversely however, if we reduce the heap memory to 128mb, then the gained memory is allocated to the JVM overhead.
{code}
Final Master Memory configuration:
  Total Process Memory:      1024.000mb (1073741824 bytes)
    Total Flink Memory:      256.000mb (268435456 bytes)
      Flink Heap Memory:     128.000mb (134217728 bytes)
      Flink Off-heap Memory: 128.000mb (134217728 bytes)
    JVM Metaspace Memory:    256.000mb (268435456 bytes)
    JVM Overhead Memory:     512.000mb (536870912 bytes)
{code}

I'm not sure whether this behavior is correct.

The documentation for {{Capped Fractionated Components}} does that "_It can also happen that the fraction is ignored if the sizes of the total memory and its other components are defined. In this case, the JVM Overhead is the rest of the total memory._"
This however only explains one of the above cases (the last one).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)