[jira] [Commented] (FLINK-16614) FLIP-116 Unified Memory Configuration for Job Manager

["Andrey Zagrebin (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-16614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117630#comment-17117630 ] 

Andrey Zagrebin commented on FLINK-16614:
-----------------------------------------

h1. New Flink Master Memory Model

h2. Overview

With [FLIP-116|https://cwiki.apache.org/confluence/display/FLINK/FLIP+116%3A+Unified+Memory+Configuration+for+Job+Managers], a new memory model has been introduced for the Flink Master. New configuration options have been introduced to control the memory consumption of the Flink Master process. This affects all types of deployments: standalone, YARN, Mesos, and the new active Kubernetes integration.

Please, check the user documentation for [more details|https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup_master.html].

If you try to reuse your previous Flink configuration without any adjustments, the new memory model can result in differently computed memory parameters for the JVM and, thus, performance or even correctness changes. See also [the migration guide|https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html#migrate-job-manager-memory-configuration].

h2. Deprecation and breaking changes

The following options are deprecated: 
* _jobmanager.heap.size_
* _jobmanager.heap.mb_

If these deprecated options are still used, they will be interpreted as one of the following new options in order to maintain backwards compatibility:
* *JVM Heap* ([_jobmanager.memory.heap.size_|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#jobmanager-memory-heap-size]) for standalone and Mesos deployments
* *Total process memory* ([_jobmanager.memory.process.size_|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#jobmanager-memory-process-size]) for containerized deployments (Kubernetes and Yarn)

It is also recommended using these new options instead of the legacy ones as they might be completely removed in the following releases. See also how to configure the [Total|https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup_master.html#configure-total-memory] and [JVM Heap|https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup_master.html#configure-jvm-heap] memory.

The following options have been removed and have no effect anymore:
* _containerized.heap-cutoff-ratio_
* _containerized.heap-cutoff-min_

There is [no container cut-off|https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html#container-cut-off-memory] anymore.

h2. JVM arguments

The _direct_ and _metaspace_ memory of the Flink Master's JVM process are now limited by configurable values:
* [_jobmanager.memory.off-heap.size_|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#jobmanager-memory-off-heap-size]
* [_jobmanager.memory.jvm-metaspace.size_|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#jobmanager-memory-jvm-metaspace-size]

See also [JVM Parameters|https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_setup.html#jvm-parameters].

*Attention:* These new limits can produce the respective _OutOfMemoryError_ exceptions if they are not configured properly or there is a respective memory leak. See also [the troubleshooting guide|https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_trouble.html#outofmemoryerror-direct-buffer-memory].


> FLIP-116 Unified Memory Configuration for Job Manager
> -----------------------------------------------------
>
>                 Key: FLINK-16614
>                 URL: https://issues.apache.org/jira/browse/FLINK-16614
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Configuration, Runtime / Coordination
>            Reporter: Andrey Zagrebin
>            Assignee: Andrey Zagrebin
>            Priority: Major
>              Labels: Umbrella
>             Fix For: 1.11.0
>
>
> This is the umbrella issue of [FLIP-116: Unified Memory Configuration for Job Managers|https://cwiki.apache.org/confluence/display/FLINK/FLIP+116%3A+Unified+Memory+Configuration+for+Job+Managers].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)